Measure of relative information in probability theory
Venn diagram showing additive and subtractive relationships various
information measures associated with correlated variables
![X](https://wikimedia.org/api/rest_v1/media/math/render/svg/68baa052181f707c662844a465bfeeb135e82bab)
and
![Y](https://wikimedia.org/api/rest_v1/media/math/render/svg/961d67d6b454b4df2301ac571808a3538b3a6d3f)
. The area contained by both circles is the
joint entropy ![{\displaystyle \mathrm {H} (X,Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/870bbc4cf58de95ebd325676f495ffda21a21c97)
. The circle on the left (red and violet) is the
individual entropy ![{\displaystyle \mathrm {H} (X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b3655e2f8fe1012c2cc749241584fdcc00bfa318)
, with the red being the conditional entropy
![{\displaystyle \mathrm {H} (X|Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b8e6e0bbd44b0c6a6ed669cd328addc7ff3daf5c)
. The circle on the right (blue and violet) is
![{\displaystyle \mathrm {H} (Y)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b65e884746ac4cde16228d02b6bdea42b7dccbd6)
, with the blue being
![{\displaystyle \mathrm {H} (Y|X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/38a0bc0e103304b83100d21ce5090c3c8c62aae6)
. The violet is the
mutual information ![\operatorname {I} (X;Y)](https://wikimedia.org/api/rest_v1/media/math/render/svg/a7771931da3da8df3d83d58ab8d42715ebbb3fa4)
.
In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable
given that the value of another random variable
is known. Here, information is measured in shannons, nats, or hartleys. The entropy of
conditioned on
is written as
.
Definition[]
The conditional entropy of
given
is defined as
![{\displaystyle \mathrm {H} (Y|X)\ =-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log {\frac {p(x,y)}{p(x)}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/f070a4d53ced96461310f786e6ef86e812ca1088)
|
|
(Eq.1)
|
where
and
denote the support sets of
and
.
Note: It is conventioned that the expressions
and
for fixed
should be treated as being equal to zero. This is because
and
[1]
Intuitive explanation of the definition :
According to the definition,
where
associates to
the information content of
given
, which is the amount of information needed to describe the event
given
. According to the law of large numbers,
is the arithmetic mean of a large number of independent realizations of
.
Motivation[]
Let
be the entropy of the discrete random variable
conditioned on the discrete random variable
taking a certain value
. Denote the support sets of
and
by
and
. Let
have probability mass function
. The unconditional entropy of
is calculated as
, i.e.
![{\displaystyle \mathrm {H} (Y)=\sum _{y\in {\mathcal {Y}}}{\mathrm {Pr} (Y=y)\,\mathrm {I} (y)}=-\sum _{y\in {\mathcal {Y}}}{p_{Y}(y)\log _{2}{p_{Y}(y)}},}](https://wikimedia.org/api/rest_v1/media/math/render/svg/56c1e9429e845f0da75056f49e820a651f70c667)
where
is the information content of the outcome of
taking the value
. The entropy of
conditioned on
taking the value
is defined analogously by conditional expectation:
![{\displaystyle \mathrm {H} (Y|X=x)=-\sum _{y\in {\mathcal {Y}}}{\Pr(Y=y|X=x)\log _{2}{\Pr(Y=y|X=x)}}.}](https://wikimedia.org/api/rest_v1/media/math/render/svg/d022808f102ccf31871c057dab73b484439235ba)
Note that
is the result of averaging
over all possible values
that
may take. Also, if the above sum is taken over a sample
, the expected value
is known in some domains as equivocation.[2]
Given discrete random variables
with image
and
with image
, the conditional entropy of
given
is defined as the weighted sum of
for each possible value of
, using
as the weights:[3]: 15
![{\displaystyle {\begin{aligned}\mathrm {H} (Y|X)\ &\equiv \sum _{x\in {\mathcal {X}}}\,p(x)\,\mathrm {H} (Y|X=x)\\&=-\sum _{x\in {\mathcal {X}}}p(x)\sum _{y\in {\mathcal {Y}}}\,p(y|x)\,\log \,p(y|x)\\&=-\sum _{x\in {\mathcal {X}}}\sum _{y\in {\mathcal {Y}}}\,p(x,y)\,\log \,p(y|x)\\&=-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log \,p(y|x)\\&=-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log {\frac {p(x,y)}{p(x)}}.\\&=\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log {\frac {p(x)}{p(x,y)}}.\\\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c200b367c0f09c8d1faad3319c6c393d3ebbe539)
Properties[]
Conditional entropy equals zero[]
if and only if the value of
is completely determined by the value of
.
Conditional entropy of independent random variables[]
Conversely,
if and only if
and
are independent random variables.
Chain rule[]
Assume that the combined system determined by two random variables
and
has joint entropy
, that is, we need
bits of information on average to describe its exact state. Now if we first learn the value of
, we have gained
bits of information. Once
is known, we only need
bits to describe the state of the whole system. This quantity is exactly
, which gives the chain rule of conditional entropy:
[3]: 17
The chain rule follows from the above definition of conditional entropy:
![{\displaystyle {\begin{aligned}\mathrm {H} (Y|X)&=\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log \left({\frac {p(x)}{p(x,y)}}\right)\\[4pt]&=\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)(\log(p(x))-\log(p(x,y)))\\[4pt]&=-\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}p(x,y)\log(p(x,y))+\sum _{x\in {\mathcal {X}},y\in {\mathcal {Y}}}{p(x,y)\log(p(x))}\\[4pt]&=\mathrm {H} (X,Y)+\sum _{x\in {\mathcal {X}}}p(x)\log(p(x))\\[4pt]&=\mathrm {H} (X,Y)-\mathrm {H} (X).\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/501bd3a915d2218c4464e1ea8cfefc3fba872320)
In general, a chain rule for multiple random variables holds:
[3]: 22
It has a similar form to chain rule in probability theory, except that addition instead of multiplication is used.
Bayes' rule[]
Bayes' rule for conditional entropy states
![{\displaystyle \mathrm {H} (Y|X)\,=\,\mathrm {H} (X|Y)-\mathrm {H} (X)+\mathrm {H} (Y).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/19d94407d6fcf7e76083457cd72c1c5efa049091)
Proof.
and
. Symmetry entails
. Subtracting the two equations implies Bayes' rule.
If
is conditionally independent of
given
we have:
![{\displaystyle \mathrm {H} (Y|X,Z)\,=\,\mathrm {H} (Y|X).}](https://wikimedia.org/api/rest_v1/media/math/render/svg/a67d62362d7e4d280e7dfc125527704259b10cad)
Other properties[]
For any
and
:
![{\displaystyle {\begin{aligned}\mathrm {H} (Y|X)&\leq \mathrm {H} (Y)\,\\\mathrm {H} (X,Y)&=\mathrm {H} (X|Y)+\mathrm {H} (Y|X)+\operatorname {I} (X;Y),\qquad \\\mathrm {H} (X,Y)&=\mathrm {H} (X)+\mathrm {H} (Y)-\operatorname {I} (X;Y),\,\\\operatorname {I} (X;Y)&\leq \mathrm {H} (X),\,\end{aligned}}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/b7c4b5411ed661515a9105888c14547b01afa021)
where
is the mutual information between
and
.
For independent
and
:
and ![{\displaystyle \mathrm {H} (X|Y)=\mathrm {H} (X)\,}](https://wikimedia.org/api/rest_v1/media/math/render/svg/65d6beb9a32406a7945d075cef6c769e9a37d02d)
Although the specific-conditional entropy
can be either less or greater than
for a given random variate
of
,
can never exceed
.
Conditional differential entropy[]
Definition[]
The above definition is for discrete random variables. The continuous version of discrete conditional entropy is called conditional differential (or continuous) entropy. Let
and
be a continuous random variables with a joint probability density function
. The differential conditional entropy
is defined as[3]: 249
![{\displaystyle h(X|Y)=-\int _{{\mathcal {X}},{\mathcal {Y}}}f(x,y)\log f(x|y)\,dxdy}](https://wikimedia.org/api/rest_v1/media/math/render/svg/126f1c66453902b42bbb9b16d5ddb62a0e591572)
|
|
(Eq.2)
|
Properties[]
In contrast to the conditional entropy for discrete random variables, the conditional differential entropy may be negative.
As in the discrete case there is a chain rule for differential entropy:
[3]: 253
Notice however that this rule may not be true if the involved differential entropies do not exist or are infinite.
Joint differential entropy is also used in the definition of the mutual information between continuous random variables:
![{\displaystyle \operatorname {I} (X,Y)=h(X)-h(X|Y)=h(Y)-h(Y|X)}](https://wikimedia.org/api/rest_v1/media/math/render/svg/c9b2ce6fa8dfeedc2c90ad4866fbc6af713e83f3)
with equality if and only if
and
are independent.[3]: 253
Relation to estimator error[]
The conditional differential entropy yields a lower bound on the expected squared error of an estimator. For any random variable
, observation
and estimator
the following holds:[3]: 255
![{\displaystyle \mathbb {E} \left[{\bigl (}X-{\widehat {X}}{(Y)}{\bigr )}^{2}\right]\geq {\frac {1}{2\pi e}}e^{2h(X|Y)}}](https://wikimedia.org/api/rest_v1/media/math/render/svg/ab916a1ac9b14193bf90b79742772b686bb771c3)
This is related to the uncertainty principle from quantum mechanics.
Generalization to quantum theory[]
In quantum information theory, the conditional entropy is generalized to the conditional quantum entropy. The latter can take negative values, unlike its classical counterpart.
See also[]
References[]