Kullback–Leibler divergence

In mathematical statistics, the Kullback–Leibler divergence, ${\displaystyle D_{\text{KL}}}$ (also called relative entropy), is a measure of how one probability distribution is different from a second, reference probability distribution.^[1][2] Applications include characterizing the relative (Shannon) entropy in information systems, randomness in continuous time-series, and information gain when comparing statistical models of inference. In contrast to variation of information, it is a distribution-wise asymmetric measure and thus does not qualify as a statistical metric of spread – it also does not satisfy the triangle inequality. In the simple case, a relative entropy of 0 indicates that the two distributions in question have identical quantities of information. In simplified terms, it is a measure of surprise, with diverse applications such as applied statistics, fluid mechanics, neuroscience and bioinformatics.

Introduction and context[]

Consider two probability distributions ${\displaystyle P}$ and ${\displaystyle Q}$. Usually, ${\displaystyle P}$ represents the data, the observations, or a measured probability distribution. Distribution ${\displaystyle Q}$ represents instead a theory, a model, a description or an approximation of ${\displaystyle P}$. The Kullback–Leibler divergence is then interpreted as the average difference of the number of bits required for encoding samples of ${\displaystyle P}$ using a code optimized for ${\displaystyle Q}$ rather than one optimized for ${\displaystyle P}$. Note that the roles of ${\displaystyle P}$ and ${\displaystyle Q}$ can be reversed in some situations where that is easier to compute, such as with the Expectation–maximization (EM) algorithm and Evidence lower bound (ELBO) computations.

Etymology[]

The relative entropy was introduced by Solomon Kullback and Richard Leibler in 1951 as the directed divergence between two distributions; Kullback preferred the term discrimination information.^[3] The divergence is discussed in Kullback's 1959 book, Information Theory and Statistics.^[2]

Definition[]

For discrete probability distributions ${\displaystyle P}$ and ${\displaystyle Q}$ defined on the same probability space, ${\displaystyle {\mathcal {X}}}$, the relative entropy from ${\displaystyle Q}$ to ${\displaystyle P}$ is defined^[4] to be

${\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\log \left({\frac {P(x)}{Q(x)}}\right).}$

which is equivalent to

${\displaystyle D_{\text{KL}}(P\parallel Q)=-\sum _{x\in {\mathcal {X}}}P(x)\log \left({\frac {Q(x)}{P(x)}}\right)}$

In other words, it is the expectation of the logarithmic difference between the probabilities ${\displaystyle P}$ and ${\displaystyle Q}$, where the expectation is taken using the probabilities ${\displaystyle P}$. Relative entropy is defined only if for all ${\displaystyle x}$, ${\displaystyle Q(x)=0}$ implies ${\displaystyle P(x)=0}$ (absolute continuity). Whenever ${\displaystyle P(x)}$ is zero the contribution of the corresponding term is interpreted as zero because

${\displaystyle \lim _{x\to 0^{+}}x\log(x)=0.}$

For distributions ${\displaystyle P}$ and ${\displaystyle Q}$ of a continuous random variable, relative entropy is defined to be the integral:^{[5]: p. 55}

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{-\infty }^{\infty }p(x)\log \left({\frac {p(x)}{q(x)}}\right)\,dx}$

where ${\displaystyle p}$ and ${\displaystyle q}$ denote the probability densities of ${\displaystyle P}$ and ${\displaystyle Q}$.

More generally, if ${\displaystyle P}$ and ${\displaystyle Q}$ are probability measures over a set ${\displaystyle {\mathcal {X}}}$, and ${\displaystyle P}$ is absolutely continuous with respect to ${\displaystyle Q}$, then the relative entropy from ${\displaystyle Q}$ to ${\displaystyle P}$ is defined as

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{\mathcal {X}}\log \left({\frac {dP}{dQ}}\right)\,dP,}$

where ${\displaystyle {\frac {dP}{dQ}}}$ is the Radon–Nikodym derivative of ${\displaystyle P}$ with respect to ${\displaystyle Q}$, and provided the expression on the right-hand side exists. Equivalently (by the chain rule), this can be written as

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{\mathcal {X}}\log \left({\frac {dP}{dQ}}\right){\frac {dP}{dQ}}\,dQ,}$

which is the entropy of ${\displaystyle Q}$ relative to ${\displaystyle P}$. Continuing in this case, if ${\displaystyle \mu }$ is any measure on ${\displaystyle {\mathcal {X}}}$ for which ${\displaystyle p={\frac {dP}{d\mu }}}$ and ${\displaystyle q={\frac {dQ}{d\mu }}}$ exist (meaning that ${\displaystyle p}$ and ${\displaystyle q}$ are absolutely continuous with respect to ${\displaystyle \mu }$), then the relative entropy from ${\displaystyle Q}$ to ${\displaystyle P}$ is given as

${\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{\mathcal {X}}p\log \left({\frac {p}{q}}\right)\,d\mu .}$

The logarithms in these formulae are taken to base 2 if information is measured in units of bits, or to base ${\displaystyle e}$ if information is measured in nats. Most formulas involving relative entropy hold regardless of the base of the logarithm.

Various conventions exist for referring to ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ in words. Often it is referred to as the divergence between ${\displaystyle P}$ and ${\displaystyle Q}$, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of ${\displaystyle P}$ from ${\displaystyle Q}$ or as the divergence from ${\displaystyle Q}$ to ${\displaystyle P}$. This reflects the asymmetry in Bayesian inference, which starts from a prior ${\displaystyle Q}$ and updates to the posterior ${\displaystyle P}$. Another common way to refer to ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is as the relative entropy of ${\displaystyle P}$ with respect to ${\displaystyle Q}$.

Basic example[]

Kullback^[2] gives the following example (Table 2.1, Example 2.1). Let ${\displaystyle P}$ and ${\displaystyle Q}$ be the distributions shown in the table and figure. ${\displaystyle P}$ is the distribution on the left side of the figure, a binomial distribution with ${\displaystyle N=2}$ and ${\displaystyle p=0.4}$. ${\displaystyle Q}$ is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes ${\displaystyle x=0}$, ${\displaystyle 1}$, or ${\displaystyle 2}$ (i.e. ${\displaystyle {\mathcal {X}}=\{0,1,2\}}$), each with probability ${\displaystyle p=1/3}$.

x	0	1	2
Distribution P(x)	${\displaystyle 9/25}$	${\displaystyle 12/25}$	${\displaystyle 4/25}$
Distribution Q(x)	${\displaystyle 1/3}$	${\displaystyle 1/3}$	${\displaystyle 1/3}$

Relative entropies ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ and ${\displaystyle D_{\text{KL}}(Q\parallel P)}$ are calculated as follows. This example uses the natural log with base e, designated ${\displaystyle \operatorname {ln} }$ to get results in nats (see units of information).

${\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}P(x)\ln \left({\frac {P(x)}{Q(x)}}\right)\\&={\frac {9}{25}}\ln \left({\frac {9/25}{1/3}}\right)+{\frac {12}{25}}\ln \left({\frac {12/25}{1/3}}\right)+{\frac {4}{25}}\ln \left({\frac {4/25}{1/3}}\right)\\&={\frac {1}{25}}\left(32\ln(2)+55\ln(3)-50\ln(5)\right)\approx 0.0852996\end{aligned}}}$

${\displaystyle {\begin{aligned}D_{\text{KL}}(Q\parallel P)&=\sum _{x\in {\mathcal {X}}}Q(x)\ln \left({\frac {Q(x)}{P(x)}}\right)\\&={\frac {1}{3}}\ln \left({\frac {1/3}{9/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{12/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{4/25}}\right)\\&={\frac {1}{3}}\left(-4\ln(2)-6\ln(3)+6\ln(5)\right)\approx 0.097455\end{aligned}}}$

Interpretations[]

The relative entropy from ${\displaystyle Q}$ to ${\displaystyle P}$ is often denoted ${\displaystyle D_{\text{KL}}(P\parallel Q)}$.

In the context of machine learning, ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is often called the information gain achieved if ${\displaystyle P}$ would be used instead of ${\displaystyle Q}$ which is currently used. By analogy with information theory, it is called the relative entropy of ${\displaystyle P}$ with respect to ${\displaystyle Q}$. In the context of coding theory, ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be constructed by measuring the expected number of extra bits required to code samples from ${\displaystyle P}$ using a code optimized for ${\displaystyle Q}$ rather than the code optimized for ${\displaystyle P}$.

Expressed in the language of Bayesian inference, ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is a measure of the information gained by revising one's beliefs from the prior probability distribution ${\displaystyle Q}$ to the posterior probability distribution ${\displaystyle P}$. In other words, it is the amount of information lost when ${\displaystyle Q}$ is used to approximate ${\displaystyle P}$.^[6] In applications, ${\displaystyle P}$ typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while ${\displaystyle Q}$ typically represents a theory, model, description, or approximation of ${\displaystyle P}$. In order to find a distribution ${\displaystyle Q}$ that is closest to ${\displaystyle P}$, we can minimize KL divergence and compute an information projection.

Relative entropy is a special case of a broader class of statistical divergences called f-divergences as well as the class of Bregman divergences. It is the only such divergence over probabilities that is a member of both classes. Although it is often intuited as a way of measuring the distance between probability distributions, the Kullback–Leibler divergence is not a true metric. It does not obey the Triangle Inequality, and in general ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ does not equal ${\displaystyle D_{\text{KL}}(Q\parallel P)}$. However, its infinitesimal form, specifically its Hessian, gives a metric tensor known as the Fisher information metric.

Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used characterization of entropy.^[7] Consequently, mutual information is the only measure of mutual dependence that obeys certain related conditions, since it can be defined in terms of Kullback–Leibler divergence.

Motivation[]

In information theory, the Kraft–McMillan theorem establishes that any directly decodable coding scheme for coding a message to identify one value ${\displaystyle x_{i}}$ out of a set of possibilities ${\displaystyle X}$ can be seen as representing an implicit probability distribution ${\displaystyle q(x_{i})=2^{-\ell _{i}}}$ over ${\displaystyle X}$, where ${\displaystyle \ell _{i}}$ is the length of the code for ${\displaystyle x_{i}}$ in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution ${\displaystyle Q}$ is used, compared to using a code based on the true distribution ${\displaystyle P}$.

${\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=-\sum _{x\in {\mathcal {X}}}p(x)\log q(x)+\sum _{x\in {\mathcal {X}}}p(x)\log p(x)\\&=\mathrm {H} (P,Q)-\mathrm {H} (P)\end{aligned}}}$

where ${\displaystyle \mathrm {H} (P,Q)}$ is the cross entropy of ${\displaystyle P}$ and ${\displaystyle Q}$, and ${\displaystyle \mathrm {H} (P)}$ is the entropy of ${\displaystyle P}$ (which is the same as the cross-entropy of P with itself).

Relative entropy ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be thought of as something like a measurement of how far the distribution Q is from the distribution P. The cross-entropy ${\displaystyle H(P,Q)}$ is itself such a measurement, but it has the defect that ${\displaystyle H(P,P)=:H(P)}$ isn't zero, so we subtract ${\displaystyle H(P)}$ to make ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ agree more closely with our notion of distance. (Unfortunately it still isn't symmetric.) Relative entropy relates to "rate function" in the theory of large deviations.^[8][9]

Properties[]

• Relative entropy is always non-negative,

${\displaystyle D_{\text{KL}}(P\parallel Q)\geq 0,}$

a result known as Gibbs' inequality, with ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ equals zero if and only if ${\displaystyle P=Q}$ almost everywhere. The entropy ${\displaystyle \mathrm {H} (P)}$ thus sets a minimum value for the cross-entropy ${\displaystyle \mathrm {H} (P,Q)}$, the expected number of bits required when using a code based on ${\displaystyle Q}$ rather than ${\displaystyle P}$; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value ${\displaystyle x}$ drawn from ${\displaystyle X}$, if a code is used corresponding to the probability distribution ${\displaystyle Q}$, rather than the "true" distribution ${\displaystyle P}$.

• no upper-bound exists for the general case. However, it is shown that if ${\displaystyle P}$ and ${\displaystyle Q}$ are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ can be calculated.^[10]

• Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under parameter transformations. For example, if a transformation is made from variable ${\displaystyle x}$ to variable ${\displaystyle y(x)}$, then, since ${\displaystyle P(x)dx=P(y)dy}$ and ${\displaystyle Q(x)dx=Q(y)dy}$ the relative entropy may be rewritten:

${\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\int _{x_{a}}^{x_{b}}P(x)\log \left({\frac {P(x)}{Q(x)}}\right)\,dx\\[6pt]&=\int _{y_{a}}^{y_{b}}P(y)\log \left({\frac {P(y)\,{\frac {dy}{dx}}}{Q(y)\,{\frac {dy}{dx}}}}\right)\,dy=\int _{y_{a}}^{y_{b}}P(y)\log \left({\frac {P(y)}{Q(y)}}\right)\,dy\end{aligned}}}$

where ${\displaystyle y_{a}=y(x_{a})}$ and ${\displaystyle y_{b}=y(x_{b})}$. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a dimensionally consistent quantity, since if ${\displaystyle x}$ is a dimensioned variable, ${\displaystyle P(x)}$ and ${\displaystyle Q(x)}$ are also dimensioned, since e.g. ${\displaystyle P(x)dx}$ is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory^[11] (such as self-information or Shannon entropy), which can become undefined or negative for non-discrete probabilities.

• Relative entropy is additive for independent distributions in much the same way as Shannon entropy. If ${\displaystyle P_{1},P_{2}}$ are independent distributions, with the joint distribution ${\displaystyle P(x,y)=P_{1}(x)P_{2}(y)}$, and ${\displaystyle Q,Q_{1},Q_{2}}$ likewise, then

${\displaystyle D_{\text{KL}}(P\parallel Q)=D_{\text{KL}}(P_{1}\parallel Q_{1})+D_{\text{KL}}(P_{2}\parallel Q_{2}).}$

• Relative entropy ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ is convex in the pair of probability mass functions ${\displaystyle (p,q)}$, i.e. if ${\displaystyle (p_{1},q_{1})}$ and ${\displaystyle (p_{2},q_{2})}$ are two pairs of probability mass functions, then

  ${\displaystyle D_{\text{KL}}(\lambda p_{1}+(1-\lambda )p_{2}\parallel \lambda q_{1}+(1-\lambda )q_{2})\leq \lambda D_{\text{KL}}(p_{1}\parallel q_{1})+(1-\lambda )D_{\text{KL}}(p_{2}\parallel q_{2}){\text{ for }}0\leq \lambda \leq 1.}$

• The following theorem is referred to as a duality formula for variational inference. It explains some important properties of the variational distributions used in variational Bayesian methods.^[12]

Theorem Consider two probability spaces ${\displaystyle (\Theta ,{\mathcal {F}},P)}$ and ${\displaystyle (\Theta ,{\mathcal {F}},Q)}$ with ${\displaystyle Q\ll P}$. Assume that there is a common dominating probability measure ${\displaystyle \lambda }$ such that ${\displaystyle P\ll \lambda }$ and ${\displaystyle Q\ll \lambda }$. Let ${\displaystyle h}$ denote any real-valued random variable on ${\displaystyle (\Theta ,{\mathcal {F}},P)}$ that satisfies ${\displaystyle h\in L_{1}(P)}$. Then the following equality holds

${\displaystyle \log E_{P}[\exp h]={\text{sup}}_{Q\ll P}\{E_{Q}[h]-D_{\text{KL}}(Q\parallel P)\}.}$

Further, the supremum on the right-hand side is attained if and only if it holds

${\displaystyle {\frac {q(\theta )}{p(\theta )}}={\frac {\exp h(\theta )}{E_{P}[\exp h]}},}$

almost surely with respect to probability measure ${\displaystyle Q}$, where ${\displaystyle p(\theta )=dP/d\lambda }$ and ${\displaystyle q(\theta )=dQ/d\lambda }$ denote the Radon-Nikodym derivatives of the probability measures ${\displaystyle P}$ and ${\displaystyle Q}$ with respect to ${\displaystyle \lambda }$, respectively.

Examples[]

Multivariate normal distributions[]

Suppose that we have two multivariate normal distributions, with means ${\displaystyle \mu _{0},\mu _{1}}$ and with (non-singular) covariance matrices ${\displaystyle \Sigma _{0},\Sigma _{1}.}$ If the two distributions have the same dimension, ${\displaystyle k}$, then the relative entropy between the distributions is as follows:^{[13]: p. 13}

${\displaystyle D_{\text{KL}}\left({\mathcal {N}}_{0}\parallel {\mathcal {N}}_{1}\right)={\frac {1}{2}}\left(\operatorname {tr} \left(\Sigma _{1}^{-1}\Sigma _{0}\right)+\left(\mu _{1}-\mu _{0}\right)^{\mathsf {T}}\Sigma _{1}^{-1}\left(\mu _{1}-\mu _{0}\right)-k+\ln \left({\frac {\det \Sigma _{1}}{\det \Sigma _{0}}}\right)\right).}$

The logarithm in the last term must be taken to base e since all terms apart from the last are base-e logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in nats. Dividing the entire expression above by ${\displaystyle \ln(2)}$ yields the divergence in bits.

A special case, and a common quantity in variational inference, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):

${\displaystyle D_{\text{KL}}\left({\mathcal {N}}\left(\left(\mu _{1},\ldots ,\mu _{k}\right)^{\mathsf {T}},\operatorname {diag} \left(\sigma _{1}^{2},\ldots ,\sigma _{k}^{2}\right)\right)\parallel {\mathcal {N}}\left(\mathbf {0} ,\mathbf {I} \right)\right)={1 \over 2}\sum _{i=1}^{k}\left(\sigma _{i}^{2}+\mu _{i}^{2}-1-\ln \left(\sigma _{i}^{2}\right)\right).}$

Relation to metrics[]

One might be tempted to call relative entropy a "distance metric" on the space of probability distributions, but this would not be correct as it is not symmetric – that is, ${\displaystyle D_{\text{KL}}(P\parallel Q)\neq D_{\text{KL}}(Q\parallel P)}$ – nor does it satisfy the triangle inequality. It generates a topology on the space of probability distributions. More concretely, if ${\displaystyle \{P_{1},P_{2},\ldots \}}$ is a sequence of distributions such that

${\displaystyle \lim _{n\to \infty }D_{\text{KL}}(P_{n}\parallel Q)=0}$

then it is said that

${\displaystyle P_{n}{\xrightarrow {D}}Q.}$

Pinsker's inequality entails that

${\displaystyle P_{n}{\xrightarrow {D}}P\Rightarrow P_{n}{\xrightarrow {TV}}P,}$

where the latter stands for the usual convergence in total variation.

Fisher information metric[]

Relative entropy is directly related to the Fisher information metric. This can be made explicit as follows. Assume that the probability distributions ${\displaystyle P}$ and ${\displaystyle Q}$ are both parameterized by some (possibly multi-dimensional) parameter ${\displaystyle \theta }$. Consider then two close by values of ${\displaystyle P=P(\theta )}$ and ${\displaystyle Q=P(\theta _{0})}$ so that the parameter ${\displaystyle \theta }$ differs by only a small amount from the parameter value ${\displaystyle \theta _{0}}$. Specifically, up to first order one has (using the Einstein summation convention)

${\displaystyle P(\theta )=P(\theta _{0})+\Delta \theta _{j}P_{j}(\theta _{0})+\cdots }$

with ${\displaystyle \Delta \theta _{j}=(\theta -\theta _{0})_{j}}$ a small change of ${\displaystyle \theta }$ in the ${\displaystyle j}$ direction, and ${\displaystyle P_{j}\left(\theta _{0}\right)={\frac {\partial P}{\partial \theta _{j}}}(\theta _{0})}$ the corresponding rate of change in the probability distribution. Since relative entropy has an absolute minimum 0 for ${\displaystyle P=Q}$, i.e. ${\displaystyle \theta =\theta _{0}}$, it changes only to second order in the small parameters ${\displaystyle \Delta \theta _{j}}$. More formally, as for any minimum, the first derivatives of the divergence vanish

${\displaystyle \left.{\frac {\partial }{\partial \theta _{j}}}\right|_{\theta =\theta _{0}}D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))=0,}$

and by the Taylor expansion one has up to second order

${\displaystyle D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))={\frac {1}{2}}\Delta \theta _{j}\Delta \theta _{k}g_{jk}(\theta _{0})+\cdots }$

where the Hessian matrix of the divergence

${\displaystyle g_{jk}(\theta _{0})=\left.{\frac {\partial ^{2}}{\partial \theta _{j}\,\partial \theta _{k}}}\right|_{\theta =\theta _{0}}D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))}$

must be positive semidefinite. Letting ${\displaystyle \theta _{0}}$ vary (and dropping the subindex 0) the Hessian ${\displaystyle g_{jk}(\theta )}$ defines a (possibly degenerate) Riemannian metric on the θ parameter space, called the Fisher information metric.

Fisher information metric theorem[]

When ${\displaystyle p_{(x,\rho )}}$ satisfies the following regularity conditions:

${\displaystyle {\frac {\partial \log(p)}{\partial \rho }},{\frac {\partial ^{2}\log(p)}{\partial \rho ^{2}}},{\frac {\partial ^{3}\log(p)}{\partial \rho ^{3}}}}$ exist,

${\displaystyle {\begin{aligned}\left|{\frac {\partial p}{\partial \rho }}\right|&<F(x):\int _{x=0}^{\infty }F(x)\,dx<\infty ,\\\left|{\frac {\partial ^{2}p}{\partial \rho ^{2}}}\right|&<G(x):\int _{x=0}^{\infty }G(x)\,dx<\infty \\\left|{\frac {\partial ^{3}\log(p)}{\partial \rho ^{3}}}\right|&<H(x):\int _{x=0}^{\infty }p(x,0)H(x)\,dx<\xi <\infty \end{aligned}}}$

where ξ is independent of ρ

${\displaystyle \left.\int _{x=0}^{\infty }{\frac {\partial p(x,\rho )}{\partial \rho }}\right|_{\rho =0}\,dx=\left.\int _{x=0}^{\infty }{\frac {\partial ^{2}p(x,\rho )}{\partial \rho ^{2}}}\right|_{\rho =0}\,dx=0}$

then:

${\displaystyle {\mathcal {D}}(p(x,0)\parallel p(x,\rho ))={\frac {c\rho ^{2}}{2}}+{\mathcal {O}}\left(\rho ^{3}\right){\text{ as }}\rho \to 0.}$

Variation of information[]

Another information-theoretic metric is Variation of information, which is roughly a symmetrization of conditional entropy. It is a metric on the set of partitions of a discrete probability space.

Relation to other quantities of information theory[]

Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.

Self-information[]

The self-information, also known as the information content of a signal, random variable, or event is defined as the negative logarithm of the probability of the given outcome occurring.

When applied to a discrete random variable, the self-information can be represented as^{[citation needed]}

${\displaystyle \operatorname {\operatorname {I} } (m)=D_{\text{KL}}\left(\delta _{\text{im}}\parallel \{p_{i}\}\right),}$

is the relative entropy of the probability distribution ${\displaystyle P(i)}$ from a Kronecker delta representing certainty that ${\displaystyle i=m}$ — i.e. the number of extra bits that must be transmitted to identify ${\displaystyle i}$ if only the probability distribution ${\displaystyle P(i)}$ is available to the receiver, not the fact that ${\displaystyle i=m}$.

Mutual information[]

The mutual information,

${\displaystyle {\begin{aligned}\operatorname {I} (X;Y)&=D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))\\&=\operatorname {E} _{X}\{D_{\text{KL}}(P(Y\mid X)\parallel P(Y))\}\\&=\operatorname {E} _{Y}\{D_{\text{KL}}(P(X\mid Y)\parallel P(X))\}\end{aligned}}}$

is the relative entropy of the product ${\displaystyle P(X)P(Y)}$ of the two marginal probability distributions from the joint probability distribution ${\displaystyle P(X,Y)}$ — i.e. the expected number of extra bits that must be transmitted to identify ${\displaystyle X}$ and ${\displaystyle Y}$ if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability ${\displaystyle P(X,Y)}$ is known, it is the expected number of extra bits that must on average be sent to identify ${\displaystyle Y}$ if the value of ${\displaystyle X}$ is not already known to the receiver.

Shannon entropy[]

The Shannon entropy,

${\displaystyle {\begin{aligned}\mathrm {H} (X)&=\operatorname {E} \left[\operatorname {I} _{X}(x)\right]\\&=\log(N)-D_{\text{KL}}\left(p_{X}(x)\parallel P_{U}(X)\right)\end{aligned}}}$

is the number of bits which would have to be transmitted to identify ${\displaystyle X}$ from ${\displaystyle N}$ equally likely possibilities, less the relative entropy of the uniform distribution on the random variates of ${\displaystyle X}$, ${\displaystyle P_{U}(X)}$, from the true distribution ${\displaystyle P(X)}$ — i.e. less the expected number of bits saved, which would have had to be sent if the value of ${\displaystyle X}$ were coded according to the uniform distribution ${\displaystyle P_{U}(X)}$ rather than the true distribution ${\displaystyle P(X)}$.

Conditional entropy[]

The conditional entropy^[14],

${\displaystyle {\begin{aligned}\mathrm {H} (X\mid Y)&=\log(N)-D_{\text{KL}}(P(X,Y)\parallel P_{U}(X)P(Y))\\&=\log(N)-D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))-D_{\text{KL}}(P(X)\parallel P_{U}(X))\\&=\mathrm {H} (X)-\operatorname {I} (X;Y)\\&=\log(N)-\operatorname {E} _{Y}\left[D_{\text{KL}}\left(P\left(X\mid Y\right)\parallel P_{U}(X)\right)\right]\end{aligned}}}$

is the number of bits which would have to be transmitted to identify ${\displaystyle X}$ from ${\displaystyle N}$ equally likely possibilities, less the relative entropy of the product distribution ${\displaystyle P_{U}(X)P(Y)}$ from the true joint distribution ${\displaystyle P(X,Y)}$ — i.e. less the expected number of bits saved which would have had to be sent if the value of ${\displaystyle X}$ were coded according to the uniform distribution ${\displaystyle P_{U}(X)}$ rather than the conditional distribution ${\displaystyle P(X|Y)}$ of ${\displaystyle X}$ given ${\displaystyle Y}$.

Cross entropy[]

When we have a set of possible events, coming from the distribution p, we can encode them (with a lossless data compression) using entropy encoding. This compresses the data by replacing each fixed-length input symbol with a corresponding unique, variable-length, prefix-free code (e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be encoded as the bits (0, 10, 11)). If we know the distribution p in advance, we can devise an encoding that would be optimal (e.g.: using Huffman coding). Meaning the messages we encode will have the shortest length on average (assuming the encoded events are sampled from p), which will be equal to Shannon's Entropy of p (denoted as ${\displaystyle \mathrm {H} (p)}$). However, if we use a different probability distribution (q) when creating the entropy encoding scheme, then a larger number of bits will be used (on average) to identify an event from a set of possibilities. This new (larger) number is measured by the cross entropy between p and q.

The cross entropy between two probability distributions (p and q) measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution q, rather than the "true" distribution p. The cross entropy for two distributions p and q over the same probability space is thus defined as follows.

${\displaystyle \mathrm {H} (p,q)=\operatorname {E} _{p}[-\log(q)]=\mathrm {H} (p)+D_{\text{KL}}(p\parallel q).}$

For explicit derivation of this, see the Motivation section above.

Under this scenario, relative entropies (kl-divergence) can be interpreted as the extra number of bits, on average, that are needed (beyond ${\displaystyle \mathrm {H} (p)}$) for encoding the events because of using q for constructing the encoding scheme instead of p.

Bayesian updating[]

In Bayesian statistics, relative entropy can be used as a measure of the information gain in moving from a prior distribution to a posterior distribution: ${\displaystyle p(x)\to p(x\mid I)}$. If some new fact ${\displaystyle Y=y}$ is discovered, it can be used to update the posterior distribution for ${\displaystyle X}$ from ${\displaystyle p(x\mid I)}$ to a new posterior distribution ${\displaystyle p(x\mid y,I)}$ using Bayes' theorem:

${\displaystyle p(x\mid y,I)={\frac {p(y\mid x,I)p(x\mid I)}{p(y\mid I)}}}$

This distribution has a new entropy:

${\displaystyle \mathrm {H} {\big (}p(x\mid y,I){\big )}=-\sum _{x}p(x\mid y,I)\log p(x\mid y,I),}$

which may be less than or greater than the original entropy ${\displaystyle \mathrm {H} (p(x\mid I))}$. However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on ${\displaystyle p(x\mid I)}$ instead of a new code based on ${\displaystyle p(x\mid y,I)}$ would have added an expected number of bits:

${\displaystyle D_{\text{KL}}{\big (}p(x\mid y,I)\parallel p(x\mid I){\big )}=\sum _{x}p(x\mid y,I)\log \left({\frac {p(x\mid y,I)}{p(x\mid I)}}\right)}$

to the message length. This therefore represents the amount of useful information, or information gain, about ${\displaystyle X}$, that has been learned by discovering ${\displaystyle Y=y}$.

If a further piece of data, ${\displaystyle Y_{2}=y_{2}}$, subsequently comes in, the probability distribution for ${\displaystyle x}$ can be updated further, to give a new best guess ${\displaystyle p(x\mid y_{1},y_{2},I)}$. If one reinvestigates the information gain for using ${\displaystyle p(x\mid y_{1},I)}$ rather than ${\displaystyle p(x\mid I)}$, it turns out that it may be either greater or less than previously estimated:

${\displaystyle \sum _{x}p(x\mid y_{1},y_{2},I)\log \left({\frac {p(x\mid y_{1},y_{2},I)}{p(x\mid I)}}\right)}$ may be ≤ or > than ${\displaystyle \displaystyle \sum _{x}p(x\mid y_{1},I)\log \left({\frac {p(x\mid y_{1},I)}{p(x\mid I)}}\right)}$

and so the combined information gain does not obey the triangle inequality:

${\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid I){\big )}}$ may be <, = or > than ${\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid y_{1},I){\big )}+D_{\text{KL}}{\big (}p(x\mid y_{1},I)\parallel p(x\mid I){\big )}}$

All one can say is that on average, averaging using ${\displaystyle p(y_{2}\mid y_{1},x,I)}$, the two sides will average out.

Bayesian experimental design[]

A common goal in Bayesian experimental design is to maximise the expected relative entropy between the prior and the posterior.^[15] When posteriors are approximated to be Gaussian distributions, a design maximising the expected relative entropy is called Bayes d-optimal.

Discrimination information[]

Relative entropy ${\textstyle D_{\text{KL}}{\bigl (}p(x\mid H_{1})\parallel p(x\mid H_{0}){\bigr )}}$ can also be interpreted as the expected discrimination information for ${\displaystyle H_{1}}$ over ${\displaystyle H_{0}}$: the mean information per sample for discriminating in favor of a hypothesis ${\displaystyle H_{1}}$ against a hypothesis ${\displaystyle H_{0}}$, when hypothesis ${\displaystyle H_{1}}$ is true.^[16] Another name for this quantity, given to it by I. J. Good, is the expected weight of evidence for ${\displaystyle H_{1}}$ over ${\displaystyle H_{0}}$ to be expected from each sample.

The expected weight of evidence for ${\displaystyle H_{1}}$ over ${\displaystyle H_{0}}$ is not the same as the information gain expected per sample about the probability distribution ${\displaystyle p(H)}$ of the hypotheses,

${\displaystyle D_{\text{KL}}(p(x\mid H_{1})\parallel p(x\mid H_{0}))\neq IG=D_{\text{KL}}(p(H\mid x)\parallel p(H\mid I)).}$

Either of the two quantities can be used as a utility function in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.

On the entropy scale of information gain there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the logit scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the Riemann hypothesis is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of loss function for uncertainty are both useful, according to how well each reflects the particular circumstances of the problem in question.

Principle of minimum discrimination information[]

The idea of relative entropy as discrimination information led Kullback to propose the Principle of Minimum Discrimination Information (MDI): given new facts, a new distribution ${\displaystyle f}$ should be chosen which is as hard to discriminate from the original distribution ${\displaystyle f_{0}}$ as possible; so that the new data produces as small an information gain ${\displaystyle D_{\text{KL}}(f\parallel f_{0})}$ as possible.

For example, if one had a prior distribution ${\displaystyle p(x,a)}$ over ${\displaystyle x}$ and ${\displaystyle a}$, and subsequently learnt the true distribution of ${\displaystyle a}$ was ${\displaystyle u(a)}$, then the relative entropy between the new joint distribution for ${\displaystyle x}$ and ${\displaystyle a}$, ${\displaystyle q(x\mid a)u(a)}$, and the earlier prior distribution would be:

${\displaystyle D_{\text{KL}}(q(x\mid a)u(a)\parallel p(x,a))=\operatorname {E} _{u(a)}\left\{D_{\text{KL}}(q(x\mid a)\parallel p(x\mid a))\right\}+D_{\text{KL}}(u(a)\parallel p(a)),}$

i.e. the sum of the relative entropy of ${\displaystyle p(a)}$ the prior distribution for ${\displaystyle a}$ from the updated distribution ${\displaystyle u(a)}$, plus the expected value (using the probability distribution ${\displaystyle u(a)}$) of the relative entropy of the prior conditional distribution ${\displaystyle p(x\mid a)}$ from the new conditional distribution ${\displaystyle q(x\mid a)}$. (Note that often the later expected value is called the conditional relative entropy (or conditional Kullback-Leibler divergence) and denoted by ${\displaystyle D_{\text{KL}}(q(x\mid a)\parallel p(x\mid a))}$ ^{[2][14]: p. 22}) This is minimized if ${\displaystyle q(x\mid a)=p(x\mid a)}$ over the whole support of ${\displaystyle u(a)}$; and we note that this result incorporates Bayes' theorem, if the new distribution ${\displaystyle u(a)}$ is in fact a δ function representing certainty that ${\displaystyle a}$ has one particular value.

MDI can be seen as an extension of Laplace's Principle of Insufficient Reason, and the Principle of Maximum Entropy of E.T. Jaynes. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see differential entropy), but the relative entropy continues to be just as relevant.

In the engineering literature, MDI is sometimes called the Principle of Minimum Cross-Entropy (MCE) or Minxent for short. Minimising relative entropy from ${\displaystyle m}$ to ${\displaystyle p}$ with respect to ${\displaystyle m}$ is equivalent to minimizing the cross-entropy of ${\displaystyle p}$ and ${\displaystyle m}$, since

${\displaystyle \mathrm {H} (p,m)=\mathrm {H} (p)+D_{\text{KL}}(p\parallel m),}$

which is appropriate if one is trying to choose an adequate approximation to ${\displaystyle p}$. However, this is just as often not the task one is trying to achieve. Instead, just as often it is ${\displaystyle m}$ that is some fixed prior reference measure, and ${\displaystyle p}$ that one is attempting to optimise by minimising ${\displaystyle D_{\text{KL}}(p\parallel m)}$ subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be ${\displaystyle D_{\text{KL}}(p\parallel m)}$, rather than ${\displaystyle \mathrm {H} (p,m)}$.

Relationship to available work[]

Surprisals^[17] add where probabilities multiply. The surprisal for an event of probability ${\displaystyle p}$ is defined as ${\displaystyle s=k\ln(1/p)}$. If ${\displaystyle k}$ is ${\displaystyle \left\{1,1/\ln 2,1.38\times 10^{-23}\right\}}$ then surprisal is in ${\displaystyle \{}$nats, bits, or ${\displaystyle J/K\}}$ so that, for instance, there are ${\displaystyle N}$ bits of surprisal for landing all "heads" on a toss of ${\displaystyle N}$ coins.

Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the average surprisal ${\displaystyle S}$ (entropy) for a given set of control parameters (like pressure ${\displaystyle P}$ or volume ${\displaystyle V}$). This constrained entropy maximization, both classically^[18] and quantum mechanically,^[19] minimizes Gibbs availability in entropy units^[20] ${\displaystyle A\equiv -k\ln(Z)}$ where ${\displaystyle Z}$ is a constrained multiplicity or partition function.

When temperature ${\displaystyle T}$ is fixed, free energy (${\displaystyle T\times A}$) is also minimized. Thus if ${\displaystyle T,V}$ and number of molecules ${\displaystyle N}$ are constant, the Helmholtz free energy ${\displaystyle F\equiv U-TS}$ (where ${\displaystyle U}$ is energy and ${\displaystyle S}$ is entropy) is minimized as a system "equilibrates." If ${\displaystyle T}$ and ${\displaystyle P}$ are held constant (say during processes in your body), the Gibbs free energy ${\displaystyle G=U+PV-TS}$ is minimized instead. The change in free energy under these conditions is a measure of available work that might be done in the process. Thus available work for an ideal gas at constant temperature ${\displaystyle T_{o}}$ and pressure ${\displaystyle P_{o}}$ is ${\displaystyle W=\Delta G=NkT_{o}\Theta (V/V_{o})}$ where ${\displaystyle V_{o}=NkT_{o}/P_{o}}$ and ${\displaystyle \Theta (x)=x-1-\ln x\geq 0}$ (see also Gibbs inequality).

More generally^[21] the work available relative to some ambient is obtained by multiplying ambient temperature ${\displaystyle T_{o}}$ by relative entropy or net surprisal ${\displaystyle \Delta I\geq 0,}$ defined as the average value of ${\displaystyle k\ln(p/p_{o})}$ where ${\displaystyle p_{o}}$ is the probability of a given state under ambient conditions. For instance, the work available in equilibrating a monatomic ideal gas to ambient values of ${\displaystyle V_{o}}$ and ${\displaystyle T_{o}}$ is thus ${\displaystyle W=T_{o}\Delta I}$, where relative entropy

${\displaystyle \Delta I=Nk\left[\Theta \left({\frac {V}{V_{o}}}\right)+{\frac {3}{2}}\Theta \left({\frac {T}{T_{o}}}\right)\right].}$

The resulting contours of constant relative entropy, shown at right for a mole of Argon at standard temperature and pressure, for example put limits on the conversion of hot to cold as in flame-powered air-conditioning or in the unpowered device to convert boiling-water to ice-water discussed here.^[22] Thus relative entropy measures thermodynamic availability in bits.

Quantum information theory[]

For density matrices ${\displaystyle P}$ and ${\displaystyle Q}$ on a Hilbert space, the quantum relative entropy from ${\displaystyle Q}$ to ${\displaystyle P}$ is defined to be

${\displaystyle D_{\text{KL}}(P\parallel Q)=\operatorname {Tr} (P(\log(P)-\log(Q))).}$

In quantum information science the minimum of ${\displaystyle D_{\text{KL}}(P\parallel Q)}$ over all separable states ${\displaystyle Q}$ can also be used as a measure of entanglement in the state ${\displaystyle P}$.

Relationship between models and reality[]

Just as relative entropy of "actual from ambient" measures thermodynamic availability, relative entropy of "reality from a model" is also useful even if the only clues we have about reality are some experimental measurements. In the former case relative entropy describes distance to equilibrium or (when multiplied by ambient temperature) the amount of available work, while in the latter case it tells you about surprises that reality has up its sleeve or, in other words, how much the model has yet to learn.

Although this tool for evaluating models against systems that are accessible experimentally may be applied in any field, its application to selecting a statistical model via Akaike information criterion are particularly well described in papers^[23] and a book^[24] by Burnham and Anderson. In a nutshell the relative entropy of reality from a model may be estimated, to within a constant additive term, by a function of the deviations observed between data and the model's predictions (like the mean squared deviation) . Estimates of such divergence for models that share the same additive term can in turn be used to select among models.

When trying to fit parametrized models to data there are various estimators which attempt to minimize relative entropy, such as maximum likelihood and maximum spacing estimators.^{[citation needed]}

Symmetrised divergence[]

Kullback and Leibler themselves actually defined the divergence as:

${\displaystyle D_{\text{KL}}(P\parallel Q)+D_{\text{KL}}(Q\parallel P)}$

which is symmetric and nonnegative. This quantity has sometimes been used for feature selection in classification problems, where ${\displaystyle P}$ and ${\displaystyle Q}$ are the conditional pdfs of a feature under two different classes. In the Banking and Finance industries, this quantity is referred to as Population Stability Index (PSI), and is used to assess distributional shifts in model features through time.

An alternative is given via the ${\displaystyle \lambda }$ divergence,

${\displaystyle D_{\lambda }(P\parallel Q)=\lambda D_{\text{KL}}(P\parallel \lambda P+(1-\lambda )Q)+(1-\lambda )D_{\text{KL}}(Q\parallel \lambda P+(1-\lambda )Q),}$

which can be interpreted as the expected information gain about ${\displaystyle X}$ from discovering which probability distribution ${\displaystyle X}$ is drawn from, ${\displaystyle P}$ or ${\displaystyle Q}$, if they currently have probabilities ${\displaystyle \lambda }$ and ${\displaystyle 1-\lambda }$ respectively.^{[clarification needed][citation needed]}

The value ${\displaystyle \lambda =0.5}$ gives the Jensen–Shannon divergence, defined by

${\displaystyle D_{\text{JS}}={\frac {1}{2}}D_{\text{KL}}(P\parallel M)+{\frac {1}{2}}D_{\text{KL}}(Q\parallel M)}$

where ${\displaystyle M}$ is the average of the two distributions,

${\displaystyle M={\frac {1}{2}}(P+Q).}$

${\displaystyle D_{JS}}$ can also be interpreted as the capacity of a noisy information channel with two inputs giving the output distributions ${\displaystyle P}$ and ${\displaystyle Q}$. The Jensen–Shannon divergence, like all f-divergences, is locally proportional to the Fisher information metric. It is similar to the Hellinger metric (in the sense that it induces the same affine connection on a statistical manifold).

Furthermore, the Jensen-Shannon divergence can be generalized using abstract statistical M-mixtures relying on an abstract mean M. ^{[25] [26]}

Relationship to other probability-distance measures[]

There are many other important measures of probability distance. Some of these are particularly connected with relative entropy. For example:

• The total variation distance, ${\displaystyle \delta (p,q)}$. This is connected to the divergence through Pinsker's inequality: ${\displaystyle \delta (P,Q)\leq {\sqrt {{\frac {1}{2}}D_{\text{KL}}(P\parallel Q)}}}$
• The family of Rényi divergences generalize relative entropy. Depending on the value of a certain parameter, ${\displaystyle \alpha }$, various inequalities may be deduced.

Other notable measures of distance include the Hellinger distance, histogram intersection, Chi-squared statistic, quadratic form distance, match distance, Kolmogorov–Smirnov distance, and earth mover's distance.^[27]

Data differencing[]

Just as absolute entropy serves as theoretical background for data compression, relative entropy serves as theoretical background for data differencing – the absolute entropy of a set of data in this sense being the data required to reconstruct it (minimum compressed size), while the relative entropy of a target set of data, given a source set of data, is the data required to reconstruct the target given the source (minimum size of a patch).

See also[]

References[]

External links[]

• Information Theoretical Estimators Toolbox
• Ruby gem for calculating Kullback–Leibler divergence
• Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory
• Matlab code for calculating Kullback–Leibler divergence for discrete distributions
• Sergio Verdú, Relative Entropy, NIPS 2009. One-hour video lecture.
• A modern summary of info-theoretic divergence measures