Evidence lower bound

In variational Bayesian methods, the evidence lower bound (often abbreviated ELBO, also sometimes called the variational lower bound^[1] or negative variational free energy) is a useful lower bound on the log-likelihood of some observed data.

Throughout this article, let ${\displaystyle \mathbf {X} }$ and ${\displaystyle \mathbf {Z} }$ be multivariate random variables, jointly-distributed with distribution ${\displaystyle P}$. So, for example ${\displaystyle P(\mathbf {X} )}$ is the marginal distribution of ${\displaystyle \mathbf {X} }$, and ${\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}$ is the conditional distribution of ${\displaystyle \mathbf {Z} }$ given ${\displaystyle \mathbf {X} }$. Then, for any distribution ${\displaystyle Q(\mathbf {Z} )}$, we have^[1]

The right-hand side of this inequality is called the evidence lower bound, or ELBO. We refer to the above inequality as the ELBO inequality.

The ELBO comes up frequently in variational Bayesian methods. In that context, the random variable ${\displaystyle \mathbf {X} }$ conceptually represents observable data, the variable ${\displaystyle \mathbf {Z} }$ represents latent, unobservable data, and ${\displaystyle P}$ represents the true joint distribution of ${\displaystyle \mathbf {X} }$ and ${\displaystyle \mathbf {Z} }$. We often wish to find an approximation of the true posterior distribution ${\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}$ via a simpler, usually parametric, distribution, and this is what ${\displaystyle Q(\mathbf {Z} )}$ conceptually represents. Finding ${\displaystyle Q}$ can be framed as an optimization problem, and in that context the ELBO inequality is often used to obtain an optimization objective.

Terminology and notation[]

In the terminology of variational Bayesian methods, the distribution ${\displaystyle P(\mathbf {X} )}$ is called the evidence. Because the log function is monotonic, the ELBO inequality can be rewritten to give a lower bound for the evidence, as

hence the name evidence lower bound. Some authors use the term evidence to mean ${\displaystyle \log P(\mathbf {X} )}$, in which case the original form of the inequality already gives a lower bound for the evidence. Some authors call ${\displaystyle \log P(\mathbf {X} )}$ the log-evidence, and some use the terms evidence and log-evidence interchangeably to refer either to ${\displaystyle P(\mathbf {X} )}$ or to ${\displaystyle \log P(\mathbf {X} )}$.

The ELBO is sometimes denoted ${\displaystyle L}$, ${\displaystyle L(Q)}$, or ${\displaystyle {\mathcal {L}}(Q)}$, as in

Strictly-speaking, the quantity ${\displaystyle {\mathcal {L}}(Q)}$ defined this way is itself a random variable jointly-distributed with ${\displaystyle \mathbf {X} }$.

Relationship to entropy[]

The ELBO is closely related to the concepts of Shannon entropy, differential entropy, and cross-entropy. Abusing notation somewhat, we may write ${\displaystyle {\mathcal {L}}(Q)=H(Q)-H(Q,P)}$, where ${\displaystyle H}$ represents Shannon or differential entropy depending on whether our random variables are discrete or continuous, and ${\displaystyle H(Q,P)=H(Q(\mathbf {Z} ),P(\mathbf {X} ,\mathbf {Z} ))}$ represents the cross-entropy between ${\displaystyle Q(\mathbf {Z} )}$ and ${\displaystyle P(\mathbf {X} ,\mathbf {Z} )}$ as a function of ${\displaystyle \mathbf {X} }$.

Relationship to Kullback–Leibler divergence[]

The ELBO inequality can be derived as a consequence of the fact that KL-divergence is always non-negative. Observe that

where ${\displaystyle D_{\mathrm {KL} }}$ is Kullback–Leibler divergence. The desired inequality follows trivially from the above equation because ${\displaystyle D_{\mathrm {KL} }(Q(\mathbf {Z} )\parallel P(\mathbf {Z} \mid \mathbf {X} ))\geq 0}$.

Motivation and use in optimization[]

Returning to the context of variational Bayesian methods,

the task of finding a distribution ${\displaystyle Q(\mathbf {Z} )}$ approximating ${\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}$ can be framed as an optimization problem where we seek to minimize some measurement of divergence between the two distributions. One such measurement, very commonly used, is KL-divergence, which is also sometimes called relative entropy. (For other quantities that measure the dissimilarity between probability distributions, see the article on divergence.) Because

it follows that minimizing ${\displaystyle D_{\mathrm {KL} }(Q(\mathbf {Z} )\parallel P(\mathbf {Z} \mid \mathbf {X} ))}$ is equivalent to maximizing the ELBO ${\displaystyle {\mathcal {L}}(Q)}$. The quantity ${\displaystyle {\mathcal {L}}(Q)}$ can be taken as a learning objective in ML tasks and architectures involving distribution approximation, for example in variational autoencoders. The reason ${\displaystyle {\mathcal {L}}(Q)}$ is commonly used as an optimization target is that it can often be computed in cases where the true posterior ${\displaystyle P(\mathbf {Z} \mid \mathbf {X} )}$ cannot.

References[]