Bernstein–von Mises theorem

Part of a series on
Bayesian statistics
Theory
Admissible decision rule Bayesian efficiency Bayesian epistemology Bayesian probability Probability interpretations Bayes' theorem Bayes factor Bayesian inference Bayesian network Prior Posterior Likelihood Conjugate prior Posterior predictive Hyperparameter Hyperprior Principle of indifference Principle of maximum entropy Empirical Bayes method Cromwell's rule Bernstein–von Mises theorem Schwarz criterion Credible interval Maximum a posteriori estimation Radical probabilism
Techniques
Bayesian linear regression Bayesian estimator Approximate Bayesian computation Markov chain Monte Carlo
Mathematics portal
v t

In Bayesian inference, the Bernstein-von Mises theorem provides the basis for using Bayesian credible sets for confidence statements in parametric models. It states that under some conditions, a posterior distribution converges in the limit of infinite data to a multivariate normal distribution centered at the maximum likelihood estimator with covariance matrix given by ${\displaystyle n^{-1}I(\theta _{0})^{-1}}$, where ${\displaystyle \theta _{0}}$ is the true population parameter and ${\displaystyle I(\theta _{0})}$ is the Fisher information matrix at the true population parameter value.^[1]

The Bernstein-von Mises theorem links Bayesian inference with frequentist inference. It assumes there is some true probabilistic process that generates the observations, as in frequentism, and then studies the quality of Bayesian methods of recovering that process, and making uncertainty statements about that process. In particular, it states that Bayesian credible sets of a certain credibility level ${\displaystyle \alpha }$ will asymptotically be confidence sets of confidence level ${\displaystyle \alpha }$, which allows for the interpretation of Bayesian credible sets.

Heuristic statement[]

In a model ${\displaystyle (P_{\theta }:\theta \in \Theta )}$, under certain regularity conditions (finite-dimensional, well-specified, smooth, existence of tests), if the prior distribution ${\displaystyle \Pi }$ on ${\displaystyle \theta }$ has a density with respect to the Lebesgue measure which is smooth enough (near ${\displaystyle \theta _{0}}$ bounded away from zero), the total variation distance between the rescaled posterior distribution (by centring and rescaling to ${\displaystyle {\sqrt {n}}(\theta -\theta _{0})}$) and a Gaussian distribution centred on any efficient estimator and with the inverse Fisher information as variance will converge in probability to zero.

Bernstein–von Mises and maximum likelihood estimation[]

In case the maximum likelihood estimator is an efficient estimator, we can plug this in, and we recover a common, more specific, version of the Bernstein–von Mises theorem.

Implications[]

The most important implication of the Bernstein–von Mises theorem is that the Bayesian inference is asymptotically correct from a frequentist point of view. This means that for large amounts of data, one can use the posterior distribution to make, from a frequentist point of view, valid statements about estimation and uncertainty.

History[]

The theorem is named after Richard von Mises and S. N. Bernstein although the first proper proof was given by Joseph L. Doob in 1949 for random variables with finite probability space.^[2] Later Lucien Le Cam, his PhD student , David A. Freedman and Persi Diaconis extended the proof under more general assumptions.^{[citation needed]}

Limitations[]

In case of a misspecified model, the posterior distribution will also become asymptotically Gaussian with a correct mean, but not necessarily with the Fisher information as the variance. This implies that Bayesian credible sets of level ${\displaystyle \alpha }$ cannot be interpreted as confidence sets of level ${\displaystyle \alpha }$.^[3]

In the case of nonparametric statistics, the Bernstein-von Mises theorem usually fails to hold with a notable exception of the Dirichlet process.

A remarkable result was found by Freedman in 1965: the Bernstein–von Mises theorem does not hold almost surely if the random variable has an infinite countable probability space; however, this depends on allowing a very broad range of possible priors. In practice, the priors used typically in research do have the desirable property even with an infinite countable probability space.

Different summary statistics such as the mode and mean may behave differently in the posterior distribution. In Freedman's examples, the posterior density and its mean can converge on the wrong result, but the posterior mode is consistent and will converge on the correct result.

Quotations[]

The statistician A. W. F. Edwards has remarked, "It is sometimes said, in defence of the Bayesian concept, that the choice of prior distribution is unimportant in practice because it hardly influences the posterior distribution at all when there are moderate amounts of data. The less said about this 'defence' the better."^[4]

Notes[]

References[]

• Vaart, A.W. van der (1998). "10.2 Bernstein–von Mises Theorem". Asymptotic Statistics. Cambridge University Press. ISBN 0-521-49603-9.
• Doob, Joseph L. (1949), Application of the theory of martingales. Colloq. Intern. du C.N.R.S (Paris), No. 13, pp. 23–27.
• Freedman, David A. (1963). On the asymptotic behaviour of Bayes estimates in the discrete case I. The Annals of Mathematical Statistics, vol. 34, pp. 1386–1403.
• Freedman, David A. (1965). On the asymptotic behaviour of Bayes estimates in the discrete case II. The Annals of Mathematical Statistics, vol. 36, pp. 454–456.
• Le Cam, Lucien (1986). Asymptotic Methods in Statistical Decision Theory, Springer. ISBN 0-387-96307-3 (Pages 336 and 618–621).
• Lorraine Schwartz (1965). On Bayes procedures. Z. Wahrscheinlichkeitstheorie, No. 4, pp. 10–26.