Rate of convergence

Differential equations
Navier–Stokes differential equations used to simulate airflow around an obstruction
show Scope Natural sciences Engineering Astronomy Physics Chemistry Biology Geology Applied mathematics Continuum mechanics Chaos theory Dynamical systems Social sciences Economics Population dynamics
Classification
show Types Ordinary Partial Differential-algebraic Integro-differential Fractional Linear Non-linear By variable type Dependent and independent variables Autonomous Complex Coupled / Decoupled Exact Homogeneous / Nonhomogeneous Features Order Operator Notation
show Relation to processes Difference (discrete analogue) Stochastic Stochastic partial Delay
Solution
show Existence and uniqueness Picard–Lindelöf theorem Peano existence theorem Carathéodory's existence theorem Cauchy–Kowalevski theorem
show General topics Wronskian Phase portrait Phase space Lyapunov / Asymptotic / Exponential stability Rate of convergence Series / Integral solutions Numerical integration Dirac delta function
show Solution methods Inspection Method of characteristics Euler Exponential response formula Finite difference (Crank–Nicolson) Finite element Infinite element Finite volume Galerkin Petrov–Galerkin Integrating factor Integral transforms Perturbation theory Runge–Kutta Separation of variables Undetermined coefficients Variation of parameters
show People Isaac Newton Gottfried Leibniz Leonhard Euler Émile Picard Józef Maria Hoene-Wroński Ernst Lindelöf Rudolf Lipschitz Augustin-Louis Cauchy John Crank Phyllis Nicolson Carl David Tolmé Runge Martin Kutta
v t

In numerical analysis, the order of convergence and the rate of convergence of a convergent sequence are quantities that represent how quickly the sequence approaches its limit. A sequence ${\displaystyle (x_{n})}$ that converges to ${\displaystyle x^{*}}$ is said to have order of convergence ${\displaystyle q\geq 1}$ and rate of convergence ${\displaystyle \mu }$ if

${\displaystyle \lim _{n\rightarrow \infty }{\frac {\left|x_{n+1}-x^{*}\right|}{\left|x_{n}-x^{*}\right|^{q}}}=\mu .}$^[1]

The rate of convergence ${\displaystyle \mu }$ is also called the asymptotic error constant. Note that this terminology is not standardized and some authors will use rate where this article uses order (e.g., ^[2]).

In practice, the rate and order of convergence provide useful insights when using iterative methods for calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, the asymptotic behavior of a sequence does not give conclusive information about any finite part of the sequence.

Similar concepts are used for discretization methods. The solution of the discretized problem converges to the solution of the continuous problem as the grid size goes to zero, and the speed of convergence is one of the factors of the efficiency of the method. However, the terminology, in this case, is different from the terminology for iterative methods.

Series acceleration is a collection of techniques for improving the rate of convergence of a series discretization. Such acceleration is commonly accomplished with sequence transformations.

Convergence speed for iterative methods[]

Q-convergence definitions[]

Suppose that the sequence ${\displaystyle (x_{k})}$ converges to the number ${\displaystyle L}$. The sequence is said to converge Q-linearly to ${\displaystyle L}$ if there exists a number ${\displaystyle \mu \in (0,1)}$ such that

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=\mu .}$

The number ${\displaystyle \mu }$ is called the rate of convergence.^[3]

The sequence is said to converge Q-superlinearly to ${\displaystyle L}$ (i.e. faster than linearly) if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=0}$

and it is said to converge Q-sublinearly to ${\displaystyle L}$ (i.e. slower than linearly) if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|}}=1.}$

If the sequence converges sublinearly and additionally

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+2}-x_{k+1}|}{|x_{k+1}-x_{k}|}}=1,}$

then it is said that the sequence ${\displaystyle (x_{k})}$ converges logarithmically to ${\displaystyle L}$.^[4] Note that unlike previous definitions, logarithmic convergence is not called "Q-logarithmic."

In order to further classify convergence, the order of convergence is defined as follows. The sequence is said to converge with order ${\displaystyle q}$ to ${\displaystyle L}$ for ${\displaystyle q\geq 1}$ if

${\displaystyle \lim _{k\to \infty }{\frac {|x_{k+1}-L|}{|x_{k}-L|^{q}}}<M}$

for some positive constant ${\displaystyle M>0}$ (not necessarily less than 1 if ${\displaystyle q>1}$). In particular, convergence with order

• ${\displaystyle q=1}$ is called linear convergence (if ${\displaystyle M<1}$),
• ${\displaystyle q=2}$ is called quadratic convergence,
• ${\displaystyle q=3}$ is called cubic convergence,
• etc.

Some sources require that ${\displaystyle q}$ is strictly greater than ${\displaystyle 1}$ since the ${\displaystyle q=1}$ case requires ${\displaystyle M<1}$ so is best treated separately.^[5] It is not necessary, however, that ${\displaystyle q}$ be an integer. For example, the secant method, when converging to a regular, simple root, has an order of φ ≈ 1.618.^{[citation needed]}

In the definitions above, the "Q-" stands for "quotient" because the terms are defined using the quotient between two successive terms.^[6]: 619 Often, however, the "Q-" is dropped and a sequence is simply said to have linear convergence, quadratic convergence, etc.

Order estimation[]

A practical method to calculate the order of convergence for a sequence is to calculate the following sequence, which converges to ${\displaystyle q}$

${\displaystyle q\approx {\frac {\log \left|{\frac {x_{k+1}-x_{k}}{x_{k}-x_{k-1}}}\right|}{\log \left|{\frac {x_{k}-x_{k-1}}{x_{k-1}-x_{k-2}}}\right|}}.}$^[7]

R-convergence definition[]

The Q-convergence definitions have a shortcoming in that they do not include some sequences, such as the sequence ${\displaystyle (b_{k})}$ below, which converge reasonably fast, but whose rate is variable. Therefore, the definition of rate of convergence is extended as follows.

Suppose that ${\displaystyle (x_{k})}$ converges to ${\displaystyle L}$. The sequence is said to converge R-linearly to ${\displaystyle L}$ if there exists a sequence ${\displaystyle (\varepsilon _{k})}$ such that

${\displaystyle |x_{k}-L|\leq \varepsilon _{k}\quad {\text{for all }}k\,,}$

and ${\displaystyle (\varepsilon _{k})}$ converges Q-linearly to zero.^[3] The "R-" prefix stands for "root". ^[6]: 620

Examples[]

Consider the sequence

${\displaystyle (a_{k})=\left\{1,{\frac {1}{2}},{\frac {1}{4}},{\frac {1}{8}},{\frac {1}{16}},{\frac {1}{32}},\ldots ,{\frac {1}{2^{k}}},...\right\}.}$

It can be shown that this sequence converges to ${\displaystyle L=0}$. To determine the type of convergence, we plug the sequence into the definition of Q-linear convergence,

${\displaystyle \lim _{k\to \infty }{\frac {\left|1/2^{k+1}-0\right|}{\left|1/2^{k}-0\right|}}=\lim _{k\to \infty }{\frac {2^{k}}{2^{k+1}}}={\frac {1}{2}}.}$

Thus, we find that ${\displaystyle (a_{k})}$ converges Q-linearly and has a convergence rate of ${\displaystyle \mu =1/2}$. More generally, for any ${\displaystyle c\in \mathbb {R} ,\mu \in (-1,1)}$, the sequence ${\displaystyle (c\mu ^{k})}$ converges linearly with rate ${\displaystyle \mu }$.

The sequence

${\displaystyle (b_{k})=\left\{1,1,{\frac {1}{4}},{\frac {1}{4}},{\frac {1}{16}},{\frac {1}{16}},\ldots ,{\frac {1}{4^{\left\lfloor {\frac {k}{2}}\right\rfloor }}},\,\ldots \right\}}$

also converges linearly to 0 with rate 1/2 under the R-convergence definition, but not under the Q-convergence definition. (Note that ${\displaystyle \lfloor x\rfloor }$ is the floor function, which gives the largest integer that is less than or equal to ${\displaystyle x}$.)

The sequence

${\displaystyle (c_{k})=\left\{{\frac {1}{2}},{\frac {1}{4}},{\frac {1}{16}},{\frac {1}{256}},{\frac {1}{65,\!536}},\ldots ,{\frac {1}{2^{2^{k}}}},\ldots \right\}}$

converges superlinearly. In fact, it is quadratically convergent.

Finally, the sequence

${\displaystyle (d_{k})=\left\{1,{\frac {1}{2}},{\frac {1}{3}},{\frac {1}{4}},{\frac {1}{5}},{\frac {1}{6}},\ldots ,{\frac {1}{k+1}},\ldots \right\}}$

converges sublinearly and logarithmically.

Linear, linear, superlinear (quadratic), and sublinear rates of convergence

Convergence speed for discretization methods[]

This section needs additional citations for verification. Please help by adding citations to reliable sources. Unsourced material may be challenged and removed. (August 2020) (Learn how and when to remove this template message)

This section may require cleanup to meet Wikipedia's quality standards. The specific problem is: There appears to be a mixture of defining convergence with regards to grid points ${\displaystyle n}$ and with step size ${\displaystyle h}$. Section should be modified for consistency and include and explaintion of alternative (equivalent?) definitions. Please help if you can. (August 2020) (Learn how and when to remove this template message)

A similar situation exists for discretization methods. The important parameter here for the convergence speed is not the iteration number k, but the number of grid points and grid spacing. In this case, the number of grid points n in a discretization process is inversely proportional to the grid spacing.

In this case, a sequence ${\displaystyle (x_{n})}$ is said to converge to L with order q if there exists a constant C such that

${\displaystyle |x_{n}-L|<Cn^{-q}{\text{ for all }}n.}$

This is written as ${\displaystyle |x_{n}-L|={\mathcal {O}}(n^{-q})}$ using big O notation.

This is the relevant definition when discussing methods for numerical quadrature or the solution of ordinary differential equations.^{[example needed]}

A practical method to estimate the order of convergence for a discretization method is pick step sizes ${\displaystyle h_{\text{new}}}$ and ${\displaystyle h_{\text{old}}}$ and calculate the resulting errors ${\displaystyle e_{\text{new}}}$ and ${\displaystyle e_{\text{old}}}$. The order of convergence is then approximated by the following formula:

${\displaystyle q\approx {\frac {\log(e_{\text{new}}/e_{\text{old}})}{\log(h_{\text{new}}/h_{\text{old}})}}.}$^{[citation needed]}

Examples (continued)[]

The sequence ${\displaystyle (d_{k})}$ with ${\displaystyle d_{k}=1/(k+1)}$ was introduced above. This sequence converges with order 1 according to the convention for discretization methods.^[why?]

The sequence ${\displaystyle (a_{k})}$ with ${\displaystyle a_{k}=2^{-k}}$, which was also introduced above, converges with order q for every number q. It is said to converge exponentially using the convention for discretization methods. However, it only converges linearly (that is, with order 1) using the convention for iterative methods.^[why?]

The order of convergence of a discretization method is related to its global truncation error (GTE).^[how?]

Acceleration of convergence[]

Many methods exist to increase the rate of convergence of a given sequence, i.e. to transform a given sequence into one converging faster to the same limit. Such techniques are in general known as "series acceleration". The goal of the transformed sequence is to reduce the computational cost of the calculation. One example of series acceleration is Aitken's delta-squared process.

References[]

Literature[]

The simple definition is used in

• Michelle Schatzman (2002), Numerical analysis: a mathematical introduction, Clarendon Press, Oxford. ISBN 0-19-850279-6.

The extended definition is used in

• Walter Gautschi (1997), Numerical analysis: an introduction, Birkhäuser, Boston. ISBN 0-8176-3895-4.
• Endre Süli and David Mayers (2003), An introduction to numerical analysis, Cambridge University Press. ISBN 0-521-00794-1.

The Big O definition is used in

• Richard L. Burden and J. Douglas Faires (2001), Numerical Analysis (7th ed.), Brooks/Cole. ISBN 0-534-38216-9

The terms Q-linear and R-linear are used in; The Big O definition when using Taylor series is used in

• Nocedal, Jorge; Wright, Stephen J. (2006). Numerical Optimization (2nd ed.). Berlin, New York: Springer-Verlag. pp. 619+620. ISBN 978-0-387-30303-1..