Learning curve (machine learning)

This article provides insufficient context for those unfamiliar with the subject. Please help by providing more context for the reader. (March 2019) (Learn how and when to remove this template message)

Learning curve showing training score and cross validation score

Part of a series on
Machine learning and data mining
Problems Classification Clustering Regression Anomaly detection Data Cleaning AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection k-NN Local outlier factor
Artificial neural network Autoencoder Cognitive computing Deep learning DeepDream Multilayer perceptron RNN LSTM GRU ESN Restricted Boltzmann machine GAN SOM Convolutional neural network U-Net Transformer Vision Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
Reinforcement learning Q-learning SARSA Temporal difference (TD)
Theory Kernel machines Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
Machine-learning venues NeurIPS ICML ML JMLR ArXiv:cs.LG
Related articles Glossary of artificial intelligence List of datasets for machine-learning research Outline of machine learning
v t

In machine learning, a learning curve (or training curve) plots the optimal value of a model's loss function for a training set against this loss function evaluated on a validation data set with same parameters as produced the optimal function. It is a tool to find out how much a machine model benefits from adding more training data and whether the estimator suffers more from a variance error or a bias error. If both the validation score and the training score converge to a value that is too low with increasing size of the training set, it will not benefit much from more training data.^[1]

The machine learning curve is useful for many purposes including comparing different algorithms,^[2] choosing model parameters during design,^[3] adjusting optimization to improve convergence, and determining the amount of data used for training.^[4]

In the machine learning domain, there are two implications of learning curves differing in the x-axis of the curves, with experience of the model graphed either as the number of training examples used for learning or the number of iterations used in training the model.^[5]

Formal definition[]

One model of a machine learning is producing a function, f(x), which given some information, x, predicts some variable, y, from training data ${\displaystyle X_{\text{train}}}$ and ${\displaystyle Y_{\text{train}}}$. It is distinct from mathematical optimization because ${\displaystyle f}$ should predict well for ${\displaystyle x}$ outside of ${\displaystyle X_{\text{train}}}$.

We often constrain the possible functions to a parameterized family of functions, ${\displaystyle \{f_{\theta }(x):\theta \in \Theta \}}$, so that our function is more generalizable^[6] or so that the function has certain properties such as those that make finding a good ${\displaystyle f}$ easier, or because we have some a priori reason to think that these properties are true.^[6]: 172

Given that it is not possible to produce a function that perfectly fits out data, it is then necessary to produce a loss function ${\displaystyle L(f_{\theta }(X),Y')}$ to measure how good our prediction is. We then define an optimization process which finds a ${\displaystyle \theta }$ which minimizes ${\displaystyle L(f_{\theta }(X_{,}Y))}$ referred to as ${\displaystyle \theta ^{*}(X,Y)}$ .

Training curve for amount of data[]

Then if our training data is ${\displaystyle \{x_{1},x_{2},\dots ,x_{n}\},\{y_{1},y_{2},\dots y_{n}\}}$ and our validation data is ${\displaystyle \{x_{1}',x_{2}',\dots x_{m}'\},\{y_{1}',y_{2}',\dots y_{m}'\}}$ a learning curve is the plot of the two curves

1. ${\displaystyle i\mapsto L(f_{\theta ^{*}(X_{i},Y_{i})}(X_{i}),Y_{i})}$
2. ${\displaystyle i\mapsto L(f_{\theta ^{*}(X_{i},Y_{i})}(X_{i}'),Y_{i}')}$

where ${\displaystyle X_{i}=\{x_{1},x_{2},\dots x_{i}\}}$

Training curve for number of iterations[]

Many optimization processes are iterative, repeating the same step until the process converges to an optimal value. Gradient descent is one such algorithm. If you define ${\displaystyle \theta _{i}^{*}}$ as the approximation of the optimal ${\displaystyle \theta }$ after ${\displaystyle i}$ steps, a learning curve is the plot of

1. ${\displaystyle i\mapsto L(f_{\theta _{i}^{*}(X,Y)}(X),Y)}$
2. ${\displaystyle i\mapsto L(f_{\theta _{i}^{*}(X,Y)}(X'),Y')}$

See also[]

• Overfitting
• Bias–variance tradeoff
• Model selection
• Cross-validation (statistics)
• Validity (statistics)
• Verification and validation

References[]