Long short-term memory

Part of a series on
Machine learning and data mining
show Problems Classification Clustering Regression Anomaly detection Data Cleaning AutoML Association rules Reinforcement learning Structured prediction Feature engineering Feature learning Online learning Semi-supervised learning Unsupervised learning Learning to rank Grammar induction
show Supervised learning (classification • regression) Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
show Clustering BIRCH CURE Hierarchical k-means Expectation–maximization (EM) DBSCAN OPTICS Mean shift
show Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE
show Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
show Anomaly detection k-NN Local outlier factor
hide Artificial neural network Autoencoder Cognitive computing Deep learning DeepDream Multilayer perceptron RNN LSTM GRU ESN Restricted Boltzmann machine GAN SOM Convolutional neural network U-Net Transformer Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
show Reinforcement learning Q-learning SARSA Temporal difference (TD)
show Theory Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory
show Machine-learning venues NeurIPS ICML ML JMLR ArXiv:cs.LG
show Related articles Glossary of artificial intelligence List of datasets for machine-learning research Outline of machine learning
v t

The Long Short-Term Memory (LSTM) cell can process data sequentially and keep its hidden state through time.

Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture^[1] used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. It can process not only single data points (such as images), but also entire sequences of data (such as speech or video). For example, LSTM is applicable to tasks such as unsegmented, connected handwriting recognition,^[2] speech recognition^[3][4] and anomaly detection in network traffic or IDSs (intrusion detection systems).

A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell.

LSTM networks are well-suited to classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series. LSTMs were developed to deal with the vanishing gradient problem that can be encountered when training traditional RNNs. Relative insensitivity to gap length is an advantage of LSTM over RNNs, hidden Markov models and other sequence learning methods in numerous applications.^{[citation needed]}

Idea[]

In theory, classic (or "vanilla") RNNs can keep track of arbitrary long-term dependencies in the input sequences. The problem with vanilla RNNs is computational (or practical) in nature: when training a vanilla RNN using back-propagation, the long-term gradients which are back-propagated can "vanish" (that is, they can tend to zero) or "explode" (that is, they can tend to infinity), because of the computations involved in the process, which use finite-precision numbers. RNNs using LSTM units partially solve the vanishing gradient problem, because LSTM units allow gradients to also flow unchanged. However, LSTM networks can still suffer from the exploding gradient problem.^[5]

Variants[]

In the equations below, the lowercase variables represent vectors. Matrices ${\displaystyle W_{q}}$ and ${\displaystyle U_{q}}$ contain, respectively, the weights of the input and recurrent connections, where the subscript ${\displaystyle _{q}}$ can either be the input gate ${\displaystyle i}$, output gate ${\displaystyle o}$, the forget gate ${\displaystyle f}$ or the memory cell ${\displaystyle c}$, depending on the activation being calculated. In this section, we are thus using a "vector notation". So, for example, ${\displaystyle c_{t}\in \mathbb {R} ^{h}}$ is not just one cell of one LSTM unit, but contains ${\displaystyle h}$ LSTM unit's cells.

LSTM with a forget gate[]

The compact forms of the equations for the forward pass of an LSTM unit with a forget gate are:^[1][6]

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\{\tilde {c}}_{t}&=\sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ {\tilde {c}}_{t}\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}}$

where the initial values are ${\displaystyle c_{0}=0}$ and ${\displaystyle h_{0}=0}$ and the operator ${\displaystyle \circ }$ denotes the Hadamard product (element-wise product). The subscript ${\displaystyle t}$ indexes the time step.

Variables[]

• ${\displaystyle x_{t}\in \mathbb {R} ^{d}}$: input vector to the LSTM unit
• ${\displaystyle f_{t}\in {(0,1)}^{h}}$: forget gate's activation vector
• ${\displaystyle i_{t}\in {(0,1)}^{h}}$: input/update gate's activation vector
• ${\displaystyle o_{t}\in {(0,1)}^{h}}$: output gate's activation vector
• ${\displaystyle h_{t}\in {(-1,1)}^{h}}$: hidden state vector also known as output vector of the LSTM unit
• ${\displaystyle {\tilde {c}}_{t}\in {(-1,1)}^{h}}$: cell input activation vector
• ${\displaystyle c_{t}\in \mathbb {R} ^{h}}$: cell state vector
• ${\displaystyle W\in \mathbb {R} ^{h\times d}}$, ${\displaystyle U\in \mathbb {R} ^{h\times h}}$ and ${\displaystyle b\in \mathbb {R} ^{h}}$: weight matrices and bias vector parameters which need to be learned during training

where the superscripts ${\displaystyle d}$ and ${\displaystyle h}$ refer to the number of input features and number of hidden units, respectively.

Activation functions[]

• ${\displaystyle \sigma _{g}}$: sigmoid function.
• ${\displaystyle \sigma _{c}}$: hyperbolic tangent function.
• ${\displaystyle \sigma _{h}}$: hyperbolic tangent function or, as the peephole LSTM paper^[7][8] suggests, ${\displaystyle \sigma _{h}(x)=x}$.

Peephole LSTM[]

A peephole LSTM unit with input (i.e. ${\displaystyle i}$), output (i.e. ${\displaystyle o}$), and forget (i.e. ${\displaystyle f}$) gates.

The figure on the right is a graphical representation of an LSTM unit with peephole connections (i.e. a peephole LSTM).^[7][8] Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state.^[9] ${\displaystyle h_{t-1}}$ is not used, ${\displaystyle c_{t-1}}$ is used instead in most places.

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}}$

Each of the gates can be thought as a "standard" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of a weighted sum. ${\displaystyle i_{t},o_{t}}$ and ${\displaystyle f_{t}}$ represent the activations of respectively the input, output and forget gates, at time step ${\displaystyle t}$.

The 3 exit arrows from the memory cell ${\displaystyle c}$ to the 3 gates ${\displaystyle i,o}$ and ${\displaystyle f}$ represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell ${\displaystyle c}$ at time step ${\displaystyle t-1}$, i.e. the contribution of ${\displaystyle c_{t-1}}$ (and not ${\displaystyle c_{t}}$, as the picture may suggest). In other words, the gates ${\displaystyle i,o}$ and ${\displaystyle f}$ calculate their activations at time step ${\displaystyle t}$ (i.e., respectively, ${\displaystyle i_{t},o_{t}}$ and ${\displaystyle f_{t}}$) also considering the activation of the memory cell ${\displaystyle c}$ at time step ${\displaystyle t-1}$, i.e. ${\displaystyle c_{t-1}}$.

The single left-to-right arrow exiting the memory cell is not a peephole connection and denotes ${\displaystyle c_{t}}$.

The little circles containing a ${\displaystyle \times }$ symbol represent an element-wise multiplication between its inputs. The big circles containing an S-like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum.

Peephole convolutional LSTM[]

Peephole convolutional LSTM.^[10] The ${\displaystyle *}$ denotes the convolution operator.

${\displaystyle {\begin{aligned}f_{t}&=\sigma _{g}(W_{f}*x_{t}+U_{f}*h_{t-1}+V_{f}\circ c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}*x_{t}+U_{i}*h_{t-1}+V_{i}\circ c_{t-1}+b_{i})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}*x_{t}+U_{c}*h_{t-1}+b_{c})\\o_{t}&=\sigma _{g}(W_{o}*x_{t}+U_{o}*h_{t-1}+V_{o}\circ c_{t}+b_{o})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}}$

Training[]

An RNN using LSTM units can be trained in a supervised fashion, on a set of training sequences, using an optimization algorithm, like gradient descent, combined with backpropagation through time to compute the gradients needed during the optimization process, in order to change each weight of the LSTM network in proportion to the derivative of the error (at the output layer of the LSTM network) with respect to corresponding weight.

A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to ${\displaystyle \lim _{n\to \infty }W^{n}=0}$ if the spectral radius of ${\displaystyle W}$ is smaller than 1.^[11][12]

However, with LSTM units, when error values are back-propagated from the output layer, the error remains in the LSTM unit's cell. This "error carousel" continuously feeds error back to each of the LSTM unit's gates, until they learn to cut off the value.

CTC score function[]

Many applications use stacks of LSTM RNNs^[13] and train them by connectionist temporal classification (CTC)^[14] to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.

Alternatives[]

Sometimes, it can be advantageous to train (parts of) an LSTM by neuroevolution^[15] or by policy gradient methods, especially when there is no "teacher" (that is, training labels).

Success[]

There have been several successful stories of training, in a non-supervised fashion, RNNs with LSTM units.

In 2018, Bill Gates called it a “huge milestone in advancing artificial intelligence” when bots developed by OpenAI were able to beat humans in the game of Dota 2.^[16] OpenAI Five consists of five independent but coordinated neural networks. Each network is trained by a policy gradient method without supervising teacher and contains a single-layer, 1024-unit Long-Short-Term-Memory that sees the current game state and emits actions through several possible action heads.^[16]

In 2018, OpenAI also trained a similar LSTM by policy gradients to control a human-like robot hand that manipulates physical objects with unprecedented dexterity.^[17]

In 2019, DeepMind's program AlphaStar used a deep LSTM core to excel at the complex video game Starcraft II.^[18] This was viewed as significant progress towards Artificial General Intelligence.^[18]

Applications[]

Applications of LSTM include:

Timeline of development[]

1995-1997: LSTM was proposed by Sepp Hochreiter and Jürgen Schmidhuber.^[42][43][1] By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem. The initial version of LSTM block included cells, input and output gates.^[44]

1999: Felix Gers and his advisor Jürgen Schmidhuber and Fred Cummins introduced the forget gate (also called “keep gate”) into LSTM architecture,^[45] enabling the LSTM to reset its own state.^[44]

2000: Gers & Schmidhuber & Cummins added peephole connections (connections from the cell to the gates) into the architecture.^[6] Additionally, the output activation function was omitted.^[44]

2009: An LSTM based model won the ICDAR connected handwriting recognition competition. Three such models were submitted by a team led by Alex Graves.^[46] One was the most accurate model in the competition and another was the fastest.^[47]

2013: LSTM networks were a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.^[48]

2014: Kyunghyun Cho et al. put forward a simplified variant called Gated recurrent unit (GRU).^[49]

2015: Google started using an LSTM for speech recognition on Google Voice.^[50][51] According to the official blog post, the new model cut transcription errors by 49%. ^[52]

2016: Google started using an LSTM to suggest messages in the Allo conversation app.^[53] In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%.^[54][55][56]

Apple announced in its Worldwide Developers Conference that it would start using the LSTM for quicktype^[57][58][59] in the iPhone and for Siri.^[60][61]

Amazon released Polly, which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology.^[62]

2017: Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks.^[63]

Researchers from Michigan State University, IBM Research, and Cornell University published a study in the Knowledge Discovery and Data Mining (KDD) conference.^[64][65][66] Their study describes a novel neural network that performs better on certain data sets than the widely used long short-term memory neural network.

Microsoft reported reaching 94.9% recognition accuracy on the , incorporating a vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory".^[67]

2019: Researchers from the University of Waterloo proposed a related RNN architecture which represents continuous windows of time. It was derived using the Legendre polynomials and outperforms the LSTM on some memory-related benchmarks.^[68]

An LSTM model climbed to third place on the in Large Text Compression Benchmark.^[69][70]

See also[]

References[]

External links[]

• Recurrent Neural Networks with over 30 LSTM papers by Jürgen Schmidhuber's group at IDSIA
• Gers, Felix (2001). "Long Short-Term Memory in Recurrent Neural Networks" (PDF). PhD thesis.
• Gers, Felix A.; Schraudolph, Nicol N.; Schmidhuber, Jürgen (Aug 2002). "Learning precise timing with LSTM recurrent networks" (PDF). Journal of Machine Learning Research. 3: 115–143.
• Abidogun, Olusola Adeniyi (2005). Data Mining, Fraud Detection and Mobile Telecommunications: Call Pattern Analysis with Unsupervised Neural Networks. Master's Thesis (Thesis). University of the Western Cape. hdl:11394/249. Archived (PDF) from the original on May 22, 2012.
  • original with two chapters devoted to explaining recurrent neural networks, especially LSTM.
• Monner, Derek D.; Reggia, James A. (2010). "A generalized LSTM-like training algorithm for second-order recurrent neural networks" (PDF). Neural Networks. 25 (1): 70–83. doi:10.1016/j.neunet.2011.07.003. PMC 3217173. PMID 21803542. High-performing extension of LSTM that has been simplified to a single node type and can train arbitrary architectures
• Dolphin, R. "LSTM Networks - A Detailed Explanation". Article.
• Herta, Christian. "How to implement LSTM in Python with Theano". Tutorial.