Residual neural network

Canonical form of a residual neural network. A layer ℓ  − 1 is skipped over activation from ℓ − 2.

A residual neural network (ResNet) is an artificial neural network (ANN) of a kind that builds on constructs known from pyramidal cells in the cerebral cortex. Residual neural networks do this by utilizing skip connections, or shortcuts to jump over some layers. Typical ResNet models are implemented with double- or triple- layer skips that contain nonlinearities (ReLU) and batch normalization in between.^[1] An additional weight matrix may be used to learn the skip weights; these models are known as HighwayNets.^[2] Models with several parallel skips are referred to as DenseNets.^[3] In the context of residual neural networks, a non-residual network may be described as a plain network.

A reconstruction of a pyramidal cell. Soma and dendrites are labeled in red, axon arbor in blue. (1) Soma, (2) Basal dendrite, (3) Apical dendrite, (4) Axon, (5) Collateral axon.

There are two main reasons to add skip connections: to avoid the problem of vanishing gradients, or to mitigate the Degradation (accuracy saturation) problem; where adding more layers to a suitably deep model leads to higher training error.^[1] During training, the weights adapt to mute the upstream layer^{[clarification needed]}, and amplify the previously-skipped layer. In the simplest case, only the weights for the adjacent layer's connection are adapted, with no explicit weights for the upstream layer. This works best when a single nonlinear layer is stepped over, or when the intermediate layers are all linear. If not, then an explicit weight matrix should be learned for the skipped connection (a HighwayNet should be used).

Skipping effectively simplifies the network, using fewer layers in the initial training stages^{[clarification needed]}. This speeds learning by reducing the impact of vanishing gradients, as there are fewer layers to propagate through. The network then gradually restores the skipped layers as it learns the feature space. Towards the end of training, when all layers are expanded, it stays closer to the manifold^{[clarification needed]} and thus learns faster. A neural network without residual parts explores more of the feature space. This makes it more vulnerable to perturbations that cause it to leave the manifold, and necessitates extra training data to recover.

Biological analogue[]

The brain has structures similar to residual nets, as cortical layer VI neurons get input from layer I, skipping intermediary layers.^[4] In the figure this compares to signals from the apical dendrite (3) skipping over layers, while the basal dendrite (2) collects signals from the previous and/or same layer.^{[note 1][5]} Similar structures exists for other layers.^[6] How many layers in the cerebral cortex compare to layers in an artificial neural network is not clear, nor whether every area in the cerebral cortex exhibits the same structure, but over large areas they appear similar. There is no evidence of anything like backpropagation taking place in the brain and existence of global “teaching signal” or iterative optimization in animal brain is not supported by neurophysiological evidence.

Forward propagation[]

Given a weight matrix ${\textstyle W^{\ell -1,\ell }}$ for connection weights from layer ${\textstyle \ell -1}$ to ${\textstyle \ell }$, and a weight matrix ${\textstyle W^{\ell -2,\ell }}$ for connection weights from layer ${\textstyle \ell -2}$ to ${\textstyle \ell }$, then the forward propagation through the activation function would be (aka HighwayNets)

${\displaystyle {\begin{aligned}a^{\ell }&:=\mathbf {g} (W^{\ell -1,\ell }\cdot a^{\ell -1}+b^{\ell }+W^{\ell -2,\ell }\cdot a^{\ell -2})\\&:=\mathbf {g} (Z^{\ell }+W^{\ell -2,\ell }\cdot a^{\ell -2})\end{aligned}}}$

where

${\textstyle a^{\ell }}$ the activations (outputs) of neurons in layer ${\textstyle \ell }$,

${\textstyle \mathbf {g} }$ the activation function for layer ${\textstyle \ell }$,

${\textstyle W^{\ell -1,\ell }}$ the weight matrix for neurons between layer ${\textstyle \ell -1}$ and ${\textstyle \ell }$, and

${\textstyle Z^{\ell }=W^{\ell -1,\ell }\cdot a^{\ell -1}+b^{\ell }}$

Absent an explicit matrix ${\textstyle W^{\ell -2,\ell }}$ (aka ResNets), forward propagation through the activation function simplifies to

${\displaystyle a^{\ell }:=\mathbf {g} (Z^{\ell }+a^{\ell -2})}$

Another way to formulate this is to substitute an identity matrix for ${\textstyle W^{\ell -2,\ell }}$, but that is only valid when the dimensions match. This is somewhat confusingly called an identity block, which means that the activations from layer ${\textstyle \ell -2}$ are passed to layer ${\textstyle \ell }$ without weighting.

In the cerebral cortex such forward skips are done for several layers. Usually all forward skips start from the same layer, and successively connect to later layers. In the general case this will be expressed as (aka DenseNets)

${\displaystyle a^{\ell }:=\mathbf {g} \left(Z^{\ell }+\sum _{k=2}^{K}W^{\ell -k,\ell }\cdot a^{\ell -k}\right)}$.

Backward propagation[]

During backpropagation learning for the normal path

${\displaystyle \Delta w^{\ell -1,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -1,\ell }}}=-\eta a^{\ell -1}\cdot \delta ^{\ell }}$

and for the skip paths (nearly identical)

${\displaystyle \Delta w^{\ell -2,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -2,\ell }}}=-\eta a^{\ell -2}\cdot \delta ^{\ell }}$.

In both cases

${\textstyle \eta }$ a learning rate (${\textstyle \eta <0)}$,

${\textstyle \delta ^{\ell }}$ the error signal of neurons at layer ${\textstyle \ell }$, and

${\textstyle a_{i}^{\ell }}$ the activation of neurons at layer ${\textstyle \ell }$.

If the skip path has fixed weights (e.g. the identity matrix, as above), then they are not updated. If they can be updated, the rule is an ordinary backpropagation update rule.

In the general case there can be ${\textstyle K}$ skip path weight matrices, thus

${\displaystyle \Delta w^{\ell -k,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -k,\ell }}}=-\eta a^{\ell -k}\cdot \delta ^{\ell }}$

As the learning rules are similar, the weight matrices can be merged and learned in the same step.

Notes[]

References[]