2. Neural Networks

Neural Network

A feedforward neural network (multi-layer perceptron) generalizes the perceptron by connecting many neurons in layers. The first layer takes the input vector $x=[x_1,\dots ,x_D]$ and passes it through weighted connections to a set of hidden neurons, such that the $j$-th neuron in the hidden layer computes
$a_j^{(1)}={\sum }_i{w}_{ji}^{(1)}{x}_i+w_{j0}^{(1)},z_j^{(1)}=h(a_j^{(1)}),$
where $h(\cdot )$ is a non-linear activation such as sigmoid, tanh, or ReLU.

The resulting activations $z^{(1)}$ form the inputs to the next layer, so a second hidden layer would compute
$z^{(2)}=h(W^{(2)}{z}^{(1)}+b^{(2)}),$
and so on until the output layer produces
$y=f(W^{(L)}{z}^{(L-1)}+b^{(L)}),$
where $f(\cdot )$ maps to the final prediction - often a softmax or linear output depending on the task:
[PLACEHOLDER FIGURE]

Each layer extracts progressively higher-level structure: early layers detect simple features, later layers combine them into more abstract ones. With enough hidden units, even a single hidden layer can approximate any continuous function per the Universal Approximation Theorem, though multiple layers often achieve the same with far fewer parameters and greater representational efficiency.

Properties

(Weight Symmetry) There are $M^L$ ways to permute an $L$-layer network with hidden width $M$.