Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.
These models are called feedforward because information flows through the function being evaluated from $x$, through the intermediate computations used to define $f$, and finally to the output $y$. There are no feedback connections in which outputs of the model are fed back into itself, in other words, there are no loops or cycles in the network. Models where we do have cycles are called recurrent neural networks, which will be discussed later.
The feedforward network consists several layers, each of which contains some number of identical units (resembles a neuron, in the sense that it receives input from many other units and computes its own activation value). Every unit in one layer is connected to every unit in the next layer; we say that the network is fully connected. The first layer is the input layer, and its units take the values of the input features. The last layer is the output layer, and it has one unit for each value the network outputs (i.e. a single unit in the case of regression or binary classification, or $K$ units in the case of $K$-class classification). All the layers in between (there must be at least one) these are known as hidden layers. The input signals are propagated in a forward direction on a layer-by-layer basis. The units in these laters are known as input units, output units, and hidden units, respectively. The number of layers is known as the depth, and the number of units in a layer is known as the width.
When training, we drive $f(x)$ to match $f^*(x)$. The training data provides us with noisy, approximate examples of $f ∗ ( x )$ evaluated at different training points. Each example $x$ is accompanied by a label $y ≈ f^∗ ( x )$, which is the desired output. The behavior of the other layers is not directly specified by the training data. The learning algorithm must decide how to use those layers to produce the desired output, but the training data do not say what each individual layer should do. Instead, the learning algorithm must decide how to use these layers to best implement an approximation of $f^∗$. Because the training data does not show the desired output for each of these layers, they are called hidden layers.
With one hidden layer, we can represent any continuous function of the input signals, and with two hidden layers even discontinuous functions can be represented. In other words, MLP networks can model any function (Cybenkp. 1989, Kurt, 1991) - given infinitely many hidden units.
Note here we deal with fully connected networks. I.e., every unit in layer $a$ is connected to every unit in layer $a+1$.
In a multilayer perceptron, the activation function is referred to as $\sigma$ (sigma).
The activation functions are necessary to introduce nonlinearities in the network.
Smooth (differentiable) activation functions are suited to perform gradient descent. As a result, the step, sign and linear function are not suitable, because they are not differentiable. In other words, we want a function that can output a value between 0 or 1. The step function does only output 0 or 1, can’t a value in between.
The smoothness of $σ$ means that small changes $Δw_j$ in the weights and $Δb$ in the bias will produce a small change $Δ\text{output}$ in the output from the neuron. This is necessary to correctly compute the gradient descent (where we are constantly modifying the values of the weights and biases).
Instead, we use (differentiable) activation functions such as the sigmoid function. We will later study more activation functions.
Before explaining the actual algorithms about MLP networks, we will simplify the expressions for neural networks with more matrix and vector notations.
In order to implement a neural network efficiently, one must be careful when using for loops. In practice, the dimensionalities of the inputs and hidden units are high. As a result, code will run very slowly if you use for loops. Leveraging the parallelism in GPUs is/was crucial for the progress of deep learning.
This gave rise to vectorization. Instead of using for loops, vectorization takes advantage of matrix algebra and highly optimized numerical linear algebra packages (e.g., BLAS) to make neural network computations run quickly. Before the deep learning era, a for loop may have been sufficient on smaller datasets, but modern deep networks and state-of-the-art datasets will be infeasible to run with for loops.
As a result, in the first layer with $m$ units and $d$ connections, we represent the weight matrix as a concatenation of the weights of each unit, i.e.,