Hyperparameters tuning

Decisions to be taken when training a multilayer perceptron (hyperparameters):

• Activation functions and loss functions;
• Number of layers and number of neurons;
• Regularization;
• Learning rate;
• Batch size.

Activation functions

The activation function ($\sigma$) for the output layer’s neurons depends on the task and the number of classes ($K$).

We distinguish:

• Binary classification ($K=1$). The sigmoid function is used.

  

• Multiclass classification ($K>1$):

  • With non mutually exclusive labels (multilabel):

    

  • With mutually exclusive labels:

    

    Softmax is a variation of the sigmoid activation, in which the output is normalized. The network outputs probability scores: $\sum^K_{i=1} \hat y_i=1$

• Regression (for any $K$):

  

The activation function ($\sigma$) for the hidden layers’ neurons is necessary to introduce non-linearities in the network. Multiple option are available:

• Sigmoid ($\text{sigmoid}$).

  • Formulas:
    • Function: $\sigma_{\text{sigmoid}}(x)=\frac{1}{1+e^{-x}}$.
    • Derivative: $\sigma'{\text{sigmoid}}(x)=\sigma{\text{sigmoid}}(x)⋅(1−\sigma_{\text{sigmoid}}(x))$.
  • Pro: Interpretable output (it’s a probability).
  • Pro: Activation function won’t blow up, as it is limited in the range [0, 1].
  • Con: vanishing gradients (small and large values of $z$ cause the gradients to be close to 0).

• Hyperbolic Tangent ($\tanh$).

  • Formulas:
    • Function: $\sigma_{\text{tanh}}(x)=\frac{2}{1+e^{-2x}}-1=\frac{e^x-e^{-x}}{e^x+e^{-x}}$.
    • Derivative: $\sigma'{\text{tanh}}(x) = 1 - \sigma{\text{tanh}}(x)^{2}$
  • Pro: allows for positive and negative output.
  • Con: same as sigmoid → vanishing gradients (small and large values of $z$ cause the gradients to be close to 0).

  

• Rectified Linear Unit ($\text{ReLU}$).

  • Formulas:

    • Function: $\sigma_{\text{ReLU}}(x)=\max(0,z)$.
    • Derivative: $\sigma'_{\text{ReLU}}(x)=\begin{Bmatrix} 1 & z>0 \\0 & z<0 \end{Bmatrix}$

  • Pro: no vanishing gradients.

  • Pro: less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations.

  • Pro: Converges faster than the sigmoid or tanh functions.

    

    Comparison in convergence between ReLU and tanh (one unit), by Krizhevsky et al. There is an 6x improvement by using ReLU.

  • Con: $∇= 0$ for negative values of $z$.

  • Con: No upper limit, hence it can blow up the activation (exploding gradients).

• Leaky Rectified Linear Unit ($\text{LReLU}$).

  • Formulas:
    • Function: $\sigma_{\text{LReLU}}(x)=\begin{cases} x && \text{if}& x\ge 0\\\alpha x && \text{if}&x<0\end{cases}$ where $\alpha$ is some gradient (usually around 0.01).
    • Derivative: $\sigma'_{\text{LReLU}}(x)=\begin{Bmatrix} 1 & z>0 \\ \alpha & z<0 \end{Bmatrix}$.
  • Pro: nonlinear but piecewise linear, thus easy to differentiate. Always $∇≠ 0$.
  • Con: As it possess linearity, it can’t be used for the complex Classification.
  • Con: No upper limit, hence it can blow up the activation (exploding gradients).

  

• Exponential Linear Unit ($\text{ELU}$).

  • Formulas:
    • Function: $\sigma_{\text{ELU}}(x)=\begin{cases} x && \text{if}& x>0\\\alpha(e^x-1) && \text{if}&x\le0\end{cases}$.
    • Derivative: $\sigma'_{\text{ELU}}(x)=\begin{cases} 1 && \text{if}& x>0\\\alpha\cdot e^x) && \text{if}&x\le0\end{cases}$.
  • Pro: ELU becomes smooth slowly until its output equal to $-α$ whereas RELU sharply smoothes. Unlike to ReLU, ELU can produce negative outputs.
  • Con: • For $x > 0$, it can blow up the activation with the output range of $[0, \infin]$ (exploding gradients).