Activation function are mathematical function used in deep learning after linear layer to shape its output and generate gradient efficiently so that it can be used to learn.Its main use is to introduce curve behavior to our network so that our network can learn difficult pattern also.There are many activation functions which are used in deep learning today for different purpose.In this post we will discuss all such activation functions with its mathematical formula and its usage in network.
- RELU(Rectified Linear Unit):-
It is said that machine learning is trapped in winter from 1970 and didn't progress until in 2011 when RELU is introduced.After its introduction machine learning achieved good progress and prestige in real world.It is very simple function but provide high functionalities to our network.
y=Max(0,x)
As graph shows it allows only positive values to pass to layers.It negate all negative values due to which its converts linear function to non-linear function.This function is mostly used in deep learning.And its used in-between two layers in deep learning.
2.SIGMOID:-
(mathematical equation in book)
As graph indicates above it converts our all input values to positive range.This activation layer mainly used in last layers.It is mainly used in binary classification.
3.Leaky RELU(Leaky Rectified Linear Unit):-
y=Max(0.1*x, x)
Leaky RELU is the advance version of RELU in which all negative is not ignored but it is scaled to very small amount.So that disadvantages of RELU is mitigated.It shows significantly good results in deep learning networks.It is used in places where RELU is used in between two linear layers.
4.ELU(Exponential Linear Unit):-
(mathematical equation in book)
This function is introduced after RELU and Leaky RELU to overcome its limitation to increase value in negative part.In above equation we have Alpha value which is use at the time of modelling.
5.Softmax:-
(mathematical equation in book)
This activation function gives relative percentage pf one output value to other output values.Let say you have five output values which represents different species of ROSE flower.And which output gives percentage of one species when you enter ROSE flower image.That percentage value is achieved by Softmax layer.
6.SOFT PLUS:-
(mathematical equation in book)
7.SINC:-
(mathematical equation in book)
8.SWISS FUNCTION:-
(mathematical equation in book)
9.SELU FUNCTION SCALED:-
(mathematical equation in book)
This function is introduced to reduce the issue of gradient which some times explode or vanish due to depress of networks.Here above Lambda an Alpha is calculated on the basis of input values.This is sometimes considered replacement of BATCH NORMALIZATION.
10.HARD SIGMOID:-
y=max(0,min(1,x)
11.Tanh:-
This function was used in early days of Deep Learning.Now also many algorithm use tanh.
CONCLUSION:-In this post we have discussed about Activation Functions and number of activation function present in deep learning(AI).In which we have discussed RELU, Leaky RELU, SIGMOID, HARD SIGMOID, ELU, SELU, SINC, SWISS Softmax,Softplus and tanh.
Leave a Comment
No Comments Yet