## Home page

Deconstructing Deep Learning + δeviations

Drop me an email | RSS feed link : Click
Format : Date | Title
TL; DR

#### Total number of posts : 89

Go To : PAPERS o ARTICLES o BOOKS o SPACE

Go to index

# Activation functions

### Reading time : ~4 mins

by Subhaditya Mukherjee

Implementing activation functions.

Activation functions are an extremely important part of any neural network. But they are actually much simpler than we make them out to be. Here are some of them. Lets define a test matrix.

test = [100 1.0 0.0 -300.0;100 1.0 0.0 300.0]


## Relu

• The Rectified Linear Unit (ReLU) activation function produces 0 as an output when x < 0, and then produces a linear with slope of 1 when x > 0.
• Nair, V., & Hinton, G. E. (2010, January). Rectified linear units improve restricted boltzmann machines. In ICML.
• f(x) = max(0,x)
relu(mat) = max.(0, mat)


## Leaky relu

• Andrew L. Maas, Awni Y. Hannun, Andrew Y. Ng (2014). Rectifier Nonlinearities Improve Neural Network Acoustic Models.
• f(x) = max(0.01x,x)
lrelu(x) = max.(0.01x, x)


## PRelu

• f(x) = max(x, x*a)
#export
prelu(x,a) = max.(x, x.*a)
prelu(test,0.10)


## Maxout

• f(x) = max(x, x*a)
maxout(x,a) = max.(x, x.*a)
maxout(test,0.10)


## Sigmoid

• f(x) = 1/(1+e^-x)
σ(x) = 1 ./(1 .+exp.(-x))
σ(test)


## Noisy Relu

• f(x) = max(0, x+Y) where YϵNormal(0,1)
using Distributions
noisyrelu(x) = max.(0, x.+rand(Distributions.Normal(), 1))
noisyrelu(test)


## Softplus

• f(x) = log(e^x+1)
softplus(x) = log.(exp.(test).+1)
softplus(test)


## Elu

• f(x) = max(x, a*(e^x-1))
elu(x,a) = max.(x, a.*(exp.(x) .-1))
elu(test,0.1)


## Swish

• f(x) = x/(1+e^(-βx))
swish(x,β) = x ./(1 .+exp.(-β.*x))
swish(test,0.1)

Related posts:  FP16  AI Superpowers Kai Fu Lee  Digital Minimalism Cal Newport  More Deep Learning, Less Crying - A guide  Super resolution  Federated Learning  Taking Batchnorm For Granted  A murder mystery and Adversarial attack  Thank you and a rain check  Pruning