Activation Function

Nonlinear functions that are applied to the output of each neuron in a neural network, activation functions are critical for enabling networks to approximate complex mappings from inputs to outputs. Mathematically, these functions introduce non-linearity into the model, allowing it to learn from data that is not linearly separable. Common activation functions include the sigmoid function, defined as σ(x) = 1 / (1 + e^(-x)), which squashes outputs to a range between 0 and 1, and the hyperbolic tangent function, tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x)), which outputs values between -1 and 1. The Rectified Linear Unit (ReLU), defined as f(x) = max(0, x), has become the dominant activation function in modern deep learning due to its ability to mitigate the vanishing gradient problem and improve training speed. The choice of activation function can significantly affect the performance of a neural network, influencing convergence rates and the ability to escape local minima. Activation functions are a fundamental component of neural network architectures, relating closely to concepts such as backpropagation and optimization algorithms.