Highway Network

A narrow hidden layer forcing compact representations.

Using same parameters across different parts of a model.

Built-in assumptions guiding learning efficiency and generalization.

The range of functions a model can represent.

Optimization using curvature information; often expensive at scale.

Matrix of second derivatives describing local curvature of loss.

Neural networks can approximate any continuous function under certain conditions.

Allows gradients to bypass layers, enabling very deep networks.