Published on

Raj - 2024 : Lecture 1 & 2

Lecture 1, Lecture 2

Threshold activation function:

σT,a,b(z)={a     if z≥Tb     otherwise \begin{gather*} \sigma_{T, a, b}(z) = \begin{cases} a \;\; \text{ if } z \geq T \\ b \;\; \text{ otherwise } \end{cases} \end{gather*}

For any decision boundary (for classification problems), one can construct an MLP with threshold activation functions that captures it with arbitrary precision.

For any continuous functions with any number of inputs (over the real), there exists an MLP (with threshold activation functions) that models this function up to any precision.

  1. The depth and width need to be sufficiently large.

  2. Depth can be traded off from exponential growth of the width of the network.