Administration

Intro – Long

Notebook – Intro to Python

Notebook – Intro to PyTorch

Hidden Layers: 0 vs. 1 vs. 2

Backpropagation – Part 1

Backpropagation – Part 2

Implement a NN in NumPy

Notebook – Implementation Redo: Classes instead of Functions (NumPy)

Classification – Softmax and Cross Entropy – Theory

Classification – Softmax and Cross Entropy – Derivatives

Notebook – Implementing Classification (NumPy)

Automatic Differentiation

Backpropagation vs. Forward Propagation (Forward vs. Reverse mode autodiff)

Tanh & Permutation Symmetries

Notebook – Symmetries: tanh, permutations, ReLU

Generalization and the Bias-Variance Trade-off

Generalization Code

L2 Regularization / Weight Decay

Dropout Regularization

Notebook – Dropout implementation (NumPy)

Notebook – Early Stopping

Weight initialization – Part 1 – What not to do

Notebook – Weight initialization Part 1

Weight initialization – Part 2 – What to do

Notebook – Weight initialization Part 2

Notebook – TensorBoard

Learning Rate Decay

Notebook – Input Normalization

Batch Normalization – Part 1: Theory

Batch Normalization – Part 2: Derivatives

Notebook – BatchNorm (PyTorch)

Notebook – BatchNorm (NumPy)

Classical Activations

ReLU Variants

A Brief History of ReLU

SGD Variants: Momentum, NAG, AdaGrad, RMSprop, AdaDelta, Adam, AdaMax, Nadam- Part 1: Theory

SGD Variants: Momentum, NAG, AdaGrad, RMSprop, AdaDelta, Adam, AdaMax, Nadam – Part 2: Code

Variational Auto Encoders (VAE)