Some Notes on Positional Embeddings

06 May 2024

Notes about positional embeddings choices such as sinusoidal and RoPE.

Basic Integrals from Kingma, Welling (2013)

08 Sep 2021

Calculations behind some integrals presented in the seminal paper “Auto-Encoding Variational Bayes”

SVD Part 2 - Insights into Ridge Regression using SVD

10 Jul 2020

SVD as a tool to explain what L2 Regularization ‘does’ for Linear Regression.

Build an LSTM from scratch in Python

12 Aug 2018

I derive equations for Backpropogation-Through-Time (BPTT) for an LSTM and implement one using NumPy.

Build an RNN from scratch in Python

31 Jul 2018

I derive equations for Backpropogation-Through-Time (BPTT) for an RNN and implement one using NumPy.

SVD Part 1 - Introduction to Singular Value Decomposition

21 Jul 2018

Interpreting matrix factors provided by SVD.

Calculus for Batch Normalization

01 Jun 2017

I show gradient calculations for a BatchNorm layer.

Backpropogating Softmax Layer of a Neural Network

20 May 2017

I show gradient calculations for Softmax.