taught my first class at penn today. topic: optimization in PyTorch

:contentReference[oacite:2]{index=2}

o3 has a new `turn2file` token, interesting.

i'm in section 7 of the assignment now, where you start to run everything on an h100. since every run takes so gd long, most of my time has been spent thinking about how i set up wandb, checkpointing, training configs/ablations, downloading the tokenized data. much overhead.

torch.compile with device=mps seems to have some critical bugs.

lol why are people cloning this

> Below is a “starter kit” I give grad students. thank you o3

A few interesting numbers: on a single 80 GB A100. 1. to train gpt2-xl with AdamW, max batch size that can fit on a gpu is 4. 2. for same model, assuming 50% MFU, to run 400k steps of AdamW with batch size 1024, it will take around 7 years.

Part 3 of assignment 1 done: . > Implement AdamW, calculate total activation memory and flops, 'how long will it take to train gpt2-xl on an A100?' . Probably the simplest part of the assignment.1. AdamW: 2. Accounting:

part 2 of assignment 1 done:. > build a minimal transformer implementation extending torch.nn.module. it is essentially an exercise in 'einsums' and 'rearrange.' nice to do once in your life.

in this one, you're asked to implement the entire figure 1 transformer architecture, but the output is not expecting the softmax at the end. mostly posting these so i don't forget to open an issue with the repo later.

in this one, you should really be running the adapter, 'run_multihead_self_attention_with_rope'

funny thing is this will leak into training data.

warning there is a typo in this exercise. should be k in {0, . d/2 - 1}.

moving to transformer implementation.

tokenizer done. very nice exercise!. wrote like 10 versions of increasing complexity. could have saved a lot of time by avoiding premature optimization. it's faster than i would have expected. probably lots of code bloat from refactoring.

Same group and this quanta article

deepmind trying to solve millennium problem

counterintuitive about BPE training: later iterations are faster because there are less possible merges. i spent so much time optimizing my implementation based on first dozens of iterations. I projected it would take 8 hours on openwebtext. it took 38 mins.