New blog post: A Fully Annotated NanoGPT Script
In 2024, Andrej Karpathy reproduced GPT-2 with a small script. Training it to GPT-2's quality level took the equivalent of 45 minutes on 8 NVIDIA H100 GPUs.
Since then, the time to reach the same bar has been driven down to under 90 seconds. That's the NanoGPT speedrun.
I took one of the submissions - not the current record but one far faster than Karpathy's - and annotated all 355 lines.
Link below.
