Out of curiosity, I checked whether the loss at the half-way point of a NanoGPT speedrun predicts the final loss, using PrimeIntellect's corpus of runs.

The trick is that you have to compare runs on the same learning schedule. The schedule decides where you are in the "cool-down" at the half-way mark, so across different schedules the half-way loss is misleading: a run that cools down late looks bad half-way and finishes great. Compare across schedules and the half-way leader is often the eventual loser.

So I grouped runs by schedule and looked within. Here are 780 runs that share one schedule but use 65 different optimizers. Half-way loss predicts final loss at a rank correlation of 0.65: a decent, though not tight, early read.

One thing jumped out while doing this. On that fixed schedule, the random seed moves the final loss more than the choice of optimizer does: about 78% of the spread across these optimizer variants is luck, only 22% is the optimizer. The big historical jumps (the Muon optimizer, changing the schedule shape) are far larger than this, but the small tweaks people try at the frontier are mostly within noise of each other.