I recently looked at the ViT paper, which shows that transformers can work better for vision than CNNs.

And then I thought, wait a minute, my AlphaZero-style AI for the wall game has a CNN!

So I checked if anyone has used transformers with the AlphaZero-style tree search, and, indeed, Leela Chess Zero's transformer policy jumped ~270 Elo over the CNN.

Wall game transformer training run is now ongoing, should be ready in a few days.