I Rebuilt Karpathy's NanoChat in JAX. Here's What XLA Gets Right and What It Gets Dead Wrong.

This content describes porting Andrej Karpathy's NanoChat from PyTorch to JAX/Flax NNX, achieving fast training on a single GPU and TPU compatibility. It details XLA's advantages in eliminating Python overhead while highlighting its limitations regarding advanced features and debugging.