GH200 study examines practical model training limits
Independent researcher Yoon Dongwook’s GH200 preprint describes training a mixture of experts model with 20.99 billion total parameters and 3.36 billion active per token. It first appeared on October 11 and was revised on October 12.
One GH200 trained the first four billion tokens. A second joined through DiLoCo, creating two independent workers. Each used one GPU for a complete model. The unfinished run covers 10.9 billion tokens.
Reported model FLOPs utilization reached 48.9 to 49.1 percent per GPU, or an estimated 47.6 to 47.8 including synchronization and checkpoints. Those figures use a BF16 peak despite partial FP8 computation. A mixed precision denominator gives about 37 percent. Document mask accounting also lowers the BF16 estimate.
The author moved 238.5 GiB of FP32 training state into Grace CPU memory. Tests used one model and instance type. Settings changed between tests. No BF16 control establishes FP8 quality. This short run does not establish final model quality.
Why the memory connection matters
NVIDIA’s architecture description, published in 2022, explains the hardware advantage. Grace Hopper connects its CPU and GPU through a coherent NVLink connection with up to 900 GB per second of total bandwidth, or 450 in each direction. Both processors can access memory on the other side. That lets software place data beyond the GPU’s own memory while retaining a fast path to it. NVIDIA also describes hardware support for synchronization between CPU and GPU threads, which helps coordinate work across the two processors.
The practical implication is that memory capacity and the connection to that memory belong in the same purchasing decision. Adding system RAM to an ordinary GPU computer does not reproduce this architecture. A plan to move training state needs a bandwidth budget as well as enough space.
How separate workers cooperate
The original DiLoCo paper, submitted in 2023, addresses a different bottleneck. Conventional distributed training exchanges information frequently across tightly connected accelerators. DiLoCo instead lets workers perform many local optimization steps before a shared update. Its inner optimizer is AdamW and its outer optimizer uses Nesterov momentum. This separates local computation from the less frequent coordination between workers. The method was designed for groups of devices with weaker connections, and its authors also investigated workers becoming available or unavailable during training.
For a team evaluating such a setup, the useful test is whether the complete workflow fits its constraints. Compare sustained throughput, synchronization time and validation quality on a representative run before committing to a longer training schedule. Keep the utilization formula visible when comparing results.
Illustrative server rack photograph by Scott McKeown from the WordPress Photo Directory, published in 2022 and available under CC0.
