When fine-tuning a large language model using the bf16 (bfloat16) data format, researchers have observed an unexpected behavior: a significant majority of the model's weights may remain completely frozen. This occurs when weights are stored in bf16 and the AdamW optimizer writes updates directly to them without an underlying fp32 (float32) master copy. Because bf16 has a much smaller mantissa than fp32, the gaps between representable numbers widen as the values themselves get larger. Specifically, a bf16 number is essentially the first half of an fp32 number; its gaps are 65,536 times wider than those of fp32, meaning a weight can only represent a change if that change is at least ~1/512 of the weight's own value.
This creates a 'weight mask' where the learning rate determines which weights are allowed to move. If a weight's magnitude is large, the update provided by the optimizer at a small learning rate (e.g., 1e-6) may be less than half of the bf16 gap, causing the update to round back to the original value and be lost. In a model like Mistral-7B, this can result in a 'precision cliff.' For a learning rate of 1e-6, weights larger than approximately 4.88e-4 are effectively frozen. Analysis of the Quiet-STaR model showed that while the training process appeared to be progressing, only about 5.4% of the 7.24 billion weights actually changed, while the remaining 95% were untouched by the optimizer.
Interestingly, this rounding behavior may act as a stabilizing 'feature' rather than a bug. When researchers attempted to rerun the same training recipe using a mathematically 'correct' fp32 master copy, the model often diverged or suffered from extreme loss spikes. The bf16 rounding effectively discards noisy or unstable updates that would otherwise cause the model to blow up at very low learning rates. While this behavior has been observed in several peer-reviewed papers, it is rarely documented as the underlying mechanism. The result is a sparse fine-tuner that only trains the smallest-magnitude weights, a dynamic that can hide training instabilities while producing a stable, albeit restricted, update to the model's parameters.