New Scaling Laws Propose Quadrillion-Parameter Transformers to Close AI Sample Efficiency Gap

Current artificial intelligence models are significantly less efficient at learning than humans, exhibiting a sample efficiency gap of roughly 10,000,000x. While humans possess approximately 100 trillion parameters and learn from only 200 million tokens, current AI models like those following the Chinchilla prescription use a tokens-per-parameter ratio of 20:1. To bridge this gap, researchers at Q suggest that models must scale up by seven orders of magnitude, potentially reaching the quadrillion-parameter scale.

Moving Beyond Fixed-Size Scaling

Standard scaling laws, such as the Chinchilla prescription, suggest a fixed model size for a fixed dataset—specifically 20 tokens per parameter. However, as the supply of high-quality training data becomes a bottleneck, models will increasingly be trained in a multi-epoch, data-constrained regime.

The new research finds that when the dataset is held fixed, the compute-optimal model size continues to grow as the compute budget increases. In this regime, the researchers found that the ensemble should hold over 300 parameters per token at 10^29 FLOPs, a massive increase over previous prescriptions.

This figure visualizes the core finding that model size should grow significantly larger than previous prescriptions when training on a fixed dataset.

“The tokens per parameter gap between AI and humans is thus 10,000,000x, and an important step forward for the sample efficiency gap may be to close it.”

— Quadrillion-parameter transformers and how to train them – Q

Hyperparameter Scaling and Weight Decay

A critical finding of the new law is how hyperparameters must adapt to larger models trained over multiple epochs. Previous data-constrained laws held weight decay fixed, which led to predicted loss increases as compute grew.

The researchers fit power laws showing that optimal weight decay grows steeply with both epochs and model size. Their law remains near-optimal at 16x the fitted compute, whereas prior heuristics fail to account for the interplay between these variables.

The chart explains the methodology for fitting power laws to hyperparameters, which is a critical component of the new scaling law.

Ensembling as a Scaling Axis

To further improve sample efficiency, the researchers integrated ensembling as a first-class scaling axis. They utilized 'snapshot ensembling,' which saves checkpoints from a single training run and averages their predictions, providing a gain in performance without additional training compute.

For very large compute budgets relative to the dataset, the law suggests allocating extra compute to 'deep ensembling'—training independent models with different seeds. This approach reduces overfitting and allows loss to continue falling as compute increases, a point where prior recipes eventually see performance degrade.

This visual supports the claim that ensembling acts as a first-class scaling axis to improve sample efficiency.

The Path to Quadrillion Parameters

The ultimate goal of these scaling laws is to reach the quadrillion-parameter scale to match human-like learning efficiency. While the current research provides the mathematical prescriptions to reach these sizes on limited datasets, the authors note that the transition will require significant advancements in hardware and algorithms.

The current models are roughly seven orders of magnitude too small to match the human brain's efficiency. Closing this gap represents a fundamental shift in how transformer models are designed and trained.