Dust represents a fundamental departure from standard neural network training methods. Unlike backpropagation which requires differentiable computations and calculates first-order gradients through the chain rule, Dust is a zeroth-order optimization method that estimates gradients by directly perturbing activations and measuring loss changes. As described in the primary source, the method adds Gaussian noise to the output of each linear layer independently at every token, runs a forward pass, and rewards each token's noise by the change in loss at that token. This approach creates what the researchers term a 'virtual population' where each token becomes a population member, enabling a single forward pass to evaluate many perturbations in parallel rather than requiring separate passes for each perturbation as in traditional evolution strategies that operate in weight space.
The core efficiency advantage of Dust stems from its activation-space perturbation approach. By perturbing activations instead of weights, Dust avoids the need to materialize every population member in weight space, making it orders of magnitude more efficient than weight-space evolution strategies like EGGROLL. According to the primary source, Dust is on the order of 10^3 to 10^4 times more efficient than a transformer implementation of EGGROLL, a state-of-the-art evolution strategy, based on extrapolations from 1M tokens upward. This efficiency gain comes from evaluating perturbations in parallel across tokens rather than sequentially across population members, with a single forward pass evaluating a population at least three orders of magnitude larger than weight-space ES on modern transformers.
Empirical results demonstrate Dust's competitiveness with established optimization methods. The primary source reports that Dust approximates backprop closely at large population sizes and in multiple settings even exceeds it, suggesting potential advantages in compute-rich regimes. Contrary to conventional wisdom that larger models are less population-efficient, the researchers found that larger models are actually more population-efficient with Dust: a 243M-parameter model outperforms a 120× smaller model at most population sizes. Furthermore, Dust's gradient estimates align better with backprop's as population grows, maintaining good alignment across all tested scales up to 1B tokens, which the researchers view as encouraging for scaling the approach to larger models and longer training runs.