The e-means project introduces a modern optimization rule that achieves a 10–45× wall-clock reduction on GPUs and an 8–15× reduction on CPUs compared to state-of-the-art k-means implementations. By replacing the traditional full-batch Lloyd's algorithm with a mini-batch approach using exponential moving averages (EMA), the method simultaneously improves clustering quality and training speed.
This chart illustrates the significant reduction in epochs required by e-means to reach the same test loss as the traditional Lloyd's algorithm.
Moving from Full-Batch to Exponential Moving Averages
Traditional k-means typically relies on Lloyd's algorithm, which requires a full pass over the entire dataset to update centroids in every iteration. While accurate, this approach is slow for massive datasets and can easily become trapped in local minima because its loss always decreases monotonically.
e-means replaces this with a mini-batch update rule using exponential moving averages (EMA). Instead of recomputing statistics from scratch, it updates the sum and count of points for each cluster at a rate of alpha, allowing the centroid to track recent assignments rather than the entire history. This approach is more stable than gradient descent because EMA updates guarantee that the centroid remains within the convex hull of the assigned points.
The figure demonstrates how the exponential moving average (EMA) update rule provides a stable and effective alternative to traditional methods.
“e-means achieves both better speed and loss.”
— e-means: 10x faster k-means at equal loss - Blog
Automated Scaling Laws and Robustness
A significant hurdle for mini-batch k-means is the need to tune hyperparameters like batch size and step size. e-means solves this by using scaling laws to automatically derive these values from the problem's characteristics (number of points, clusters, and target epochs). This ensures that the algorithm performs optimally without manual intervention.
The project utilizes these scaling laws to eliminate the need for manual hyperparameter tuning in mini-batch k-means.
The project also improves the reliability of clustering by implementing a 'dead-cluster revival' strategy. When a cluster ends up with no assigned points, e-means revives it by splitting a large, active cluster rather than simply resampling a random point. This preserves the capacity of the codebook and prevents the effective number of clusters from shrinking during training.
Performance Gains and Trade-offs
Experimental validation across 24 different datasets shows that e-means consistently reaches lower test loss than Lloyd's algorithm. For example, on the VIBE embedding benchmark, e-means reached the target error in 1.6 to 2.4 epochs, a massive reduction from the 25 epochs required by standard implementations.
While the algorithmic changes provide the foundation for these gains, the project also utilizes specialized kernels to ensure these improvements manifest as wall-clock speedups. The primary trade-off is that the method's efficiency relies on these optimized kernels to translate the reduction in required epochs into practical time savings on hardware like GPUs.
This visual demonstrates the practical wall-clock speedups achieved by combining algorithmic improvements with specialized GPU kernels.
Chart drawn by HN Digest from figures reported by e-means.github.io; values are shown as written in the source.