Accurate benchmarking is critical for making high-stakes decisions regarding server platforms, programming stacks, and database vendors. However, many organizations fall into the trap of 'casual benchmarking'—measuring one metric while actually testing another—which can lead to poor architectural choices that persist for years. Brendan Gregg proposes a shift toward 'active benchmarking' to ensure that the data collected actually reflects the intended target of the test.
The Pitfalls of Passive Benchmarking
Passive benchmarking occurs when a benchmark is executed and the results are simply recorded without further investigation. This 'fire and forget' style often results in impressive bar charts that lack supporting technical evidence, making it difficult to determine why a specific result occurred.
Without real-time analysis, it is easy to overlook problems that invalidate the test. For example, a benchmark might appear slow because of a network bottleneck or a software-imposed resource control, rather than the performance of the component being tested.
The Active Benchmarking Methodology
Active benchmarking transforms raw data into information by analyzing the system while the benchmark is still running. This process mirrors standard performance analysis but benefits from a known, controlled workload: the benchmark itself.
To perform active benchmarking, users should run the test for a long duration—potentially hours—to reach a steady state. During this time, they should use OS-provided tools such as vmstat, iostat, mpstat, sar, top, or eBPF-based tools to identify the true limiters of the system.
A key goal of this method is to answer the question: 'Why is the result X and not 2X?' By identifying the limiting factor—such as a single-threaded application on a multi-core server—the user can confirm that the benchmark is testing what they intended.
Identifying System Bottlenecks
Active benchmarking helps identify specific common pitfalls that can skew results. These include being perturbed by neighboring processes, being throttled by network speeds between the client and server, or testing disk I/O instead of the intended file system I/O.
By using a structured methodology like the USE Method (Utilization, Saturation, and Errors), engineers can pinpoint whether a bottleneck is a hardware resource or a software limitation. This ensures that the final conclusions are based on a clear understanding of the system's constraints.
Validation and Statistical Integrity
Statistical analysis is a valuable tool for developing confidence in numerical results, but it is not a substitute for active benchmarking. If the initial data is flawed due to an unobserved bottleneck, statistical methods can make those false results appear more trustworthy.
The recommended workflow is to perform active benchmarking first to ensure the data is valid, followed by statistical analysis to refine the findings. This sequence ensures that the final conclusions are based on a sound foundation of evidence.