An adventure is only an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.
- G. K. Chesterton
An adventure is only an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.



For TCP/IP, changes of a few percent in link propagation delays and other parameters caused order of magnitude shifts in bandwidth allocation between competing connections. For memory systems, changes in the essentially arbitrary order in which functions were arranged in memory caused changes in runtime of tens of percent for single benchmarks, and of a few percent when averaged across a suite of benchmarks. In both applications the measured variability is larger than performance increases often reported for new improved designs, suggesting that many published measurements of the benefits of new schemes may be erroneous or at least irreproducible.
To make TCP/IP and memory systems measurable enough to make benchmark results meaningful and convincing, randomness must be added. [...] We show how to choose reasonable amounts of randomness based on measuring configuration sensitivity, and propose specific recipies for randomizing TCP/IP and memory systems. Substantial reductions in the configuration sensitivity are demonstrated, making measurements much more robust and meaningful. The accuracy of the results increases with the number of runs and thus is limited only by the available computing resources.