3 ms·
Disabling the GC means your benchmark is worthless. 1. Valid GC during execution: you are removing an actual factor; 2. "Invalid" i.e. unrelated GC during exe
by rue 16y ago
Disabling the GC means your benchmark is worthless.
1. Valid GC during execution: you are removing an actual factor;
2. "Invalid" i.e. unrelated GC during execution: your benchmark is too narrow in scope.
- Groxx 16y agoOn the other hand, 10k+ iterations likely isn't anything you normally experience, and may incur abnormal GC cost due to the sheer number. Though that's typically detectable by doing multiple sizes of tests.
- cheald 16y agoIt depends on what you're benchmarking. If you're benching wall time, then yes, you need to have GC on, and you need to do a ridiculous number of iterations to average out GC runtimes. You also need to invoke the GC before you begin your benchmark; doing a collection pass early in the benchmark for all the objects created on startup/setup skews the benchmark. Separate benchmarks should be conducted for execution speed (CPU time) and garbage generation. Saying "X is faster than Y" tends to suggest a CPU time benchmark. "X is faster than Y with W delta garbage and Z delta less time spent in GC" suggests a proper wall time benchmark.