4 ms·
FYI Chrome performance is not generally guided by microbenchmarks, for the exact reasons you mention. It is guided by full-scale benchmarks (e.g. render the top
by jeffbee 6y ago
FYI Chrome performance is not generally guided by microbenchmarks, for the exact reasons you mention. It is guided by full-scale benchmarks (e.g. render the top 10000 sites) and by ChromeOS-wide profiles gathered in the wild. If a performance change doesn't work in the wild as indicated by profiles then it generally will be backed out and reconsidered. This is consistent with Google's backend performance culture where microbenchmarks are fine and good but changes need to be vetted on a full-scale production loadtest fixture.
- brundolf 6y agoYep, makes sense to me. Unit tests assume that a clean-room environment translates reasonably well to the end result; the more naturally complex or unruly a product or a target metric is, the more your testing process should lean towards integration/real-world testing.
- throwawayg5TS 6y agoIt looks like you are confusing testing and benchmarking. Those are different problems that require different skills and tools. Integration/end-to-end tests are terrible for performance benchmarks. They are designed to capture functional issues at the seams. They are usually heavy and not very diverse (because very fragile and/or expensive to maintain), usually focused on a few key critical functional paths, e.g. "can I put something in my shopping cart and pay". That's pretty much the opposite of real-world". The "real-world" is the distribution of what the end-user experiences. With proper tracing, one can identify the key real-world hotspots (including the data associated to it) and then focus on optimizing that.