3 ms·
Looking forward to reading this. My “favorite” database benchmarking bugaboo is unrealistic random data distribution in the test set, typically by relying on th
by twoodfin 3y ago
Looking forward to reading this. My “favorite” database benchmarking bugaboo is unrealistic random data distribution in the test set, typically by relying on the uniform results you’ll get out of the default random() function in your favorite language library.
Application data is almost never uniformly random. As a result, you’ll miss the real-world effects of a wide range of system behavior: CPU branch prediction, data compression, adaptive query optimization, …
- eatonphil 3y agoI think you'd at least get a reasonable worst-case behavior by working with (cryptographically secure) random data, no? I agree you wouldn't see many optimizations that likely would take effect in the real world. But worst-case scenarios are helpful to know too.
- convolvatron 3y agouniform distribution isn't always the worst case. just for an example, if you stuff a btree with pages and pages of the same key it starts to get funny. it also doesn't really exercise the join ordering machine correctly if every key has at most one correlate.
- eatonphil 3y agoYeah that's another good worst-case to think about.
- o11c 3y agoSome useful distributions: * Generate a random number in [1, NUMBITS], then generate a random number with that many bits. * Generate a random number and count its trailing (or leading, if easier in your environment) zeroes, then generate a random number with that many bits. Be sure to correctly handle the case where the first random number is zero. * Generate two random numbers and take the minimum.