Here's my little contribution / anecdote. I worked on automated testing of a product in the general category of SDS block device. We wanted to see some "reali

Here's my little contribution / anecdote. I worked on automated testing of a product in the general category of SDS block device. We wanted to see some "realistic" benchmarks (instead of totally synthetic FIO-based ones), so we decided to do, beside other things, some benchmarks of PostgreSQL on top of our block device. At that time we've already developed a convention for what would be the "optimal" way to configure a filesystem on top of our block device -- which is a whole other story, but I won't reproduce it here. The most tempting aspect of PostgreSQL was the existence of pgbench tool that I assumed would offer some sort of a standardized workload that would allow me to compare my results to results produced by other testers on other hardware... Little did I know how and in how many ways PostgreSQL performance was dependent on configuration. The most important pieces are the memory buffers and fsync-related stuff. Eventually, by reading more documentation and some educational material about PostgreSQL I realized that, essentially, I can make the same exact benchmark on the same exact hardware to very hundreds if not thousands of times in performance for the same exact test. I also realized that storage's performance had very little to do with the benchmark's scores as in order to game the benchmark one would simply add more memory.