4 ms·

These not-real-world microbenchmarks are definitely useless from an engineering perspective. However they aren't completely useless. Their use is marketing. It

by Strom 10y ago

These not-real-world microbenchmarks are definitely useless from an engineering perspective. However they aren't completely useless. Their use is marketing. It gets the word out to developers that this product X is really good! So what if it doesn't translate to real world scenarios or even if the numbers are completely fabricated and you can't even reproduce this under lab conditions. [1] Very few people care enough to look at things that closely. Just seeing a bunch of posts claiming product X is really good is enough to leave a strong impression that product X really is that great. Perception is reality, and perception is usually better influenced by massive claims (even if untrue) rather than realstic iterative progress. Our product is 5% faster than state of the art! just doesn't have that viral headline nature that you need to win over the hearts of the masses. As for why would someone do this. Maybe they don't know better, or maybe they are doing it because they have decided to invest in some technology tribe and thus profit from that tribe surviving, and even more from growing. This is a pretty automatic behavior for humans. Take any tribal war, e.g. XBox One vs PS4. Those who happen to own a XBox One (perhaps as a gift) can be seen at various places passionately arguing that XBox One is better than PS4, even if objectively it has worse hardware and less highly acclaimed exlcusive games. The person is on the XBox One tribe, and working towards getting more users to own XBox One will mean that more developer investments are also made towards the XBox One thanks to the bigger userbase. Thus even if the original claims to get users into the tribe were false, if growth is big enough it may work out well enough at the end. -- [1] The RethinkDB postmortem [2] had a great paragraph about these microbenchmarks. People wanted RethinkDB to be fast on workloads they actually tried, rather than “real world” workloads we suggested. For example, they’d write quick scripts to measure how long it takes to insert ten thousand documents without ever reading them back. MongoDB mastered these workloads brilliantly, while we fought the losing battle of educating the market. [2] http://www.defstartup.org/2017/01/18/why-rethinkdb-failed.html http://www.defstartup.org/2017/01/18/why-rethinkdb-failed.ht...

deleted 10y ago
[deleted]
mcherm 10y ago
Your example from RethinkDB really struck home to me. The idea that superior technology might lose out due to poor marketing or (in this case) a system that is optimized for the real world rather than being optimized for benchmarks really disturbs me. And (this is just my personality) I don't like being disturbed about something without trying to "solve" it. So here's my best thought on how to handle the situation where a team feels that they have a superior product which is losing out to another product that is optimized for benchmarks: > Provide a setting called something like "speed mode". In this mode it is completely optimized for the benchmarks, at the cost of everything else. Default to running without "speed mode", but for anyone who is running benchmarks ask them if they've tried it in "speed mode". A truly competent evaluator will insist on trying the system with the options that are really used in the real world, but then the competent evaluator won't be using an unreliable benchmark anyway. Anyone running the benchmarks just to see how well it works will be likely to turn on something named "speed mode", or at least to do so if asked to. Forums will eventually fill up with people recommending "for real-world loads, you should disable 'speed mode' as it doesn't actually speed them up". Hmm... sounds cool, but I'm not so sure it would actually work. The danger is that you would instead develop a reputation for "cheating" on benchmarks. This is why I'm not very good at marketing.

by Strom 10y ago

deleted 10y ago

[deleted]

mcherm 10y ago

Your example from RethinkDB really struck home to me. The idea that superior technology might lose out due to poor marketing or (in this case) a system that is optimized for the real world rather than being optimized for benchmarks really disturbs me. And (this is just my personality) I don't like being disturbed about something without trying to "solve" it. So here's my best thought on how to handle the situation where a team feels that they have a superior product which is losing out to another product that is optimized for benchmarks: > Provide a setting called something like "speed mode". In this mode it is completely optimized for the benchmarks, at the cost of everything else. Default to running without "speed mode", but for anyone who is running benchmarks ask them if they've tried it in "speed mode". A truly competent evaluator will insist on trying the system with the options that are really used in the real world, but then the competent evaluator won't be using an unreliable benchmark anyway. Anyone running the benchmarks just to see how well it works will be likely to turn on something named "speed mode", or at least to do so if asked to. Forums will eventually fill up with people recommending "for real-world loads, you should disable 'speed mode' as it doesn't actually speed them up". Hmm... sounds cool, but I'm not so sure it would actually work. The danger is that you would instead develop a reputation for "cheating" on benchmarks. This is why I'm not very good at marketing.