4 ms·

Because all a benchmark tells you is the result of one use case. It's not an estimate of what your use will result in. It's basically a sales pitch. And sadly,

Because all a benchmark tells you is the result of one use case. It's not an estimate of what your use will result in. It's basically a sales pitch. And sadly, upper management falls for it all the time, often without an evaluation period or even spitballing it for a week with the teams that would be using it. Tons of factors will change the result for a particular use case, regardless of how "normalized" the test tries to be (especially considering most are designed for high-load high-performance high-scale scenarios, which isn't the average real-world use case; most people just have unoptimized queries, or a shit legacy stack, or they expect to eventually have a big use case and were told to find either the fastest or cheapest solution, which may not even be in AWS to begin with, and so on...). The AWS cost is just another benchmark, so it's just as much a lie.

diminoten 7y ago
The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.
- dragonwriter 7y ago
  > The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
  - diminoten 7y ago
    The problem you're having is that everybody already knows that, and plenty of folks can still extract value out of a benchmark anyway.
    - dragonwriter 7y ago
      > The problem you're having I'm not having a problem. > is that everybody already knows that, Certainly, some people act like they don't, at least in the context of specific benchmarks. And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology. > and plenty of folks can still extract value out of a benchmark anyway. Understanding both the general issue and, ideally, the specific areas of potential concern in relation to particular benchmarks and your intended use is a big part of being able to effectively extract value from a benchmark.and, yes, lots of people do recognize those facts and extract value from benchmarks. Others don't, and still apply benchmarks in decision-making, but it's less clear that they are extracting value. Confusing the measurement most readily available with the measurement most relevant to need is a common problem (and not just with benchmarks.)
      - diminoten 7y ago
        The problem you're having is you can't figure out how to use benchmarks to understand how a system works. You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
        peterwwillis 7y ago
        > Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. You can correlate lots of things. Poor countries increase penis size. Ice cream leads to murder. Cheese kills people by tangling them in their bedsheets. These things are strongly correlated. But they have no causative relationship. To put it another way, a single benchmark test used find a solution is scientifically equivalent to finding one "average" man, making him run a series of athletic tests, and making his performance the benchmark for men's athletics.
rickn 7y ago
(MemSQL VP of Product Management here) We didn't intend the benchmarks to be a sales pitch. We are proud of the performance of MemSQL and what it has been able to achieve in our customers' workloads. We wanted a way to show what we are capable of with concrete numbers. We chose the standard benchmarks because they are well understood, not because they were necessarily representative of any given customer. In general, benchmarks are useful to understand the strengths and weaknesses of a product at a basic level and how it compares to its peers, but we strongly encourage anyone evaluating their options to do a proper POC comparison on their actual use cases.

diminoten 7y ago

The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.

dragonwriter 7y ago
> The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
- diminoten 7y ago
  The problem you're having is that everybody already knows that, and plenty of folks can still extract value out of a benchmark anyway.
  - dragonwriter 7y ago
    > The problem you're having I'm not having a problem. > is that everybody already knows that, Certainly, some people act like they don't, at least in the context of specific benchmarks. And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology. > and plenty of folks can still extract value out of a benchmark anyway. Understanding both the general issue and, ideally, the specific areas of potential concern in relation to particular benchmarks and your intended use is a big part of being able to effectively extract value from a benchmark.and, yes, lots of people do recognize those facts and extract value from benchmarks. Others don't, and still apply benchmarks in decision-making, but it's less clear that they are extracting value. Confusing the measurement most readily available with the measurement most relevant to need is a common problem (and not just with benchmarks.)
    - diminoten 7y ago
      The problem you're having is you can't figure out how to use benchmarks to understand how a system works. You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
      - peterwwillis 7y ago
        > Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. You can correlate lots of things. Poor countries increase penis size. Ice cream leads to murder. Cheese kills people by tangling them in their bedsheets. These things are strongly correlated. But they have no causative relationship. To put it another way, a single benchmark test used find a solution is scientifically equivalent to finding one "average" man, making him run a series of athletic tests, and making his performance the benchmark for men's athletics.

rickn 7y ago

(MemSQL VP of Product Management here) We didn't intend the benchmarks to be a sales pitch. We are proud of the performance of MemSQL and what it has been able to achieve in our customers' workloads. We wanted a way to show what we are capable of with concrete numbers. We chose the standard benchmarks because they are well understood, not because they were necessarily representative of any given customer. In general, benchmarks are useful to understand the strengths and weaknesses of a product at a basic level and how it compares to its peers, but we strongly encourage anyone evaluating their options to do a proper POC comparison on their actual use cases.