4 ms·

The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. Is the correlation 1

The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. Is the correlation 1:1? No. Is it still relevant when making a decision about what technology to use? Absolutely.

dragonwriter 7y ago
> The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.
- diminoten 7y ago
  The problem you're having is that everybody already knows that, and plenty of folks can still extract value out of a benchmark anyway.
  - dragonwriter 7y ago
    > The problem you're having I'm not having a problem. > is that everybody already knows that, Certainly, some people act like they don't, at least in the context of specific benchmarks. And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology. > and plenty of folks can still extract value out of a benchmark anyway. Understanding both the general issue and, ideally, the specific areas of potential concern in relation to particular benchmarks and your intended use is a big part of being able to effectively extract value from a benchmark.and, yes, lots of people do recognize those facts and extract value from benchmarks. Others don't, and still apply benchmarks in decision-making, but it's less clear that they are extracting value. Confusing the measurement most readily available with the measurement most relevant to need is a common problem (and not just with benchmarks.)
    - diminoten 7y ago
      The problem you're having is you can't figure out how to use benchmarks to understand how a system works. You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
      - peterwwillis 7y ago
        > Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. You can correlate lots of things. Poor countries increase penis size. Ice cream leads to murder. Cheese kills people by tangling them in their bedsheets. These things are strongly correlated. But they have no causative relationship. To put it another way, a single benchmark test used find a solution is scientifically equivalent to finding one "average" man, making him run a series of athletic tests, and making his performance the benchmark for men's athletics.
        diminoten 7y ago
        Sure, if you do it wrong. But if you do it right, you can get a lot of informative, valuable data out of a benchmark result that can accurately help decide between two (or more) technologies. Like I've been saying, benchmark results are plenty useful if you know how to use them.
      - dragonwriter 7y ago
        > The problem you're having Again, I'm not having a problem. > is you can't figure out how to use benchmarks to understand how a system works No, I have no partucular problem evaluating whether a benchmark is useful to a decision and if so, how. Nor have I said anything indicating any such problem. > You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" No, I'm saying he naive statement upthread that benchmark performance correlates with every possible use of a technology is nonsense. That's it. > Other folks don't do that Actually, some do, but that's neither here nor there. > and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Once again, yes, lots of people have the skill to figure out whether and in what way particular benchmarks have utility for their usecases. I explicitly said that in the post you are responding to. That's very different than your claim that I reacted against, which is that any benchmark inherently correlates with every use case, which is—again—complete nonsense.

dragonwriter 7y ago

> The whole point of benchmarks is that their "use case" correlates roughly with every use case related to the technology being benchmarked. That's the idea of benchmarks, but it's not actually true of real benchmarks, because invariably some factors which can improve a particular benchmarks beaean inverse relation to performance for some other use cases of the technology.

diminoten 7y ago
The problem you're having is that everybody already knows that, and plenty of folks can still extract value out of a benchmark anyway.
- dragonwriter 7y ago
  > The problem you're having I'm not having a problem. > is that everybody already knows that, Certainly, some people act like they don't, at least in the context of specific benchmarks. And some people outright claim the opposite of what you say everyone understands, e.g., by claiming that benchmarks inherently correlate with every possible use of a technology. > and plenty of folks can still extract value out of a benchmark anyway. Understanding both the general issue and, ideally, the specific areas of potential concern in relation to particular benchmarks and your intended use is a big part of being able to effectively extract value from a benchmark.and, yes, lots of people do recognize those facts and extract value from benchmarks. Others don't, and still apply benchmarks in decision-making, but it's less clear that they are extracting value. Confusing the measurement most readily available with the measurement most relevant to need is a common problem (and not just with benchmarks.)
  - diminoten 7y ago
    The problem you're having is you can't figure out how to use benchmarks to understand how a system works. You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Try not to get wrapped around the axle on "everyone", by the way, it's not literal.
    - peterwwillis 7y ago
      > Other folks don't do that, and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. You can correlate lots of things. Poor countries increase penis size. Ice cream leads to murder. Cheese kills people by tangling them in their bedsheets. These things are strongly correlated. But they have no causative relationship. To put it another way, a single benchmark test used find a solution is scientifically equivalent to finding one "average" man, making him run a series of athletic tests, and making his performance the benchmark for men's athletics.
      - diminoten 7y ago
        Sure, if you do it wrong. But if you do it right, you can get a lot of informative, valuable data out of a benchmark result that can accurately help decide between two (or more) technologies. Like I've been saying, benchmark results are plenty useful if you know how to use them.
    - dragonwriter 7y ago
      > The problem you're having Again, I'm not having a problem. > is you can't figure out how to use benchmarks to understand how a system works No, I have no partucular problem evaluating whether a benchmark is useful to a decision and if so, how. Nor have I said anything indicating any such problem. > You're throwing up your hands and saying, "TOO DIFFERENT FROM REALITY!" No, I'm saying he naive statement upthread that benchmark performance correlates with every possible use of a technology is nonsense. That's it. > Other folks don't do that Actually, some do, but that's neither here nor there. > and while not 1:1, they are able to correlate the performance of a benchmark with the performance of their own use case. Once again, yes, lots of people have the skill to figure out whether and in what way particular benchmarks have utility for their usecases. I explicitly said that in the post you are responding to. That's very different than your claim that I reacted against, which is that any benchmark inherently correlates with every use case, which is—again—complete nonsense.