4 ms·

He doesn't have to, it's enough just to point out gaping flaws in the 'study' to call it into question. The onus is on the study's author to make it comprehens

He doesn't have to, it's enough just to point out gaping flaws in the 'study' to call it into question. The onus is on the study's author to make it comprehensive and thorough, not the critics. This one in particular is clearly incomplete, especially when things like Haskell, Java, and Python are represented by a single framework, and not even the best or most optimized ones. For example, why not use Yesod/Warp, BlueEyes, and Tornado on PyPy, respectively, for those instead? Also, the submission title is editorializing flame bait. The study's author didn't make any claims whatsoever about 'Haskell', 'Java', and 'Python', etc. but rather about Snap, Webbit, and ws4py.

davidw 14y ago
> The onus is on the study's author to make it comprehensive and thorough, not the critics. It's always possible for the critics to go out there and do something better, rather than limiting themselves to pointing out flaws.
- politician 14y ago
  Doing both is also possible...
- calibraxis 14y ago
  The problem is that it takes effort to make good comparative benchmarks — for a platform you're not totally immersed in, you'd ideally speak with the community about your methods. Furthermore, good benchmarks require real investigation explaining the differences; otherwise you could be measuring something silly. And you should graph some of the stats by time. Maybe in an ideal world, quality benchmarks would be commonplace. But in this world, doing good benchmarks in response to everyone's bad benchmarks is a heavy burden.
- pyre 14y ago
  So what you're really saying is that no one is 'allowed' to make a critical remark unless they are willing to stand up and do better themselves?
- cantankerous 14y ago
  Writing good benchmarks is hard. Spotting bad ones is easier. I personally am glad that it's this way. Otherwise, we'd be taking bunk benchmarks at face value because we can't tell for ourselves.
- SkyMarshal 14y ago
  Problem of induction - for theories that cannot be logically or mathematically proven, and can only be confirmed by empirical evidence, no amount of confirming evidence can prove the theory true, but one single refuting observation can disprove it. So maybe the study's author only cared about webbit, ws4py, and Snap, and that was the implicit constraint of his test. If so, that's fine, and the submitter mangled the author's intent with an overly-editorialized, senationalized title. But if the author was actually using webbit, ws4py, and Snap as proxies for all of Java, Python, and Haskell, then no one should be surprised that it gets quickly shot down. There's value in critics debunking BS quickly, even if they don't provide a better alternative. Absence of bad knowledge is better than presence of it.
duaneb 14y ago
I don't see why it's not the critics' onus to produce better benchmarks.
- cantankerous 14y ago
  That's like one posting a bad argument on here, another saying it's a bad argument, then the first saying it's not a bad argument because the second hasn't supplied a better argument to replace the first's argument. Nobody has to accept this benchmark as indicative of anything if they don't find it robust enough.
- amalcon 14y ago
  The burden of proof is always on the one making the claim, not on the one refuting it. See Russell's Teapot, et al.

davidw 14y ago

> The onus is on the study's author to make it comprehensive and thorough, not the critics. It's always possible for the critics to go out there and do something better, rather than limiting themselves to pointing out flaws.

politician 14y ago
Doing both is also possible...
calibraxis 14y ago
The problem is that it takes effort to make good comparative benchmarks — for a platform you're not totally immersed in, you'd ideally speak with the community about your methods. Furthermore, good benchmarks require real investigation explaining the differences; otherwise you could be measuring something silly. And you should graph some of the stats by time. Maybe in an ideal world, quality benchmarks would be commonplace. But in this world, doing good benchmarks in response to everyone's bad benchmarks is a heavy burden.
pyre 14y ago
So what you're really saying is that no one is 'allowed' to make a critical remark unless they are willing to stand up and do better themselves?
cantankerous 14y ago
Writing good benchmarks is hard. Spotting bad ones is easier. I personally am glad that it's this way. Otherwise, we'd be taking bunk benchmarks at face value because we can't tell for ourselves.
SkyMarshal 14y ago
Problem of induction - for theories that cannot be logically or mathematically proven, and can only be confirmed by empirical evidence, no amount of confirming evidence can prove the theory true, but one single refuting observation can disprove it. So maybe the study's author only cared about webbit, ws4py, and Snap, and that was the implicit constraint of his test. If so, that's fine, and the submitter mangled the author's intent with an overly-editorialized, senationalized title. But if the author was actually using webbit, ws4py, and Snap as proxies for all of Java, Python, and Haskell, then no one should be surprised that it gets quickly shot down. There's value in critics debunking BS quickly, even if they don't provide a better alternative. Absence of bad knowledge is better than presence of it.

duaneb 14y ago

I don't see why it's not the critics' onus to produce better benchmarks.

cantankerous 14y ago
That's like one posting a bad argument on here, another saying it's a bad argument, then the first saying it's not a bad argument because the second hasn't supplied a better argument to replace the first's argument. Nobody has to accept this benchmark as indicative of anything if they don't find it robust enough.
amalcon 14y ago
The burden of proof is always on the one making the claim, not on the one refuting it. See Russell's Teapot, et al.