3 ms·

> Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad What are you talking about? Where did that

> Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad What are you talking about? Where did that happen? > when you admit to having zero idea what the actual specific problem here is? This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to. > What is not obvious is any prejudice, malice, or incompetence. Please stop. Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017. 1: https://web.archive.org/web/20170901020804/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170901020804/http://benchmarks... 2: https://web.archive.org/web/20170918163900/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170918163900/http://benchmarks...

dahart 8y ago
> What are you talking about? Where did that happen? I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057 https://news.ycombinator.com/item?id=19527057 "You'll see this time and time again in internet benchmarks comparing performance." > It is perfectly valid to bring up a question that you don't know the answer to. I agree. It's a bummer that's not really what happened here. >> What is not obvious is any prejudice, malice, or incompetence. > Please stop. The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." If you'd like me not to call out speculation, then please assume good faith and don't speculate next time. > From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
- kbenson 8y ago
  > The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. Perhaps I misinterpreted what you said. You started the paragraph referring to the top level comment, which is me. I took the "you're" in "You’re assuming incompetence when you could just go look it up." to be a general "you", and commentary on my original comment. > From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." I actually looked this up before the GP comment, and almost included it myself. I can see now that you were implicating the comment you replied to. I didn't think that was the case, because I apparently didn't interpret that comment remotely in the same way you did. > Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right? I never implied it was. For that matter, I didn't really interpret the comment in question as stating that either. The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. That doesn't require purposefully tanking one benchmark, it just requires them to be much better versed in optimizing one language than another and a lack of awareness about this. As they say, never attribute to malice what can be explained by incompetence. In fact, if you read the comment carefully, they even call out to this with the "not necessarily malice" remark.
  - dahart 8y ago
    > The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. The project doesn’t talk about favorites or seem to want to make certain languages look good. Jumping to the conclusion that bias is involved isn’t the good faith interpretation, even if you state with a positive sounding framing. The good faith interpretation is to take the stated project goals at face value, and assume that the participants have done a good job.
    - kbenson 8y ago
deleted 8y ago
[deleted]
igouy 8y ago
Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program? 5 July, Node 8.1.3, TypeScript 2.4.1 https://web.archive.org/web/20170715120038/http://benchmarksgame.alioth.debian.org/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170715120038/http://benchmarks... 1 Sep, Node 8.4.0, TypeScript 2.5.2 https://web.archive.org/web/20170922144419/http://benchmarksgame.alioth.debian.org:80/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170922144419/http://benchmarks... ---- How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?
- kbenson 8y ago
  > Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program? Yes. > How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment? The suspicion was incorrect. That's why it was presented as a suspicion, not as fact. I have no reason to defend it if it's incorrect, but I still defend that it was valid to raise questions, given the facts on the ground. We've now shown there was something that changed very drastically at that time, and while it's less likely it's the benchmarks themselves (unless one or both of those are fairly out of date Node versions)[1], it still points towards something to be aware of in the results presented. Namely, they rely on a lot of underlying assumptions which should be looked at if you care about the numbers. 1: Also, I imagine the V8 devs probably considered the performance of TypeScript in that case to be a bug, given how horrible the performance regression from JavaScript is and that it's still javaScript running. It's possible that TypeScript was doing something really odd, but given the exposure and Microsoft's backing and developer time, I think that's a less likely scenario than some optimization that should have been triggered was missing, which happens quite often.
  - igouy 8y ago
    Please add a correction to your original comment, to prevent readers from being misled. (If it's closed to edits, I'm sure HN staff will open it when you ask). > I still defend that it was valid to raise questions Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause; and, without evidence, implied that assumed cause led to widespread problems with the analysis. In other words -- innuendo.

dahart 8y ago

> What are you talking about? Where did that happen? I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057 https://news.ycombinator.com/item?id=19527057 "You'll see this time and time again in internet benchmarks comparing performance." > It is perfectly valid to bring up a question that you don't know the answer to. I agree. It's a bummer that's not really what happened here. >> What is not obvious is any prejudice, malice, or incompetence. > Please stop. The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." If you'd like me not to call out speculation, then please assume good faith and don't speculate next time. > From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?

kbenson 8y ago
> The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. Perhaps I misinterpreted what you said. You started the paragraph referring to the top level comment, which is me. I took the "you're" in "You’re assuming incompetence when you could just go look it up." to be a general "you", and commentary on my original comment. > From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." I actually looked this up before the GP comment, and almost included it myself. I can see now that you were implicating the comment you replied to. I didn't think that was the case, because I apparently didn't interpret that comment remotely in the same way you did. > Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right? I never implied it was. For that matter, I didn't really interpret the comment in question as stating that either. The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. That doesn't require purposefully tanking one benchmark, it just requires them to be much better versed in optimizing one language than another and a lack of awareness about this. As they say, never attribute to malice what can be explained by incompetence. In fact, if you read the comment carefully, they even call out to this with the "not necessarily malice" remark.
- dahart 8y ago
  > The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. The project doesn’t talk about favorites or seem to want to make certain languages look good. Jumping to the conclusion that bias is involved isn’t the good faith interpretation, even if you state with a positive sounding framing. The good faith interpretation is to take the stated project goals at face value, and assume that the participants have done a good job.
  - kbenson 8y ago

deleted 8y ago

[deleted]

igouy 8y ago

Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program? 5 July, Node 8.1.3, TypeScript 2.4.1 https://web.archive.org/web/20170715120038/http://benchmarksgame.alioth.debian.org/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170715120038/http://benchmarks... 1 Sep, Node 8.4.0, TypeScript 2.5.2 https://web.archive.org/web/20170922144419/http://benchmarksgame.alioth.debian.org:80/u64q/program.php?test=fannkuchredux&lang=typescript&id=2 https://web.archive.org/web/20170922144419/http://benchmarks... ---- How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment?

kbenson 8y ago
> Do you agree that those very different times were measurements of the same TypeScript fannkuch-redux program? Yes. > How should we now assess your "suspiciously like entirely different algorithms were used in each implementation" comment? The suspicion was incorrect. That's why it was presented as a suspicion, not as fact. I have no reason to defend it if it's incorrect, but I still defend that it was valid to raise questions, given the facts on the ground. We've now shown there was something that changed very drastically at that time, and while it's less likely it's the benchmarks themselves (unless one or both of those are fairly out of date Node versions)[1], it still points towards something to be aware of in the results presented. Namely, they rely on a lot of underlying assumptions which should be looked at if you care about the numbers. 1: Also, I imagine the V8 devs probably considered the performance of TypeScript in that case to be a bug, given how horrible the performance regression from JavaScript is and that it's still javaScript running. It's possible that TypeScript was doing something really odd, but given the exposure and Microsoft's backing and developer time, I think that's a less likely scenario than some optimization that should have been triggered was missing, which happens quite often.
- igouy 8y ago
  Please add a correction to your original comment, to prevent readers from being misled. (If it's closed to edits, I'm sure HN staff will open it when you ask). > I still defend that it was valid to raise questions Of course, it's valid to question a measurement that looks strange but your comment went further than that -- your comment, without evidence, assumed a cause; and, without evidence, implied that assumed cause led to widespread problems with the analysis. In other words -- innuendo.