4 ms·
This is unfortunately pure speculation on top of pure speculation, which is the problem I have with the top comment. You’re assuming incompetence when you could
by dahart 8y ago
This is unfortunately pure speculation on top of pure speculation, which is the problem I have with the top comment. You’re assuming incompetence when you could just go look it up. Why assume it’s someone who doesn’t know? Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad, when you admit to having zero idea what the actual specific problem here is?
The test that lowered TypeScript’s score in the paper is called fannkuch-redux, and here are the sources in question:
https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/fannkuch-redux/fannkuchredux.node-4.node https://github.com/greensoftwarelab/Energy-Languages/blob/ma...
https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/fannkuch-redux/fannkuchredux.ts https://github.com/greensoftwarelab/Energy-Languages/blob/ma...
They are both contributed by the same person, and there is no bubble sort involved. So now you know.
I don’t see an obvious reason one would be slower, but they’re also quite different. Maybe the algorithmic complexity is different. Maybe the cross-compilation is doing something bad with memory allocation. Note the input sizes for this test are very small, it would be easy for a difference in temporary variables the compiler injects to cause a serious problem.
What is not obvious is any prejudice, malice, or incompetence.
- kbenson 8y ago> Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad What are you talking about? Where did that happen? > when you admit to having zero idea what the actual specific problem here is? This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to. > What is not obvious is any prejudice, malice, or incompetence. Please stop. Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017. 1: https://web.archive.org/web/20170901020804/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170901020804/http://benchmarks... 2: https://web.archive.org/web/20170918163900/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170918163900/http://benchmarks...
- dahart 8y ago> What are you talking about? Where did that happen? I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057 https://news.ycombinator.com/item?id=19527057 "You'll see this time and time again in internet benchmarks comparing performance." > It is perfectly valid to bring up a question that you don't know the answer to. I agree. It's a bummer that's not really what happened here. >> What is not obvious is any prejudice, malice, or incompetence. > Please stop. The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." If you'd like me not to call out speculation, then please assume good faith and don't speculate next time. > From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
- kbenson 8y ago> The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. Perhaps I misinterpreted what you said. You started the paragraph referring to the top level comment, which is me. I took the "you're" in "You’re assuming incompetence when you could just go look it up." to be a general "you", and commentary on my original comment. > From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." I actually looked this up before the GP comment, and almost included it myself. I can see now that you were implicating the comment you replied to. I didn't think that was the case, because I apparently didn't interpret that comment remotely in the same way you did. > Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right? I never implied it was. For that matter, I didn't really interpret the comment in question as stating that either. The more charitable interpretation is not that they are trying to make another language look bad, but that they are trying to make their favorite language look good. That doesn't require purposefully tanking one benchmark, it just requires them to be much better versed in optimizing one language than another and a lack of awareness about this. As they say, never attribute to malice what can be explained by incompetence. In fact, if you read the comment carefully, they even call out to this with the "not necessarily malice" remark.