7 ms·

My best guess: probably written by someone that doeant know how to write performance TS. I say this with no skin in the game. I write neither JS nor TS. What I

My best guess: probably written by someone that doeant know how to write performance TS. I say this with no skin in the game. I write neither JS nor TS. What I have observed in language benchmarks over the years, is that the benchmarks are rarely written by an expert, but usually by someone with cursory knowledge of the language. E.g. just enough to be dangerous. Often times, these sorts of benchmarks are done with prejudice (not necessarily malice). The benchmarks are written by someone with something to prove: my chosen tech stack performs better, and let me show you why. A favorite of mine is Perl vs Python comparisons, where you see an idiomatic Perl implementation vs a non idiomatic Python implementation (or other way around). Typically in a head-to-head comparison, the benchmarks are developed by the same individual whom likely has above average knowledge in their favorite and below average in the target they're trying to show as inferior. You'll see this time and time again in internet benchmarks comparing performance. Unless you can see the code from all benchmarks involved, my suggestion is to avoid them. I mean, for all I know, the author of the benchmark was unaware of the built in sort and instead bubble sorted.

hopler 8y ago
I thought the benchmark game is set up so every language's advocates can tune their language's programs. The only chance at a fair comparison is if every language gets the best implementation it can find for the challenges. There's nothing else that approaches fair comparison of apples and oranges.
dahart 8y ago
This is unfortunately pure speculation on top of pure speculation, which is the problem I have with the top comment. You’re assuming incompetence when you could just go look it up. Why assume it’s someone who doesn’t know? Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad, when you admit to having zero idea what the actual specific problem here is? The test that lowered TypeScript’s score in the paper is called fannkuch-redux, and here are the sources in question: https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/fannkuch-redux/fannkuchredux.node-4.node https://github.com/greensoftwarelab/Energy-Languages/blob/ma... https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/fannkuch-redux/fannkuchredux.ts https://github.com/greensoftwarelab/Energy-Languages/blob/ma... They are both contributed by the same person, and there is no bubble sort involved. So now you know. I don’t see an obvious reason one would be slower, but they’re also quite different. Maybe the algorithmic complexity is different. Maybe the cross-compilation is doing something bad with memory allocation. Note the input sizes for this test are very small, it would be easy for a difference in temporary variables the compiler injects to cause a serious problem. What is not obvious is any prejudice, malice, or incompetence.
- kbenson 8y ago
  > Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad What are you talking about? Where did that happen? > when you admit to having zero idea what the actual specific problem here is? This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to. > What is not obvious is any prejudice, malice, or incompetence. Please stop. Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017. 1: https://web.archive.org/web/20170901020804/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170901020804/http://benchmarks... 2: https://web.archive.org/web/20170918163900/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170918163900/http://benchmarks...
  - dahart 8y ago
    > What are you talking about? Where did that happen? I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057 https://news.ycombinator.com/item?id=19527057 "You'll see this time and time again in internet benchmarks comparing performance." > It is perfectly valid to bring up a question that you don't know the answer to. I agree. It's a bummer that's not really what happened here. >> What is not obvious is any prejudice, malice, or incompetence. > Please stop. The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." If you'd like me not to call out speculation, then please assume good faith and don't speculate next time. > From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?
deleted 8y ago
[deleted]

hopler 8y ago

I thought the benchmark game is set up so every language's advocates can tune their language's programs. The only chance at a fair comparison is if every language gets the best implementation it can find for the challenges. There's nothing else that approaches fair comparison of apples and oranges.

dahart 8y ago

This is unfortunately pure speculation on top of pure speculation, which is the problem I have with the top comment. You’re assuming incompetence when you could just go look it up. Why assume it’s someone who doesn’t know? Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad, when you admit to having zero idea what the actual specific problem here is? The test that lowered TypeScript’s score in the paper is called fannkuch-redux, and here are the sources in question: https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/fannkuch-redux/fannkuchredux.node-4.node https://github.com/greensoftwarelab/Energy-Languages/blob/ma... https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/fannkuch-redux/fannkuchredux.ts https://github.com/greensoftwarelab/Energy-Languages/blob/ma... They are both contributed by the same person, and there is no bubble sort involved. So now you know. I don’t see an obvious reason one would be slower, but they’re also quite different. Maybe the algorithmic complexity is different. Maybe the cross-compilation is doing something bad with memory allocation. Note the input sizes for this test are very small, it would be easy for a difference in temporary variables the compiler injects to cause a serious problem. What is not obvious is any prejudice, malice, or incompetence.

kbenson 8y ago
> Why use that to wander off into rant land about prejudices and make broad claims that internet benchmarks are bad What are you talking about? Where did that happen? > when you admit to having zero idea what the actual specific problem here is? This is the comment section for a submission about an article referencing the paper. I brought it up for discussion. It is perfectly valid to bring up a question that you don't know the answer to. > What is not obvious is any prejudice, malice, or incompetence. Please stop. Edit: From another comment, and some deeper digging of my own from that, you might find the archived results of the fannkuch-redux interesting. From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. The paper was released in October 2017. 1: https://web.archive.org/web/20170901020804/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170901020804/http://benchmarks... 2: https://web.archive.org/web/20170918163900/http://benchmarksgame.alioth.debian.org/u64q/typescript.html https://web.archive.org/web/20170918163900/http://benchmarks...
- dahart 8y ago
  > What are you talking about? Where did that happen? I was responding directly to @hermitdev. Did you get your threads crossed? What I'm talking about happened immediately above in the parent comment, beginning with "Often times, these sorts of benchmarks are done with prejudice" https://news.ycombinator.com/item?id=19527057 https://news.ycombinator.com/item?id=19527057 "You'll see this time and time again in internet benchmarks comparing performance." > It is perfectly valid to bring up a question that you don't know the answer to. I agree. It's a bummer that's not really what happened here. >> What is not obvious is any prejudice, malice, or incompetence. > Please stop. The parent comment explicitly stated an assumption of both incompetence and prejudice and I responded directly to that. From the HN guidelines: "Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith." If you'd like me not to call out speculation, then please assume good faith and don't speculate next time. > From 2017-08-01[1] to 2017-09-18[2], the benchmark changed from a running time of 1,204.93 second to a running time of 131.39 seconds. Yes! Now we are getting somewhere. It appears that would change the outcome of the paper. Perhaps it was a mistake. That might mean it was nothing more than an oversight that already got fixed. It doesn't mean there is any other coloring of the study at all, nor that there was any intention or agenda to make TypeScript look bad, right?

deleted 8y ago

[deleted]