5 ms·

This paper is very bad. I wish people would stop citing it. There’s serious methodological flaws. And yes I say that even though Rust performs very well in it.

coldtea 4y ago
"There’s serious methodological flaws" such as?
- steveklabnik 4y ago
  The largest one is that it is measuring benchmark game programs. There is no reason to believe that those style of programs are representative of all programs. Furthermore, there’s very weird results as well. Why is TypeScript so much worse than JavaScript? TypeScript is a superset of JavaScript and compiles to it. The JavaScript program could be exactly the same as the TypeScript one, but isn’t for some reason. It’s been a while and I’m on my phone so I can’t check, but IIRC the reason the above is even a question is that they made a benchmarking framework but didn’t share it, so nobody can reproduce their results.
  - vorpalhex 4y ago
    > Why is TypeScript so much worse than JavaScript? TypeScript is a superset of JavaScript and compiles to it Typescript produces garbage JS output that creates all sorts of weird bugs, hiccups and strange quirks. It doesn't surprise me that TS generated JS would perform much worse than directly written JS.
    - steveklabnik 4y ago
      Even if you take a JavaScript file and run it through tsc, it adds stuff? That is indeed surprising to me! It wouldn’t shock me that there were differences if you used a bunch of fancy TypeScript features, but at the limit I would expect no overhead if you don’t use any of them. Maybe that expectation is misguided.
    - rcme 4y ago
      It depends on your compiler settings but the code can very closely match equivalent JS if you want.
  - Rochus 4y ago
    These are not "serious methodological flaws"; you do not consider the results credible, which might be justified or not, it's an opinion.
misnome 4y ago
We had a session at a recent biochemistry scientific software conference where someone was spending a lot of time in “more of a comment than a question” questions pushing that this paper meant that everyone should stop writing python and go back to Fortran.
- deleted 4y ago
  [deleted]
Rochus 4y ago
I cannot confirm your assessment. The paper has some little flaws - as all papers do - but the method and results are interesting and relevant. As with all performance measurements - or measurements in general - there is a measurement error. As engineers we have learned to cope with such errors.

coldtea 4y ago

"There’s serious methodological flaws" such as?

steveklabnik 4y ago
The largest one is that it is measuring benchmark game programs. There is no reason to believe that those style of programs are representative of all programs. Furthermore, there’s very weird results as well. Why is TypeScript so much worse than JavaScript? TypeScript is a superset of JavaScript and compiles to it. The JavaScript program could be exactly the same as the TypeScript one, but isn’t for some reason. It’s been a while and I’m on my phone so I can’t check, but IIRC the reason the above is even a question is that they made a benchmarking framework but didn’t share it, so nobody can reproduce their results.
- vorpalhex 4y ago
  > Why is TypeScript so much worse than JavaScript? TypeScript is a superset of JavaScript and compiles to it Typescript produces garbage JS output that creates all sorts of weird bugs, hiccups and strange quirks. It doesn't surprise me that TS generated JS would perform much worse than directly written JS.
  - steveklabnik 4y ago
    Even if you take a JavaScript file and run it through tsc, it adds stuff? That is indeed surprising to me! It wouldn’t shock me that there were differences if you used a bunch of fancy TypeScript features, but at the limit I would expect no overhead if you don’t use any of them. Maybe that expectation is misguided.
  - rcme 4y ago
    It depends on your compiler settings but the code can very closely match equivalent JS if you want.
- Rochus 4y ago
  These are not "serious methodological flaws"; you do not consider the results credible, which might be justified or not, it's an opinion.

misnome 4y ago

We had a session at a recent biochemistry scientific software conference where someone was spending a lot of time in “more of a comment than a question” questions pushing that this paper meant that everyone should stop writing python and go back to Fortran.

deleted 4y ago
[deleted]

Rochus 4y ago

I cannot confirm your assessment. The paper has some little flaws - as all papers do - but the method and results are interesting and relevant. As with all performance measurements - or measurements in general - there is a measurement error. As engineers we have learned to cope with such errors.