3 ms·

but the benchmark is stupid: https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts https://github.com/alangpierce/sucrase/blob/main/benchmark/

by merb 4y ago

but the benchmark is stupid: https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts https://github.com/alangpierce/sucrase/blob/main/benchmark/b... > Like all JavaScript code run in V8, Sucrase runs more slowly at first, then gets faster as the just-in-time compiler applies more optimizations. From a rough measurement, Sucrase is about 2x faster after running for 3 seconds than after running for 1 second. swc (written in Rust) and esbuild (written in Go) don't have this effect because they're pre-compiled, so comparing them with Sucrase gets significantly different results depending on how large of a codebase is being tested and whether each compiler is allowed a "warm up" period before the benchmark is run. (worse it disables esbuild and swc's multi-threading... https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts#L257 https://github.com/alangpierce/sucrase/blob/main/benchmark/b... https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts#L231 https://github.com/alangpierce/sucrase/blob/main/benchmark/b...) fake it till ya make it. it's like saying "if I disable everything and wait for 5 minutes it's faster"

RyanCavanaugh 4y ago
I don't see how this is a rebuttal to the claim. 636,975 is more than 2x 304,526, so assuming the quoted paragraph is correct, sucrase is still the highest-throughput transpiler even during its warm-up phase. Probably this isn't true for the first 100 milliseconds of execution during the very first warm-up stages, but if the transpile phase is that short, it's basically irrelevant anyway.
- merb 4y ago
  the warm-up phase is the whole reason why esbuild and swc exists btw. and also the sample is basically a small hello world. any real project would do a little more and the jit would not optimize as much. also 2x 300k is not really the way how multi-threading/concurrency/parallelism works... especially not golang, which should not be run with GOMAXPROCS=1....
  - cxr 4y ago
    As far as I can follow your argument, it seems to be that the native tools perform better on the small inputs (that only ever take a second or three to complete), and that the creators ("maintainers"?) of Sucrase have their thumb on the scale or something, since they emphasize in their results the very large inputs that will involve longer running jobs where their tool has had a chance to warm up. In other words, the tools you're defending only do better when it doesn't really matter, and the tool you're downplaying outperforms the competition on the jobs where it actually does matter. If anything is backwards (or "stupid", as you put it) then that seems to be it. Maybe I'm misunderstanding. In that case, please provide a better benchmark.
alangpierce 4y ago
Hi, Sucrase author here. To be clear, the benchmark in the README does not allow JIT warm-up. The Sucrase numbers would be better if it did. From testing just now (add `warmUp: true` to `benchmarkJest`), Sucrase is a little over 3x faster than swc if you allow warm-up, but it seemed unfair to disregard warm-up for the comparison in the README. It's certainly fair to debate whether 360k lines of code is a realistic codebase size for the benchmark; the higher-scale the test case, the better Sucrase looks. > worse it disables esbuild and swc's multi-threading At some point I'm hoping to update the README benchmark to run all tools in parallel, which should be more convincing despite the added variability: https://github.com/alangpierce/sucrase/issues/730 https://github.com/alangpierce/sucrase/issues/730 . In an ideal environment, the results are pretty much the same as a per-core benchmark, but I do expect that Node's parallelism overhead and the JIT warm-up cost across many cores would make Sucrase less competitive than the current numbers.

by merb 4y ago

RyanCavanaugh 4y ago

I don't see how this is a rebuttal to the claim. 636,975 is more than 2x 304,526, so assuming the quoted paragraph is correct, sucrase is still the highest-throughput transpiler even during its warm-up phase. Probably this isn't true for the first 100 milliseconds of execution during the very first warm-up stages, but if the transpile phase is that short, it's basically irrelevant anyway.

merb 4y ago
the warm-up phase is the whole reason why esbuild and swc exists btw. and also the sample is basically a small hello world. any real project would do a little more and the jit would not optimize as much. also 2x 300k is not really the way how multi-threading/concurrency/parallelism works... especially not golang, which should not be run with GOMAXPROCS=1....
- cxr 4y ago
  As far as I can follow your argument, it seems to be that the native tools perform better on the small inputs (that only ever take a second or three to complete), and that the creators ("maintainers"?) of Sucrase have their thumb on the scale or something, since they emphasize in their results the very large inputs that will involve longer running jobs where their tool has had a chance to warm up. In other words, the tools you're defending only do better when it doesn't really matter, and the tool you're downplaying outperforms the competition on the jobs where it actually does matter. If anything is backwards (or "stupid", as you put it) then that seems to be it. Maybe I'm misunderstanding. In that case, please provide a better benchmark.

alangpierce 4y ago

Hi, Sucrase author here. To be clear, the benchmark in the README does not allow JIT warm-up. The Sucrase numbers would be better if it did. From testing just now (add `warmUp: true` to `benchmarkJest`), Sucrase is a little over 3x faster than swc if you allow warm-up, but it seemed unfair to disregard warm-up for the comparison in the README. It's certainly fair to debate whether 360k lines of code is a realistic codebase size for the benchmark; the higher-scale the test case, the better Sucrase looks. > worse it disables esbuild and swc's multi-threading At some point I'm hoping to update the README benchmark to run all tools in parallel, which should be more convincing despite the added variability: https://github.com/alangpierce/sucrase/issues/730 https://github.com/alangpierce/sucrase/issues/730 . In an ideal environment, the results are pretty much the same as a per-core benchmark, but I do expect that Node's parallelism overhead and the JIT warm-up cost across many cores would make Sucrase less competitive than the current numbers.