5 ms·

Each of the benchmarks seem to take a single-digit number of milliseconds in Elsa/QuickJS. My guess is that these benchmarks were chosen because the startup ti

Each of the benchmarks seem to take a single-digit number of milliseconds in Elsa/QuickJS. My guess is that these benchmarks were chosen because the startup time for QuickJS is probably less than the startup time for V8, but I struggle to imagine that V8 is slower once the code has warmed up. Any meaningful benchmark with JIT runtimes should take several seconds, at a minimum, in my opinion. 8 milliseconds is not enough. From what I can tell, the benchmark is including the startup time, and it isn't keeping the interpreter warm -- it starts a fresh instance for each iteration of the benchmark. This is one of the benchmarks: https://github.com/elsaland/elsa/blob/master/testing/bench_scripts/fs.js https://github.com/elsaland/elsa/blob/master/testing/bench_s... Literally a single line of JavaScript.

mkl 6y ago
QuickJS's own benchmarks are here: https://bellard.org/quickjs/bench.html https://bellard.org/quickjs/bench.html V8, with or without its JIT, wins hands down. QuickJS may be small and fast to start, but fast actual computations seem not to be its strength (or intended target). Big discussion of QuickJS last year: https://news.ycombinator.com/item?id=20411154 https://news.ycombinator.com/item?id=20411154
Waterluvian 6y ago
It’s amazing watching V8s JIT warm up. And eye opening on how to measure performance when profiling. A heavy call can take 200ms, then 100, then 80, 70. And suddenly where I thought the low hanging fruit is has vanished.
cxr 6y ago
> Any meaningful benchmark with JIT runtimes should take several seconds, at a minimum, in my opinion. 8 milliseconds is not enough. Maybe a benchmark for JIT runtimes, but you should always keep in mind what the expected use case is. I recently ported a program from lang X to lang Y. It's a command-line program that you're expected to invoke in your terminal, and it processes some input and then spits something out (not unlike, say, a compiler or a static site generator) but it's expected to be used "interactively" i.e. no significant pauses for processing typical workloads, just like you expect from invoking mkdir or ls. Zero percent of the effort during porting was spent on optimization. In fact, the lang X version uses specialized, algorithmically-sophisticated collection classes, and the lang Y version does the dumbest possible thing. Despite this, for typical workloads, the Y version completes sooner than it takes the X version to do the same. No amount of benchmarking in the world is going to change the fact that the dumb Y version is faster than the clever X version. For our program to have taken "several seconds, at a minimum" to run would have been a failure case, so there would have been no value in benchmarking that, let alone optimizing for it. I expect the use cases are the same for this project, which is supposed to be processing untrusted scriptlets that need to finish in the smallest amount of wall clock time possible.
- coder543 6y ago
  > For our program to have taken "several seconds, at a minimum" to run would have been a failure case, so there would have been no value in benchmarking that, let alone optimizing for it. I expect the use cases are the same for this project, which is supposed to be processing untrusted scriptlets that need to finish in the smallest amount of wall clock time possible. If the goal is to compare to NodeJS and Deno, which are largely used to write long-lived servers, then I disagree entirely. Even if the main use case for Elsa were to build Lambda-esque services, I would again disagree. Services like that gain significant benefit from reusing a single instance of the program, rather than creating it from scratch over and over again. Even without a JIT, parsing the program over and over is a waste of time, and reusing an instance allows you to amortize that cost (and the cost of JIT) across a number of uses, rather than paying that price every time. The benchmarks provided are extremely short, such as the one-liner benchmark I mentioned before, so for all we know, QuickJS is slower at parsing JavaScript than V8 is, and it would be slower at running "real" programs than V8, even though V8 has a JIT to deal with. This is without mentioning that V8 has a JIT-less mode, which the benchmarks aren't comparing against, even though it might be more comparable. The only use case for an 8 millisecond benchmark with startup penalty every time is interactive CLI tools, as you alluded to, and this is far from the expected use case I see described in the project. But even then... the benchmarks showed that the startup time and total performance for V8 were comparable on the human scale. The difference between 8 milliseconds and 36 milliseconds is completely irrelevant to humans. It isn't like V8 is taking 10 seconds to initialize or something. V8 is just running slower for a short period of time as it learns about what it is executing. If the CLI tool were doing more than a non-trivial amount of work, the V8 JIT would begin to accelerate past QuickJS. So, the benchmarks are irrelevant for the CLI use case, and irrelevant for seemingly every use case that has been presented. If the benchmarks were showing that V8 could not complete these tasks in a reasonable amount of time, then they might be showing something useful. These benchmarks are measuring the use case of interactive CLI tools that are written in a single line of JavaScript... which is a very specific use case that I don't believe is intended to be the target market of the project. I appreciate your anecdote, but this isn't my first time thinking about benchmarks, and these benchmarks seem pretty seriously flawed to me. This is why I made my first comment, and your comment has done nothing to negate any of the reasoning that went into my first comment.

mkl 6y ago

QuickJS's own benchmarks are here: https://bellard.org/quickjs/bench.html https://bellard.org/quickjs/bench.html V8, with or without its JIT, wins hands down. QuickJS may be small and fast to start, but fast actual computations seem not to be its strength (or intended target). Big discussion of QuickJS last year: https://news.ycombinator.com/item?id=20411154 https://news.ycombinator.com/item?id=20411154

Waterluvian 6y ago

It’s amazing watching V8s JIT warm up. And eye opening on how to measure performance when profiling. A heavy call can take 200ms, then 100, then 80, 70. And suddenly where I thought the low hanging fruit is has vanished.

cxr 6y ago

> Any meaningful benchmark with JIT runtimes should take several seconds, at a minimum, in my opinion. 8 milliseconds is not enough. Maybe a benchmark for JIT runtimes, but you should always keep in mind what the expected use case is. I recently ported a program from lang X to lang Y. It's a command-line program that you're expected to invoke in your terminal, and it processes some input and then spits something out (not unlike, say, a compiler or a static site generator) but it's expected to be used "interactively" i.e. no significant pauses for processing typical workloads, just like you expect from invoking mkdir or ls. Zero percent of the effort during porting was spent on optimization. In fact, the lang X version uses specialized, algorithmically-sophisticated collection classes, and the lang Y version does the dumbest possible thing. Despite this, for typical workloads, the Y version completes sooner than it takes the X version to do the same. No amount of benchmarking in the world is going to change the fact that the dumb Y version is faster than the clever X version. For our program to have taken "several seconds, at a minimum" to run would have been a failure case, so there would have been no value in benchmarking that, let alone optimizing for it. I expect the use cases are the same for this project, which is supposed to be processing untrusted scriptlets that need to finish in the smallest amount of wall clock time possible.

coder543 6y ago
> For our program to have taken "several seconds, at a minimum" to run would have been a failure case, so there would have been no value in benchmarking that, let alone optimizing for it. I expect the use cases are the same for this project, which is supposed to be processing untrusted scriptlets that need to finish in the smallest amount of wall clock time possible. If the goal is to compare to NodeJS and Deno, which are largely used to write long-lived servers, then I disagree entirely. Even if the main use case for Elsa were to build Lambda-esque services, I would again disagree. Services like that gain significant benefit from reusing a single instance of the program, rather than creating it from scratch over and over again. Even without a JIT, parsing the program over and over is a waste of time, and reusing an instance allows you to amortize that cost (and the cost of JIT) across a number of uses, rather than paying that price every time. The benchmarks provided are extremely short, such as the one-liner benchmark I mentioned before, so for all we know, QuickJS is slower at parsing JavaScript than V8 is, and it would be slower at running "real" programs than V8, even though V8 has a JIT to deal with. This is without mentioning that V8 has a JIT-less mode, which the benchmarks aren't comparing against, even though it might be more comparable. The only use case for an 8 millisecond benchmark with startup penalty every time is interactive CLI tools, as you alluded to, and this is far from the expected use case I see described in the project. But even then... the benchmarks showed that the startup time and total performance for V8 were comparable on the human scale. The difference between 8 milliseconds and 36 milliseconds is completely irrelevant to humans. It isn't like V8 is taking 10 seconds to initialize or something. V8 is just running slower for a short period of time as it learns about what it is executing. If the CLI tool were doing more than a non-trivial amount of work, the V8 JIT would begin to accelerate past QuickJS. So, the benchmarks are irrelevant for the CLI use case, and irrelevant for seemingly every use case that has been presented. If the benchmarks were showing that V8 could not complete these tasks in a reasonable amount of time, then they might be showing something useful. These benchmarks are measuring the use case of interactive CLI tools that are written in a single line of JavaScript... which is a very specific use case that I don't believe is intended to be the target market of the project. I appreciate your anecdote, but this isn't my first time thinking about benchmarks, and these benchmarks seem pretty seriously flawed to me. This is why I made my first comment, and your comment has done nothing to negate any of the reasoning that went into my first comment.