4 ms·
> For our program to have taken "several seconds, at a minimum" to run would have been a failure case, so there would have been no value in benchmarking that, l
by coder543 6y ago
> For our program to have taken "several seconds, at a minimum" to run would have been a failure case, so there would have been no value in benchmarking that, let alone optimizing for it. I expect the use cases are the same for this project, which is supposed to be processing untrusted scriptlets that need to finish in the smallest amount of wall clock time possible.
If the goal is to compare to NodeJS and Deno, which are largely used to write long-lived servers, then I disagree entirely.
Even if the main use case for Elsa were to build Lambda-esque services, I would again disagree. Services like that gain significant benefit from reusing a single instance of the program, rather than creating it from scratch over and over again. Even without a JIT, parsing the program over and over is a waste of time, and reusing an instance allows you to amortize that cost (and the cost of JIT) across a number of uses, rather than paying that price every time.
The benchmarks provided are extremely short, such as the one-liner benchmark I mentioned before, so for all we know, QuickJS is slower at parsing JavaScript than V8 is, and it would be slower at running "real" programs than V8, even though V8 has a JIT to deal with. This is without mentioning that V8 has a JIT-less mode, which the benchmarks aren't comparing against, even though it might be more comparable.
The only use case for an 8 millisecond benchmark with startup penalty every time is interactive CLI tools, as you alluded to, and this is far from the expected use case I see described in the project.
But even then... the benchmarks showed that the startup time and total performance for V8 were comparable on the human scale. The difference between 8 milliseconds and 36 milliseconds is completely irrelevant to humans. It isn't like V8 is taking 10 seconds to initialize or something. V8 is just running slower for a short period of time as it learns about what it is executing. If the CLI tool were doing more than a non-trivial amount of work, the V8 JIT would begin to accelerate past QuickJS.
So, the benchmarks are irrelevant for the CLI use case, and irrelevant for seemingly every use case that has been presented. If the benchmarks were showing that V8 could not complete these tasks in a reasonable amount of time, then they might be showing something useful.
These benchmarks are measuring the use case of interactive CLI tools that are written in a single line of JavaScript... which is a very specific use case that I don't believe is intended to be the target market of the project.
I appreciate your anecdote, but this isn't my first time thinking about benchmarks, and these benchmarks seem pretty seriously flawed to me. This is why I made my first comment, and your comment has done nothing to negate any of the reasoning that went into my first comment.
- cxr 6y ago> your comment has done nothing to negate any of the reasoning that went into my first comment It's not clear to me that I'm supposed to have done that or why, but on that topic: likewise.