4 ms·

I'm getting about 2.5 bn op/s, which suggests this is rather down to the JIT's ability to elide microbenchmarking code. Edit: Worth pointing out that the linke

I'm getting about 2.5 bn op/s, which suggests this is rather down to the JIT's ability to elide microbenchmarking code. Edit: Worth pointing out that the linked page runs on an eight year old version of the "benchmarkjs" package, which is described as a robust benchmarking package, but while the linked blog posts discusses some issues with JS benchmarking, how to trick the JIT into actually still running your NOPs is not discussed.

austincheney 7y ago
You make a couple of assumptions and I am not sure they are correct. 1. JIT's ability to elide microbenchmarking code This can be easily debunked using a combination of DOM instructions and taking note of how dramatically the ops/s count drops and then comparing that difference in ops/s against the same difference from another browser like Chrome. If both browsers report wildly different base numbers but similar ratios then there is no cheating for micro-benchmarking numbers. 2. Worth pointing out that the linked page runs on an eight year old version of the "benchmarkjs" package I found a different site that reports similar numbers so either they are using the same old code or the numbers valid without regard for the old code. https://jsbench.me/ https://jsbench.me/
- blattimwind 7y ago
  > If both browsers report wildly different base numbers but similar ratios then there is no cheating for micro-benchmarking numbers. I don't have a lot of experience with JS benchmarking, but for the JVM microbenchmarking is genuinely difficult and relatively worthless because the JIT is really good at killing off huge swaths of code if the results aren't used and side effects aren't observed. And even if you do, it's not unusual for it to optimize your code for the microbenchmark in a way that's not directly applicable to production use. Unless the benchmarking code around this does some transformations on the source, I would expect a JIT (or an optimizing interpreter) to simply remove a side-effect-free statement. I would certainly expect a state-of-the-art JIT that's heavily bound to the DOM implementation to infer "document.getElementById("#foo");" is a NOP. > This can be easily debunked using a combination of DOM instructions and taking note of how dramatically the ops/s count drops and then comparing that difference in ops/s against the same difference from another browser like Chrome. To get back to your particular example, since getElementById in your microbench runs at approximately 100+ times the speed of querySelector (~15 mio op/s, which seems like what you'd expect) I would suspect that querySelector() is not optimized out, because it can actually throw an exception depending on the input.
  - austincheney 7y ago
    Each approach is equally capable of resulting in an error state if unexpected input is supplied, for example pass an array into the document.getElementById. The largest important difference is that the input to the querySelector must be parsed as a coded instruction, a selector, while the input to getElementById is a string literal that does not get parsed.

austincheney 7y ago

You make a couple of assumptions and I am not sure they are correct. 1. JIT's ability to elide microbenchmarking code This can be easily debunked using a combination of DOM instructions and taking note of how dramatically the ops/s count drops and then comparing that difference in ops/s against the same difference from another browser like Chrome. If both browsers report wildly different base numbers but similar ratios then there is no cheating for micro-benchmarking numbers. 2. Worth pointing out that the linked page runs on an eight year old version of the "benchmarkjs" package I found a different site that reports similar numbers so either they are using the same old code or the numbers valid without regard for the old code. https://jsbench.me/ https://jsbench.me/

blattimwind 7y ago
> If both browsers report wildly different base numbers but similar ratios then there is no cheating for micro-benchmarking numbers. I don't have a lot of experience with JS benchmarking, but for the JVM microbenchmarking is genuinely difficult and relatively worthless because the JIT is really good at killing off huge swaths of code if the results aren't used and side effects aren't observed. And even if you do, it's not unusual for it to optimize your code for the microbenchmark in a way that's not directly applicable to production use. Unless the benchmarking code around this does some transformations on the source, I would expect a JIT (or an optimizing interpreter) to simply remove a side-effect-free statement. I would certainly expect a state-of-the-art JIT that's heavily bound to the DOM implementation to infer "document.getElementById("#foo");" is a NOP. > This can be easily debunked using a combination of DOM instructions and taking note of how dramatically the ops/s count drops and then comparing that difference in ops/s against the same difference from another browser like Chrome. To get back to your particular example, since getElementById in your microbench runs at approximately 100+ times the speed of querySelector (~15 mio op/s, which seems like what you'd expect) I would suspect that querySelector() is not optimized out, because it can actually throw an exception depending on the input.
- austincheney 7y ago
  Each approach is equally capable of resulting in an error state if unexpected input is supplied, for example pass an array into the document.getElementById. The largest important difference is that the input to the querySelector must be parsed as a coded instruction, a selector, while the input to getElementById is a string literal that does not get parsed.