3 ms·

Yeah benchmarks can be much more difficult to get right than it might seem at first glance. Good thing they didn't try to write their own benchmarking code, oth

Yeah benchmarks can be much more difficult to get right than it might seem at first glance. Good thing they didn't try to write their own benchmarking code, otherwise they might have fallen into those traps you just mentioned. Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1]. [1]: https://mathiasbynens.be/notes/javascript-benchmarking https://mathiasbynens.be/notes/javascript-benchmarking

cogman10 6y ago
You are making the assumption that the library doesn't have the flaws I mentioned. It does (You can go read the source yourself https://raw.githubusercontent.com/bestiejs/benchmark.js/2.1.2/benchmark.js https://raw.githubusercontent.com/bestiejs/benchmark.js/2.1.... ) There's no portion of the code that does warmups. There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. There is a lot of code though... so that's... good? You make the assumption that just because a lib is popular or widely used it is "correct" or "the best". When it comes to microbenchmarks, that's usually flawed. Very VERY few people actually get them right, benchmark.js is no exception. That, of course, doesn't mean that benchmark.js can't be useful. For macrobenchmarks it will be roughly right. However, for something as small as leftpad, it's almost certainly not the right way to measure performance.
- Ajedi32 6y ago
  Okay, you win. I'm not going to read the entire source of that package just to make a point. Though I do find it strange that the author of that library would write an entire blog post on the topic and then not take his own advice in the implementation of the library he wrote.
  - cogman10 6y ago
    Tell me, where in that blog post does he mention doing warm up cycles or avoiding having the JIT optimize away the method? (Hint, he doesn't mention that... so, no, he didn't actually miss his own advice.) The article is completely consumed with getting the timing of benchmarking right. Which, to be fair, is a place where microbenchmarks often go wrong. It, however, isn't the ONLY place they go wrong.
- phpnode 6y ago
  > There's no portion of the code that does warmups. This isn't true - Benchmark.js will repeatedly rerun benchmarks until it gathers statistically meaningful results. > There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. True, and there's actually nothing benchmark.js can do to ensure that doesn't happen in the general case but when this does happen the results are usually pretty obvious - we'd see billions of ops/sec. Incidentally the left-pad benchmarks do not suffer from this issue.
cjaybo 6y ago
> Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1]. This is the "benchmark lib" that was mentioned in the very first sentece of the comment you replied to.

cogman10 6y ago

You are making the assumption that the library doesn't have the flaws I mentioned. It does (You can go read the source yourself https://raw.githubusercontent.com/bestiejs/benchmark.js/2.1.2/benchmark.js https://raw.githubusercontent.com/bestiejs/benchmark.js/2.1.... ) There's no portion of the code that does warmups. There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. There is a lot of code though... so that's... good? You make the assumption that just because a lib is popular or widely used it is "correct" or "the best". When it comes to microbenchmarks, that's usually flawed. Very VERY few people actually get them right, benchmark.js is no exception. That, of course, doesn't mean that benchmark.js can't be useful. For macrobenchmarks it will be roughly right. However, for something as small as leftpad, it's almost certainly not the right way to measure performance.

Ajedi32 6y ago
Okay, you win. I'm not going to read the entire source of that package just to make a point. Though I do find it strange that the author of that library would write an entire blog post on the topic and then not take his own advice in the implementation of the library he wrote.
- cogman10 6y ago
  Tell me, where in that blog post does he mention doing warm up cycles or avoiding having the JIT optimize away the method? (Hint, he doesn't mention that... so, no, he didn't actually miss his own advice.) The article is completely consumed with getting the timing of benchmarking right. Which, to be fair, is a place where microbenchmarks often go wrong. It, however, isn't the ONLY place they go wrong.
phpnode 6y ago
> There's no portion of the code that does warmups. This isn't true - Benchmark.js will repeatedly rerun benchmarks until it gathers statistically meaningful results. > There's no portion of code that "blackholes" the results to keep the JIT from optimizing away the code under benchmark. True, and there's actually nothing benchmark.js can do to ensure that doesn't happen in the general case but when this does happen the results are usually pretty obvious - we'd see billions of ops/sec. Incidentally the left-pad benchmarks do not suffer from this issue.

cjaybo 6y ago

> Luckily, they didn't, and instead pulled in the the `benchmark` library as a development dependency. The author of said library works on V8, and already considered all those problems and much, much, more[1]. This is the "benchmark lib" that was mentioned in the very first sentece of the comment you replied to.