3 ms·

Apparently some people don't understand what I'm driving at, so I'll explain: V8 could choose to look at both microbenchmark performance and real web performan

Apparently some people don't understand what I'm driving at, so I'll explain: V8 could choose to look at both microbenchmark performance and real web performance, as both are useful signals for how V8 performs in general. The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. But, frankly, I find that incredibly unlikely, and I'd love to see actual examples of such a thing. Otherwise, this just seems like, and now I'm going to return to that idiom: throwing the proverbial baby out with the bathwater, here, simply because someone somewhere decided that microbenchmarks are universally "bad" while website performance tests are universally "good". Of course, this presupposes that this is, in fact, what the V8 team is doing.

dwild 7y ago
> The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. That's not the only reason. It's about putting the effort where it's matters. Sure gaining 30% speed in synthetic benchmark seems great, but if in real world scenario theses things doesn't happen that often (and spoiler alert, they don't), it will means actually much less than 30% of the speed gains. That's not a good metric to go toward. It's simply about investing the development time that you got, where it matters for the user. If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
- lostmsu 7y ago
  Just pondering here: imagine that sites they choose are poorly written, and they spend considerable effort optimizing for these bad patterns. Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser?
  - dwild 7y ago
    > Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser? I doubt it would, great pattern aren't just good for humans, they are also much more predictable, and that's quite good for JIT compiler. If it does, I'll just refer to my last sentence which still apply: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
- bradknowles 7y ago
  So, use the synthetic benchmark as a regression test. Progress is only counted towards the real world sites, but if you make performance markedly worse on the regression test, then that's a bug.
  - dwild 7y ago
    > if you make performance markedly worse on the regression test, then that's a bug. Not at all, it's about not investing time on what doesn't matter. For sure it will means you spend less time on something else and that will slowly get worse. That's why feature get deprecated and removed. Supporting something not used has a cost. I'll repeat my last paragraph: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite. If you do need that speed, show them.
bleah1000 7y ago
The thing about performance is that it's almost always a trade-off. For very mature projects, you hardly ever get a win on everything, you usually get an increase here and a decrease over there. It's very likely that optimizing for the microbenchmarks is actually hurting the real world performance, which is why it's gotten worse over time. So, optimizing for the microbenchmarks is more likely than not to actively harm real world performance. For example, if you have a caching scheme that was designed to make the microbenchmark better, it could very easily make real world performance worse. Like if the microbenchmark used 100 sized X blocks, so you pre allocated those on start-up, great microbenchmark speed up. But in the real world that hurts start-up time, and the cache is probably not fully utilized. A lot of times it's good to use microbenchmarks as a signpost, to indicate if some performance tuning causes a massive degradation that is unexpected.

dwild 7y ago

> The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. That's not the only reason. It's about putting the effort where it's matters. Sure gaining 30% speed in synthetic benchmark seems great, but if in real world scenario theses things doesn't happen that often (and spoiler alert, they don't), it will means actually much less than 30% of the speed gains. That's not a good metric to go toward. It's simply about investing the development time that you got, where it matters for the user. If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.

lostmsu 7y ago
Just pondering here: imagine that sites they choose are poorly written, and they spend considerable effort optimizing for these bad patterns. Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser?
- dwild 7y ago
  > Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser? I doubt it would, great pattern aren't just good for humans, they are also much more predictable, and that's quite good for JIT compiler. If it does, I'll just refer to my last sentence which still apply: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
bradknowles 7y ago
So, use the synthetic benchmark as a regression test. Progress is only counted towards the real world sites, but if you make performance markedly worse on the regression test, then that's a bug.
- dwild 7y ago
  > if you make performance markedly worse on the regression test, then that's a bug. Not at all, it's about not investing time on what doesn't matter. For sure it will means you spend less time on something else and that will slowly get worse. That's why feature get deprecated and removed. Supporting something not used has a cost. I'll repeat my last paragraph: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite. If you do need that speed, show them.

bleah1000 7y ago

The thing about performance is that it's almost always a trade-off. For very mature projects, you hardly ever get a win on everything, you usually get an increase here and a decrease over there. It's very likely that optimizing for the microbenchmarks is actually hurting the real world performance, which is why it's gotten worse over time. So, optimizing for the microbenchmarks is more likely than not to actively harm real world performance. For example, if you have a caching scheme that was designed to make the microbenchmark better, it could very easily make real world performance worse. Like if the microbenchmark used 100 sized X blocks, so you pre allocated those on start-up, great microbenchmark speed up. But in the real world that hurts start-up time, and the cache is probably not fully utilized. A lot of times it's good to use microbenchmarks as a signpost, to indicate if some performance tuning causes a massive degradation that is unexpected.