4 ms·

Chrome used to rely on fixed synthetic benchmarks like this for guiding their performance optimizations, but quickly realized they aren't a great indicator of r

Chrome used to rely on fixed synthetic benchmarks like this for guiding their performance optimizations, but quickly realized they aren't a great indicator of real world performance and ditched them in favor of instead evaluating Chrome's performance against a collection of popular, real-world websites. See https://v8.dev/blog/real-world-performance https://v8.dev/blog/real-world-performance for a discussion of their bench marking methodology.

deleted 7y ago
[deleted]
CarelessExpert 7y ago
Why not both? Something about babies and bathwater...
- CarelessExpert 7y ago
  Apparently some people don't understand what I'm driving at, so I'll explain: V8 could choose to look at both microbenchmark performance and real web performance, as both are useful signals for how V8 performs in general. The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. But, frankly, I find that incredibly unlikely, and I'd love to see actual examples of such a thing. Otherwise, this just seems like, and now I'm going to return to that idiom: throwing the proverbial baby out with the bathwater, here, simply because someone somewhere decided that microbenchmarks are universally "bad" while website performance tests are universally "good". Of course, this presupposes that this is, in fact, what the V8 team is doing.
  - dwild 7y ago
    > The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. That's not the only reason. It's about putting the effort where it's matters. Sure gaining 30% speed in synthetic benchmark seems great, but if in real world scenario theses things doesn't happen that often (and spoiler alert, they don't), it will means actually much less than 30% of the speed gains. That's not a good metric to go toward. It's simply about investing the development time that you got, where it matters for the user. If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
    - lostmsu 7y ago
      Just pondering here: imagine that sites they choose are poorly written, and they spend considerable effort optimizing for these bad patterns. Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser?
      - dwild 7y ago
        > Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser? I doubt it would, great pattern aren't just good for humans, they are also much more predictable, and that's quite good for JIT compiler. If it does, I'll just refer to my last sentence which still apply: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
    - bradknowles 7y ago
      So, use the synthetic benchmark as a regression test. Progress is only counted towards the real world sites, but if you make performance markedly worse on the regression test, then that's a bug.
      - dwild 7y ago
        > if you make performance markedly worse on the regression test, then that's a bug. Not at all, it's about not investing time on what doesn't matter. For sure it will means you spend less time on something else and that will slowly get worse. That's why feature get deprecated and removed. Supporting something not used has a cost. I'll repeat my last paragraph: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite. If you do need that speed, show them.
  - bleah1000 7y ago
    The thing about performance is that it's almost always a trade-off. For very mature projects, you hardly ever get a win on everything, you usually get an increase here and a decrease over there. It's very likely that optimizing for the microbenchmarks is actually hurting the real world performance, which is why it's gotten worse over time. So, optimizing for the microbenchmarks is more likely than not to actively harm real world performance. For example, if you have a caching scheme that was designed to make the microbenchmark better, it could very easily make real world performance worse. Like if the microbenchmark used 100 sized X blocks, so you pre allocated those on start-up, great microbenchmark speed up. But in the real world that hurts start-up time, and the cache is probably not fully utilized. A lot of times it's good to use microbenchmarks as a signpost, to indicate if some performance tuning causes a massive degradation that is unexpected.
digitarald 7y ago
Cautious that this is just the v8 team though.
oefrha 7y ago
I wish there will be some Chromium/Blink/v8 experts shedding some light on what might be going on in this thread. Otherwise it’s just gonna dissolve into another switch-to-Firefox call-to-action megathread that happens twice a day now.
the_duke 7y ago
Related: current Firefox is worse in almost every benchmark, but feels snappier or at least on par with Chrome on most sites.
- sterlind 7y ago
  I've noticed this too. I think it might be a result of the UX. Unscientific of me, but Firefox feels like the buttons directly trigger behavior, the page renders as it comes in, and the content is "closer" to the engine somehow. Chrome feels like the buttons signal intents and behavior is orchestrated behind the scenes to feel smooth. They're both good approaches, but Firefox feels more like I'm driving stick, if that makes sense.
Jasper_ 7y ago
I've seen a real-world regression in the performance of a site I develop [0]. It's math-heavy, so maybe that's not "real-world" enough for them. Filed a bug [1] but it's finnicky to repro, and the V8 team seems unable to replicate it in-house. [0] https://noclip.website/ https://noclip.website/ [1] https://bugs.chromium.org/p/chromium/issues/detail?id=1044492 https://bugs.chromium.org/p/chromium/issues/detail?id=104449...
smittywerben 7y ago
If you want to view "real world" performance try this awesome website[0] on some of your regular sites. It includes comparisons with chrome, chrome+ublock, and brave. Also previous hn discussion[1]. I think the outlier cases are interesting. The Gamespot test[2] loads 2x slower and with 2x page size using default uBlock. Strange glass-card websites these days. [0] https://webtest.app/ https://webtest.app/ [1] https://news.ycombinator.com/item?id=21052969 https://news.ycombinator.com/item?id=21052969 [2] https://webtest.app/?url=https://www.gamespot.com/ https://webtest.app/?url=https://www.gamespot.com/

deleted 7y ago

[deleted]

CarelessExpert 7y ago

Why not both? Something about babies and bathwater...

CarelessExpert 7y ago
Apparently some people don't understand what I'm driving at, so I'll explain: V8 could choose to look at both microbenchmark performance and real web performance, as both are useful signals for how V8 performs in general. The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. But, frankly, I find that incredibly unlikely, and I'd love to see actual examples of such a thing. Otherwise, this just seems like, and now I'm going to return to that idiom: throwing the proverbial baby out with the bathwater, here, simply because someone somewhere decided that microbenchmarks are universally "bad" while website performance tests are universally "good". Of course, this presupposes that this is, in fact, what the V8 team is doing.
- dwild 7y ago
  > The only reason to not use microbenchmarks is because you think they will somehow lead you astray and cause you to optimize things such that user experience actually gets worse. That's not the only reason. It's about putting the effort where it's matters. Sure gaining 30% speed in synthetic benchmark seems great, but if in real world scenario theses things doesn't happen that often (and spoiler alert, they don't), it will means actually much less than 30% of the speed gains. That's not a good metric to go toward. It's simply about investing the development time that you got, where it matters for the user. If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
  - lostmsu 7y ago
    Just pondering here: imagine that sites they choose are poorly written, and they spend considerable effort optimizing for these bad patterns. Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser?
    - dwild 7y ago
      > Could it make harder to make a truly performant website for Chrome, because switching to better patters makes your code loose due to them being less optimized in the browser? I doubt it would, great pattern aren't just good for humans, they are also much more predictable, and that's quite good for JIT compiler. If it does, I'll just refer to my last sentence which still apply: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite.
  - bradknowles 7y ago
    So, use the synthetic benchmark as a regression test. Progress is only counted towards the real world sites, but if you make performance markedly worse on the regression test, then that's a bug.
    - dwild 7y ago
      > if you make performance markedly worse on the regression test, then that's a bug. Not at all, it's about not investing time on what doesn't matter. For sure it will means you spend less time on something else and that will slowly get worse. That's why feature get deprecated and removed. Supporting something not used has a cost. I'll repeat my last paragraph: > If your usage is closer to synthetic benchmark and thus you lost close to that 30% of performance since Chrome 75, then take your real world scenario and send that to the V8 teams, I'm sure they'll be happy to consider it in the test suite. If you do need that speed, show them.
- bleah1000 7y ago
  The thing about performance is that it's almost always a trade-off. For very mature projects, you hardly ever get a win on everything, you usually get an increase here and a decrease over there. It's very likely that optimizing for the microbenchmarks is actually hurting the real world performance, which is why it's gotten worse over time. So, optimizing for the microbenchmarks is more likely than not to actively harm real world performance. For example, if you have a caching scheme that was designed to make the microbenchmark better, it could very easily make real world performance worse. Like if the microbenchmark used 100 sized X blocks, so you pre allocated those on start-up, great microbenchmark speed up. But in the real world that hurts start-up time, and the cache is probably not fully utilized. A lot of times it's good to use microbenchmarks as a signpost, to indicate if some performance tuning causes a massive degradation that is unexpected.

digitarald 7y ago

Cautious that this is just the v8 team though.

oefrha 7y ago

I wish there will be some Chromium/Blink/v8 experts shedding some light on what might be going on in this thread. Otherwise it’s just gonna dissolve into another switch-to-Firefox call-to-action megathread that happens twice a day now.

the_duke 7y ago

Related: current Firefox is worse in almost every benchmark, but feels snappier or at least on par with Chrome on most sites.

sterlind 7y ago
I've noticed this too. I think it might be a result of the UX. Unscientific of me, but Firefox feels like the buttons directly trigger behavior, the page renders as it comes in, and the content is "closer" to the engine somehow. Chrome feels like the buttons signal intents and behavior is orchestrated behind the scenes to feel smooth. They're both good approaches, but Firefox feels more like I'm driving stick, if that makes sense.

Jasper_ 7y ago

I've seen a real-world regression in the performance of a site I develop [0]. It's math-heavy, so maybe that's not "real-world" enough for them. Filed a bug [1] but it's finnicky to repro, and the V8 team seems unable to replicate it in-house. [0] https://noclip.website/ https://noclip.website/ [1] https://bugs.chromium.org/p/chromium/issues/detail?id=1044492 https://bugs.chromium.org/p/chromium/issues/detail?id=104449...

smittywerben 7y ago

If you want to view "real world" performance try this awesome website[0] on some of your regular sites. It includes comparisons with chrome, chrome+ublock, and brave. Also previous hn discussion[1]. I think the outlier cases are interesting. The Gamespot test[2] loads 2x slower and with 2x page size using default uBlock. Strange glass-card websites these days. [0] https://webtest.app/ https://webtest.app/ [1] https://news.ycombinator.com/item?id=21052969 https://news.ycombinator.com/item?id=21052969 [2] https://webtest.app/?url=https://www.gamespot.com/ https://webtest.app/?url=https://www.gamespot.com/