3 ms·

Thank you for the belly-laugh. It's Goodhart's Law in graph form. "Oh, is this metric important? Let me get right on that." No shade intended towards the Lady

by jerf 1y ago

Thank you for the belly-laugh. It's Goodhart's Law in graph form. "Oh, is this metric important? Let me get right on that." No shade intended towards the Ladybird team. You were given the terms and you're behaving rationally in response to them. More power to you. It's just a fantastic demonstration of what it looks like to very suddenly be developing against a very specific metric.

xmprt 1y ago
Have you tried using Ladybird recently? Admittedly I haven't but I've seen the rapid progress they've made over the last year. They might just be targeting this arbitrary metric but I'm inclined to believe that they've made real progress towards building a usable browser.
pizlonator 1y ago
I think that this metric correlates well with the browser being usable on real websites Also, I don't think that the Ladybird folks are just doing the bare minimum to only increase their score on WPT. They're implementing each feature in such a way that basic browsing seems to work better and that their WPT score improves.
- fabrice_d 1y ago
  The wpt score is not that well balanced. Look at https://staging.wpt.fyi/results/?product=servo&product=ladybird https://staging.wpt.fyi/results/?product=servo&product=ladyb... : out of about 2 million tests, more than half are for the "encoding" category. Good encoding support is needed for sure, but likely not at that level of prevalence.
  - jerf 1y ago
    It seems my communication did not adequately convey the fact that I have no problem with the Ladybird team doing this. It makes perfect sense and is the right thing to do. However, a jump like that means precisely and exactly what I said it means; very suddenly, that metric became much more important to the team. It is written straight into the graph. A large number of encoding-related tests that were probably relatively easy to fix in bulk is certainly a plausible explanation. A lot of people are imputing to me assumptions that they are bringing to my post, such as assuming that such improvements must be fake or bad or somehow otherwise cheating. Nope. Moreover, if you are thinking that, don't take it up with me, go take it up with the graph directly. It's not my graph. I'm just amused at the spectacular demonstration of Goodhart's Law. Are the commentators who think I'm being critical of the Ladybird project going to defend their implicit proposition that the browser got twice as good in whatever period that graph is in, a week or a month or whatever? Of course that's not the case.
    - fabrice_d 1y ago
      fwiw, I'm not imputing you any assumptions. I'm just pointing out that using wpt score as a criteria is not necessarily a good proxy for browser readiness. So I'm not sure why Apple uses that, other than... there's no other objective measure? Of course it's fair game for browser engines to improve their score!
    - tolerance 1y ago
      Dude at this point just raise your knickers up and criticize the thing. You’ve got the most valuable observation about this topic on your side. The graph is jarring and for someone only recently made familiar with Goodhart’s Law this is a great example of it in practice. You must be further well-informed enough to defend any issues you actually have with the project outright instead of this small war of attrition playing out waaay down here. Too much useful insight is withheld or misappropriated these days.
    - trflynn89 1y ago
      > However, a jump like that means precisely and exactly what I said it means; very suddenly, that metric became much more important to the team. It is written straight into the graph. Not really, though. The latest jump was from implementing some CSS Typed OM features, which has been in-progress work for a while now. The 6k increase in the test score was a bit of a happy surprise. It's also not that much of a jump when you zoom out and see it's "just" a continuation of a steady increase in score over a long period.
fragmede 1y ago
I mean, sure, but can you point to any work that you think they should be doing that they're not doing because they're chasing this benchmark instead of doing whatever it is you think they should be doing?
dzaima 1y ago
While it is kinda unfortunate to have one unbalanced test suite as the major external progress indicator, there are.. like no other good options that don't involve someone manually going through like the top 1000 sites or something and checking whether they look good. That leaves having no progress indication whatsoever, which is also pretty bad. And, in any case, implementing more of the standards is just simply good, and would need to be done at some point anyway.

by jerf 1y ago

xmprt 1y ago

Have you tried using Ladybird recently? Admittedly I haven't but I've seen the rapid progress they've made over the last year. They might just be targeting this arbitrary metric but I'm inclined to believe that they've made real progress towards building a usable browser.

pizlonator 1y ago

I think that this metric correlates well with the browser being usable on real websites Also, I don't think that the Ladybird folks are just doing the bare minimum to only increase their score on WPT. They're implementing each feature in such a way that basic browsing seems to work better and that their WPT score improves.

fabrice_d 1y ago
The wpt score is not that well balanced. Look at https://staging.wpt.fyi/results/?product=servo&product=ladybird https://staging.wpt.fyi/results/?product=servo&product=ladyb... : out of about 2 million tests, more than half are for the "encoding" category. Good encoding support is needed for sure, but likely not at that level of prevalence.
- jerf 1y ago
  It seems my communication did not adequately convey the fact that I have no problem with the Ladybird team doing this. It makes perfect sense and is the right thing to do. However, a jump like that means precisely and exactly what I said it means; very suddenly, that metric became much more important to the team. It is written straight into the graph. A large number of encoding-related tests that were probably relatively easy to fix in bulk is certainly a plausible explanation. A lot of people are imputing to me assumptions that they are bringing to my post, such as assuming that such improvements must be fake or bad or somehow otherwise cheating. Nope. Moreover, if you are thinking that, don't take it up with me, go take it up with the graph directly. It's not my graph. I'm just amused at the spectacular demonstration of Goodhart's Law. Are the commentators who think I'm being critical of the Ladybird project going to defend their implicit proposition that the browser got twice as good in whatever period that graph is in, a week or a month or whatever? Of course that's not the case.
  - fabrice_d 1y ago
    fwiw, I'm not imputing you any assumptions. I'm just pointing out that using wpt score as a criteria is not necessarily a good proxy for browser readiness. So I'm not sure why Apple uses that, other than... there's no other objective measure? Of course it's fair game for browser engines to improve their score!
  - tolerance 1y ago
    Dude at this point just raise your knickers up and criticize the thing. You’ve got the most valuable observation about this topic on your side. The graph is jarring and for someone only recently made familiar with Goodhart’s Law this is a great example of it in practice. You must be further well-informed enough to defend any issues you actually have with the project outright instead of this small war of attrition playing out waaay down here. Too much useful insight is withheld or misappropriated these days.
  - trflynn89 1y ago
    > However, a jump like that means precisely and exactly what I said it means; very suddenly, that metric became much more important to the team. It is written straight into the graph. Not really, though. The latest jump was from implementing some CSS Typed OM features, which has been in-progress work for a while now. The 6k increase in the test score was a bit of a happy surprise. It's also not that much of a jump when you zoom out and see it's "just" a continuation of a steady increase in score over a long period.

fragmede 1y ago

I mean, sure, but can you point to any work that you think they should be doing that they're not doing because they're chasing this benchmark instead of doing whatever it is you think they should be doing?

dzaima 1y ago

While it is kinda unfortunate to have one unbalanced test suite as the major external progress indicator, there are.. like no other good options that don't involve someone manually going through like the top 1000 sites or something and checking whether they look good. That leaves having no progress indication whatsoever, which is also pretty bad. And, in any case, implementing more of the standards is just simply good, and would need to be done at some point anyway.