5 ms·

Imho, the problem with Lighthouse (and Pagespeed before it) isn't that they're not perfect, it's that they assign scores/grades. When Google assigns a score to

Imho, the problem with Lighthouse (and Pagespeed before it) isn't that they're not perfect, it's that they assign scores/grades. When Google assigns a score to something, people understand it to mean highest score = best and start optimizing for the grade Google gives them, not for performance and user experience, which the grade is supposed to represent. It would be more fruitful to list the issues and their severity but not add overall scores, because scores change the objective from fixing the problems to getting high scores. They occasionally also have the have bugs where they will punish something with a worse score that is actually an improvement in the real world, discouraging people from doing the right thing ("I want my site to load faster, but Google is buggy and will rank me down if I do").

freedomben 6y ago
I mostly agree with you, although I think having a "score" can be really useful. It is only as useful as the underlying formula for calculating it, however. I view it like I do code coverage: it doesn't mean your code is well tested just because it has 100% coverage, but if the number is going down with each commit then that does likely indicate that testing is not a priority and isn't happening with regularity. The real problem comes in when people (especially managers) assume there is qualitative information inherent in the score, when there is not, or that it is a quantitative measure of test quality, when it is not.
ljm 6y ago
The problem, as I see it, is it more or less makes Google the arbiter of the internet, and when their purpose is to provide internet advertising then I see a conflict of interest. Especially when things like "supporting AMP" will likely impact your score. While unfortunately many businesses have to care about Google SEO due to their complete monopoly status in the search field (one that is of course growing increasingly unhealthy), I'd prefer to not let Google be an ML-driven judge, jury and executioner when it comes to being visible on the net.
- anoncareer0212 6y ago
  I'm not sure the argument in your post or the parents hold water - lighthouse scores aren't a "judge, jury, and executioner" sometimes dev tools are just dev tools, and nothing is perfect.
  - the_pwner224 6y ago
    Wouldn't be surprised if they use it in search rankings, chances are they already do. And not being on Google effectively == dead. They already take into account page load speed (which discriminates against complex but useful & good web apps) and iirc also whether you have AMP.
    - anoncareer0212 6y ago
      Maybe Google's CEO runs lighthouse on a server in his closet and personally has hacked it into something ads something something data something something search My point being, I'm not personally amenable to conspiratorial thinking, it correlates with argument dysfunction
    - aww_dang 6y ago
      Do a test and compare your own optimized site to an amp site. See if you can't beat it. Not that hard when you consider the 3rd party requests amp requires.
      - the_pwner224 6y ago
        I never said amp is good or fast, just that (iirc, but 90% sure) Google penalizes you for not having amp.
        aww_dang 6y ago
        My experience is that AMP is necessary for google news etc. Haven't seen the outcome you speak of as compared to regular pages which are optimized.
magicalist 6y ago
> They occasionally also have the have bugs where they will punish something with a worse score that is actually an improvement in the real world If there's a bug and it lists an issue of "you got the aria roles here wrong" when they aren't, it really isn't any different than a less than 100 score with that issue listed underneath in terms of feeling pressure to fix it. Completely automating accessibility testing isn't possible (the article calls this out), but it's still a net good for these tools to exist.
hbbio 6y ago
You've just described the school exam system
jameslk 6y ago
My guess is if there wasn't a Lighthouse score, the focus would be on another singular metric instead, like First Contentful Paint for web performance. The score, while imperfect, helps at least to have a focus on something that weighs multiple metrics. As an industry, I'm wary of having a single company define these scores. It seems like a conflict of interest. I'd rather have this in control by a standards group.
- _puk 6y ago
  My issue with the score is that it is numerical. If I get 91, I want 100. Having a traffic light style system whereby green is good, red is bad etc. would give a decent target without the costly Uber optimisation for perfection. To your point of a single company defining these scores - a standards body would be great, I agree, but there's a danger it'll be like all the other web standards bodies, whereby Google is a member, albeit one with pretty much all the power.
madvoid 6y ago
The word for this general phenomena for those that are curious is Goodhart's law: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
aww_dang 6y ago
Depends on what the right thing is in your specific context. If you drop from 100% to 98% but increase your conversion rate 5x, the trade off should be obvious. From my experience anything 95% or better is basically perfect. Although stuff like "time to first byte" might impact the number of pages crawled when you have millions of pages etc. There will always be edge cases. The article makes a funny point, but I'm not sure it is a practical one. Who would honestly go through those steps? Maybe someone copy-pasting or installing templates could make a few of those errors? Overall the score is helpful and the specific errors detected are always visible in the breakdown below the score. As with anything if you don't understand the underlying technology, tooling and metrics can be troublesome. In those cases, a simple score is still better than giving a novice developer a complex breakdown.
- sukilot 6y ago
  The article's point is that you can accidentally break accessibility. Lots of sites so garbage like not showing text if JS is off.
CivBase 6y ago
Yeah, I don't use static analysis tools to "grade" my code. I use them to automatically find simple issues I might have overlooked. If ESLint doesn't find any issues, it doesn't mean my Javascript is actually good. It just means it isn't bad in the ways ESLint is looking for.
twoclicksnorth 6y ago
When a measure becomes a target, it ceases to be a good measure.
- TulliusCicero 6y ago
  And yet, making it a target can still be a positive move. The upside of having it be a score, is that makes it easier to have a big cultural push to deal with the issues covered in the score. So the measure isn't as good, but you have more people taking it seriously.
mister_hn 6y ago
He solution is simple: start disengaging with Goole and it's scores and move to other search engines (DuckDuckGo, Ecosia?), So these "optimization" will stop all together and the monopoly as well
nathanyz 6y ago
Yeah, we did an analysis* on the recent upgrades to use LCP, and found that the score now heavily incentivizes Iframe/Embed content even though they are a big negative for actual performance. [*] https://buzz.swarmify.com/how-to-get-a-100-score-on-lighthouse-pagespeed-in-one-easy-step/ https://buzz.swarmify.com/how-to-get-a-100-score-on-lighthou...

freedomben 6y ago

I mostly agree with you, although I think having a "score" can be really useful. It is only as useful as the underlying formula for calculating it, however. I view it like I do code coverage: it doesn't mean your code is well tested just because it has 100% coverage, but if the number is going down with each commit then that does likely indicate that testing is not a priority and isn't happening with regularity. The real problem comes in when people (especially managers) assume there is qualitative information inherent in the score, when there is not, or that it is a quantitative measure of test quality, when it is not.

ljm 6y ago

The problem, as I see it, is it more or less makes Google the arbiter of the internet, and when their purpose is to provide internet advertising then I see a conflict of interest. Especially when things like "supporting AMP" will likely impact your score. While unfortunately many businesses have to care about Google SEO due to their complete monopoly status in the search field (one that is of course growing increasingly unhealthy), I'd prefer to not let Google be an ML-driven judge, jury and executioner when it comes to being visible on the net.

anoncareer0212 6y ago
I'm not sure the argument in your post or the parents hold water - lighthouse scores aren't a "judge, jury, and executioner" sometimes dev tools are just dev tools, and nothing is perfect.
- the_pwner224 6y ago
  Wouldn't be surprised if they use it in search rankings, chances are they already do. And not being on Google effectively == dead. They already take into account page load speed (which discriminates against complex but useful & good web apps) and iirc also whether you have AMP.
  - anoncareer0212 6y ago
    Maybe Google's CEO runs lighthouse on a server in his closet and personally has hacked it into something ads something something data something something search My point being, I'm not personally amenable to conspiratorial thinking, it correlates with argument dysfunction
  - aww_dang 6y ago
    Do a test and compare your own optimized site to an amp site. See if you can't beat it. Not that hard when you consider the 3rd party requests amp requires.
    - the_pwner224 6y ago
      I never said amp is good or fast, just that (iirc, but 90% sure) Google penalizes you for not having amp.
      - aww_dang 6y ago
        My experience is that AMP is necessary for google news etc. Haven't seen the outcome you speak of as compared to regular pages which are optimized.

magicalist 6y ago

> They occasionally also have the have bugs where they will punish something with a worse score that is actually an improvement in the real world If there's a bug and it lists an issue of "you got the aria roles here wrong" when they aren't, it really isn't any different than a less than 100 score with that issue listed underneath in terms of feeling pressure to fix it. Completely automating accessibility testing isn't possible (the article calls this out), but it's still a net good for these tools to exist.

hbbio 6y ago

You've just described the school exam system

jameslk 6y ago

My guess is if there wasn't a Lighthouse score, the focus would be on another singular metric instead, like First Contentful Paint for web performance. The score, while imperfect, helps at least to have a focus on something that weighs multiple metrics. As an industry, I'm wary of having a single company define these scores. It seems like a conflict of interest. I'd rather have this in control by a standards group.

_puk 6y ago
My issue with the score is that it is numerical. If I get 91, I want 100. Having a traffic light style system whereby green is good, red is bad etc. would give a decent target without the costly Uber optimisation for perfection. To your point of a single company defining these scores - a standards body would be great, I agree, but there's a danger it'll be like all the other web standards bodies, whereby Google is a member, albeit one with pretty much all the power.

madvoid 6y ago

The word for this general phenomena for those that are curious is Goodhart's law: https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law

aww_dang 6y ago

Depends on what the right thing is in your specific context. If you drop from 100% to 98% but increase your conversion rate 5x, the trade off should be obvious. From my experience anything 95% or better is basically perfect. Although stuff like "time to first byte" might impact the number of pages crawled when you have millions of pages etc. There will always be edge cases. The article makes a funny point, but I'm not sure it is a practical one. Who would honestly go through those steps? Maybe someone copy-pasting or installing templates could make a few of those errors? Overall the score is helpful and the specific errors detected are always visible in the breakdown below the score. As with anything if you don't understand the underlying technology, tooling and metrics can be troublesome. In those cases, a simple score is still better than giving a novice developer a complex breakdown.

sukilot 6y ago
The article's point is that you can accidentally break accessibility. Lots of sites so garbage like not showing text if JS is off.

CivBase 6y ago

Yeah, I don't use static analysis tools to "grade" my code. I use them to automatically find simple issues I might have overlooked. If ESLint doesn't find any issues, it doesn't mean my Javascript is actually good. It just means it isn't bad in the ways ESLint is looking for.

twoclicksnorth 6y ago

When a measure becomes a target, it ceases to be a good measure.

TulliusCicero 6y ago
And yet, making it a target can still be a positive move. The upside of having it be a score, is that makes it easier to have a big cultural push to deal with the issues covered in the score. So the measure isn't as good, but you have more people taking it seriously.

mister_hn 6y ago

He solution is simple: start disengaging with Goole and it's scores and move to other search engines (DuckDuckGo, Ecosia?), So these "optimization" will stop all together and the monopoly as well

nathanyz 6y ago

Yeah, we did an analysis* on the recent upgrades to use LCP, and found that the score now heavily incentivizes Iframe/Embed content even though they are a big negative for actual performance. [*] https://buzz.swarmify.com/how-to-get-a-100-score-on-lighthouse-pagespeed-in-one-easy-step/ https://buzz.swarmify.com/how-to-get-a-100-score-on-lighthou...