5 ms·

Yay, "our users are our beta-testers" hard at work. How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? I agree wit

by lynx97 2y ago

Yay, "our users are our beta-testers" hard at work. How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? I agree with parent, this is an awful approach.

anon7000 2y ago
The scale is far too big. You cannot take into account the millions of permutations of websites and browser extensions when testing changes like this.
frosted-flakes 2y ago
They're not testing the browser, they're testing the Web. Testing in the real world with real users is the only way to adequately probe the effects of potentially breaking changes like this. This is not new either. Many such changes have been reverted after discovering that they broke more things than expected. One example is the Object.groupBy static method, which was initially Array.prototype.group.
- lynx97 2y ago
  > They're not testing the browser, they're testing the Web. That sounds like a pretty lame excuse. So if a self-driving car manufacturer does testing, they are not testing the car, they are testing the environment? Sounds like a pretty neat trick, maybe marketing should adopt this attitude.
arp242 2y ago
You can't test against the entire web, especially internationally. Do you know what the most popular Vietnamese sites are? I don't either. Do that for >200 languages. Never mind of course there's tons of non-public stuff.
- bryanrasmussen 2y ago
  [flagged]
  - tomalbrc 2y ago
    1. Your “could” is nonsense. 2. Better not to get salty about some downvotes
    - bryanrasmussen 2y ago
      >1. Your “could” is nonsense. quote: Do you know what the most popular Vietnamese sites are? I don't either. Do that for >200 languages. was the example given, it's true I do not know what the most popular Vietnamese sites are, and the person who posted that and pretty much anyone on this site does not know what the most popular Vietnamese sites are or the same thing for 200 plus languages, but there are a few companies out there for which finding that out is pretty much child's play. One of those companies makes the Chrome browser. Also testing against the entire web, obviously asking people to report problems is not really testing against the entire web, it is testing against a subset of the web. This should not really need to be pointed out to the H part of HN, but here I am. A similar thing would be if you had a big index of the web and you tested against that. >2. Better not to get salty about some downvotes your and my definitions about saltiness seem to vary a great deal. on edit: Noting really that the thing to determine is how many sites actually use the UA styles for H1 instead of overriding, which would also be child's play for Google, and determining the most popular sites or pages that do so. I believe that the sites that do so are few and far between and not exceptionally popular, as well as probably very simply styled, if this belief is true it would also be relatively simple for someone to figure out what side effects were likely - if they had the data and processing power to do so. But Mozilla does not, therefore they must ask people to tell them if they have problems.
- codedokode 2y ago
  You can test on special testing versions. Browser instances behaving inconsistently with the same version only add more problems to developers.
eru 2y ago
> How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? Presumably they have done as much of that evaluation as is reasonable already.
- ozim 2y ago
  Second that - I assume that they did asnmuch as they could and this is just defense in depth one additional step.
  - codedokode 2y ago
    That is not defence in depth, that is causing annoyance to web developers.
- joquarky 2y ago
  In my experience, that's probably not a good presumption anymore. It feels to me like usability testing and quality assurance have both joined security and accessibility on the back burner now.
hulitu 2y ago
> How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? "Who would pay for it ?" Sincerely, Microsoft™
mvdtnz 2y ago
Yeah why didn't the developer simply test every website on the internet.
zcorpan 2y ago
Some testing has already happened before starting the rollout. The change has been shipping in Firefox Nightly for a year. I have analyzed impact of affected pages in the HTTP Archive dataset (about 12,000,000 pages), twice: https://github.com/whatwg/html/issues/7867#issuecomment-1977647444 https://github.com/whatwg/html/issues/7867#issuecomment-1977... https://github.com/whatwg/html/issues/7867#issuecomment-2595987424 https://github.com/whatwg/html/issues/7867#issuecomment-2595... The next step is either rollout or ship directly.
chuckadams 2y ago
> Yay, "our users are our beta-testers" hard at work. That is kind of part of the bargain when you download a version of the browser explicitly labeled as a beta.

by lynx97 2y ago

Yay, "our users are our beta-testers" hard at work. How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? I agree with parent, this is an awful approach.

anon7000 2y ago

The scale is far too big. You cannot take into account the millions of permutations of websites and browser extensions when testing changes like this.

frosted-flakes 2y ago

They're not testing the browser, they're testing the Web. Testing in the real world with real users is the only way to adequately probe the effects of potentially breaking changes like this. This is not new either. Many such changes have been reverted after discovering that they broke more things than expected. One example is the Object.groupBy static method, which was initially Array.prototype.group.

lynx97 2y ago
> They're not testing the browser, they're testing the Web. That sounds like a pretty lame excuse. So if a self-driving car manufacturer does testing, they are not testing the car, they are testing the environment? Sounds like a pretty neat trick, maybe marketing should adopt this attitude.

arp242 2y ago

You can't test against the entire web, especially internationally. Do you know what the most popular Vietnamese sites are? I don't either. Do that for >200 languages. Never mind of course there's tons of non-public stuff.

bryanrasmussen 2y ago
[flagged]
- tomalbrc 2y ago
  1. Your “could” is nonsense. 2. Better not to get salty about some downvotes
  - bryanrasmussen 2y ago
    >1. Your “could” is nonsense. quote: Do you know what the most popular Vietnamese sites are? I don't either. Do that for >200 languages. was the example given, it's true I do not know what the most popular Vietnamese sites are, and the person who posted that and pretty much anyone on this site does not know what the most popular Vietnamese sites are or the same thing for 200 plus languages, but there are a few companies out there for which finding that out is pretty much child's play. One of those companies makes the Chrome browser. Also testing against the entire web, obviously asking people to report problems is not really testing against the entire web, it is testing against a subset of the web. This should not really need to be pointed out to the H part of HN, but here I am. A similar thing would be if you had a big index of the web and you tested against that. >2. Better not to get salty about some downvotes your and my definitions about saltiness seem to vary a great deal. on edit: Noting really that the thing to determine is how many sites actually use the UA styles for H1 instead of overriding, which would also be child's play for Google, and determining the most popular sites or pages that do so. I believe that the sites that do so are few and far between and not exceptionally popular, as well as probably very simply styled, if this belief is true it would also be relatively simple for someone to figure out what side effects were likely - if they had the data and processing power to do so. But Mozilla does not, therefore they must ask people to tell them if they have problems.
codedokode 2y ago
You can test on special testing versions. Browser instances behaving inconsistently with the same version only add more problems to developers.

eru 2y ago

> How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? Presumably they have done as much of that evaluation as is reasonable already.

ozim 2y ago
Second that - I assume that they did asnmuch as they could and this is just defense in depth one additional step.
- codedokode 2y ago
  That is not defence in depth, that is causing annoyance to web developers.
joquarky 2y ago
In my experience, that's probably not a good presumption anymore. It feels to me like usability testing and quality assurance have both joined security and accessibility on the back burner now.

hulitu 2y ago

> How about devs evaluate a change by doing some testing, instead of using users as guinea pigs? "Who would pay for it ?" Sincerely, Microsoft™

mvdtnz 2y ago

Yeah why didn't the developer simply test every website on the internet.

zcorpan 2y ago

Some testing has already happened before starting the rollout. The change has been shipping in Firefox Nightly for a year. I have analyzed impact of affected pages in the HTTP Archive dataset (about 12,000,000 pages), twice: https://github.com/whatwg/html/issues/7867#issuecomment-1977647444 https://github.com/whatwg/html/issues/7867#issuecomment-1977... https://github.com/whatwg/html/issues/7867#issuecomment-2595987424 https://github.com/whatwg/html/issues/7867#issuecomment-2595... The next step is either rollout or ship directly.

chuckadams 2y ago

> Yay, "our users are our beta-testers" hard at work. That is kind of part of the bargain when you download a version of the browser explicitly labeled as a beta.