3 ms·

What I meant is that, in the areas that are not well standardized, and where the browsers don't agree with each other, who is to say that Chrome is right and th

by k_sze 5y ago

What I meant is that, in the areas that are not well standardized, and where the browsers don't agree with each other, who is to say that Chrome is right and that Firefox is the one being incompatible/non-conformant? In this sense, the scores are not independent. Among the areas listed, I understand that at least 10 focus areas are well-standardized and the major browser vendors agree amongst themselves regarding the interpretation of the standard. So that means 60% of the score should be independent. There are 5 other areas that were pulled from Google's and Microsoft's Compat 2021; those might not be well standardized, in which case 30% of the score might be disputable. (UPDATE: I think the 5 additional areas are also well standardized, so I guess you could also say they are independent and not subject to dispute.) And then the remaining 10% investigative areas are quite probably disputable.

jgraham 5y ago
You are correct; the real success in terms of improving the web for users and authors will be if all the browser engines end up aligning on behaviour and getting a good score. Nevertheless having per-browser scores is useful for the browser teams themselves to help them prioritise their own contribution to that overall success. You're also correct that there are cases where the behaviour as expected by current specs / tests might turn out to not be web compatible and so alignment will require changing the expectations and might reduce scores for implementations that were previously following the spec. That is already a thing that happens (although it would be great if it didn't). In practice, most of the focus areas are quite new parts of the platform with well written standards and not too much existing content. So I don't expect this to be a big effect. The "web compat" area has some cases where this outcome is more likely, but it's comprised of user-experienced problems in real sites, so the incentive to fix the problem is much greater than the incentive to avoid a temporary drop in the interop score. The investigate areas are specifically those where it's unclear what the path to interoperability looks like, or how to measure interoperability. In those areas, figuring out what we can realistically do to ship the feature in a way that works the same across browsers is the point. Hopefully, if we run this kind of thing again in the future, we have a clear picture of what's web compatible, and can turn those investigate areas into focus areas that are scored on the basis of a testsuite that everyone agrees is useful.

by k_sze 5y ago

jgraham 5y ago

You are correct; the real success in terms of improving the web for users and authors will be if all the browser engines end up aligning on behaviour and getting a good score. Nevertheless having per-browser scores is useful for the browser teams themselves to help them prioritise their own contribution to that overall success. You're also correct that there are cases where the behaviour as expected by current specs / tests might turn out to not be web compatible and so alignment will require changing the expectations and might reduce scores for implementations that were previously following the spec. That is already a thing that happens (although it would be great if it didn't). In practice, most of the focus areas are quite new parts of the platform with well written standards and not too much existing content. So I don't expect this to be a big effect. The "web compat" area has some cases where this outcome is more likely, but it's comprised of user-experienced problems in real sites, so the incentive to fix the problem is much greater than the incentive to avoid a temporary drop in the interop score. The investigate areas are specifically those where it's unclear what the path to interoperability looks like, or how to measure interoperability. In those areas, figuring out what we can realistically do to ship the feature in a way that works the same across browsers is the point. Hopefully, if we run this kind of thing again in the future, we have a clear picture of what's web compatible, and can turn those investigate areas into focus areas that are scored on the basis of a testsuite that everyone agrees is useful.