4 ms·

Hi, I work on web-platform-tests. I'm not really sure I follow your complaint, so I wonder if there's a point of misunderstanding. The tests written for DOM AP

Hi, I work on web-platform-tests. I'm not really sure I follow your complaint, so I wonder if there's a point of misunderstanding. The tests written for DOM APIs certainly require running javascript to execute the test — that's a given — but most of the rendering tests are reftests. These consist of two files, one using the feature under test and one avoiding that feature [1]. The test passes when the two renderings are identical. That does make these tests difficult to run "by hand", but in terms of implementation it's possible to automate in anything that's capable of rendering to an image. Reftests are preferred over comparing rendering to a fixed image because fixed images tend to be invalidated due to unrelated/permitted changes in the rendering (e.g. a different font or different antialiasing choices). These will typically apply to both test and reference and so are handled automatically. Reftests obviously aren't suitable to bootstrap getting very basic things working, so there are some manual tests; I suppose one could hope to avoid that by inventing a special kind of output inspection automation just for those basic tests, but it's hard to justify. IN general, however, the "the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat" is exactly what wpt offers. There is some work required to stand up the necessary infrastructure to run the tests automatically, but it's common for new implementations of the platform to make use of web-platform-tests (e.g. Servo and Flow have both made use of wpt). In terms of coverage it's very difficult to demonstrate that all the requirements of a spec, and all its interactions with other specs, are completely covered. There have been various proposals for adding test metadata to try to measure coverage, but these have largely proved impractical. Nevertheless some CSS specs have links from the spec to the relevant test cases. And if you find cases that aren't covered by exisitng tests it would be great to get a PR to add new tests [2] [1] https://web-platform-tests.org/writing-tests/reftests.html https://web-platform-tests.org/writing-tests/reftests.html [2] https://github.com/web-platform-tests/wpt https://github.com/web-platform-tests/wpt

andrewmcwatters 5y ago
I’m not sure what you define as “getting very basic things working” but there’s no acceptable threshold for me. Are line boxes very basic? Is overflow compositing very basic? It’s irrelevant whether or not one is bootstrapping. It is unacceptable to me to use the software I’m testing to test itself if I can’t establish a baseline for the rest of the tests. Otherwise, all of the tests could be invalid one day based on a regression and I wouldn’t know. But this is explicitly done with reftests. And I don’t know what you’re talking about with the DOM API tests, there are several layout and compositor tests that require JavaScript and it’s unclear why other than the test author decided to use JavaScript to verify DOM metrics. That’s fine, I just don’t want that in my layout tests. I’m OK with a subset of the tests just not being automated based on font rasterization and line height calculations. The standards make those expectations clear, but everything else feels like fair game to me. I can either test for it, or I can’t. If I can’t, it’s not automated. And if it’s automated, I need information about what is failing, and much of the reftests don’t explicitly point out what they’re testing. The assert data is useful when it’s available, but even then sometimes test authors neglect to point out explicitly what they’re testing for. Plenty of the layout tests in particular are just box model renders that should match with no explanation as to what section or text is being tested against. That’s just not acceptable. It’s not even about interactions between specs, the specs isolated are not well tested. If you’re testing, let’s say, the width calculation of a box based on children boxes’ widths in normal flow, it needs to say that. And I think I’ve only ever see some of WebKit’s test’s help links reference actual sections. Otherwise the WPT provide little to no value to me. My team has been better off writing our own tests because we directly refer to the recommended or living standards texts by section, quote it and link to it, and as a result there’s no ambiguity about what property, calculation, or raster output we’re testing for.

andrewmcwatters 5y ago

I’m not sure what you define as “getting very basic things working” but there’s no acceptable threshold for me. Are line boxes very basic? Is overflow compositing very basic? It’s irrelevant whether or not one is bootstrapping. It is unacceptable to me to use the software I’m testing to test itself if I can’t establish a baseline for the rest of the tests. Otherwise, all of the tests could be invalid one day based on a regression and I wouldn’t know. But this is explicitly done with reftests. And I don’t know what you’re talking about with the DOM API tests, there are several layout and compositor tests that require JavaScript and it’s unclear why other than the test author decided to use JavaScript to verify DOM metrics. That’s fine, I just don’t want that in my layout tests. I’m OK with a subset of the tests just not being automated based on font rasterization and line height calculations. The standards make those expectations clear, but everything else feels like fair game to me. I can either test for it, or I can’t. If I can’t, it’s not automated. And if it’s automated, I need information about what is failing, and much of the reftests don’t explicitly point out what they’re testing. The assert data is useful when it’s available, but even then sometimes test authors neglect to point out explicitly what they’re testing for. Plenty of the layout tests in particular are just box model renders that should match with no explanation as to what section or text is being tested against. That’s just not acceptable. It’s not even about interactions between specs, the specs isolated are not well tested. If you’re testing, let’s say, the width calculation of a box based on children boxes’ widths in normal flow, it needs to say that. And I think I’ve only ever see some of WebKit’s test’s help links reference actual sections. Otherwise the WPT provide little to no value to me. My team has been better off writing our own tests because we directly refer to the recommended or living standards texts by section, quote it and link to it, and as a result there’s no ambiguity about what property, calculation, or raster output we’re testing for.