4 ms·

Oh man, if you think browsers are well tested automatically, you’re very far off reality. There’s basically no automated test suite even for CSS 2.1 that all v

Oh man, if you think browsers are well tested automatically, you’re very far off reality. There’s basically no automated test suite even for CSS 2.1 that all vendors can rely on that matches behavior specified by the standards. Luckily standards like that are so mature that it’s not needed as badly, but it definitely keeps incumbents safe from competition.

hu3 5y ago
Are you speaking from experience? Because a cursory glance at Chromium src tells me they do have automated tests for IndexedDB: https://source.chromium.org/chromium/chromium/src/+/main:content/browser/indexed_db/indexed_db_tombstone_sweeper_unittest.cc?q=indexdb&ss=chromium https://source.chromium.org/chromium/chromium/src/+/main:con...
- andrewmcwatters 5y ago
  Yes, I am. I'm an implementor. Some of the most fundamental parts of the web don't have comprehensive test suites. IndexedDB is relatively much more modern. However, if you do stumble on some tests for basic things like the box model, I'd love to know where they are! Tests like those would allow other vendors to know whether or not they're adhering to specification, programmatically. Earlier standards are very descriptive, rather than telling you how to implement the actual web technologies. That's left as an exercise for the reader. Seriously!
  - hu3 5y ago
    Interesting. This makes it even more bafling to me that Safari doesn't have basic unit tests for a modern feature such as IndexedDb. It's certainly not for a lack of funds. I guess applications living in the browser don't align well with Apple's interest in pushing their app store.
    - andrewmcwatters 5y ago
      A part of it is interest, too. Simply because there is a exceedingly small number of vendors, building comprehensive test suites that benefit newcomers is just not in most organizations' best interests. There actually are some partial tests from the W3C regarding basic web technologies, but they're not really automated ones in the way you'd think.
      - shawnz 5y ago
        I don't think the other poster was talking about automated tests for conformance to the specification, just basic unit tests indicating that the feature works at all by any reasonable definition.
        lostmsu 5y ago
        That kind of an issue is generally tested against by integration tests. Depending on the root cause unit tests for the entry point code might not have caught it.
        shawnz 5y ago
        You are right, an integration test would probably be better for demonstrating that this feature works. I only mean to say that you don't need a whole compliance testing suite to catch a major bug like this.
  - onion2k 5y ago
    However, if you do stumble on some tests for basic things like the box model, I'd love to know where they are! Its not really a test suite, but back in the early days of CSS we had https://en.m.wikipedia.org/wiki/Acid2 https://en.m.wikipedia.org/wiki/Acid2 It'd be easy to write a set of tests based on Acid2. There's also https://en.m.wikipedia.org/wiki/Acid3 https://en.m.wikipedia.org/wiki/Acid3 but that doesn't quite test things 'properly'.
    - andrewmcwatters 5y ago
      Yes, I fondly remember the Acid tests. Unfortunately, they aren’t very useful for implementors as they don’t cover the entire breadth of the specifications in isolation. In addition the specs have changed somewhat so now CSS 2.1 and the modules that existed at that point in time have matured beyond what the tests were written for, so a browser matching the spec today would fail. They were very fun for end users to see whether or not a browser was in compliance, though!
      - exikyut 5y ago
        I wonder if the path of least resistance would be to slowly convince the people that like writing Acid* tests to take things up a notch and rearchitect the next version(s) so that success is a measurement of collective+isolated implementational correctness. Such an effort would probably involve a higher level of ongoing support from the vendor side of the fence (in terms of motivation and iterative discussion etc), but relative to other options might be, as I said, the path of least resistance. </armchair_idea>
- ec109685 5y ago
  Seems like Safari could just adopt these tests…
- gok 5y ago
  So does WebKit/Safari. Hundreds of automated tests for IndexedDB, in fact. https://github.com/WebKit/WebKit/tree/main/LayoutTests/storage/indexeddb https://github.com/WebKit/WebKit/tree/main/LayoutTests/stora...
mh- 5y ago
I assume you're aware of the WPT project (contributed to by all of the major browser vendors these days), and its CSS 2.1 coverage is lacking in a way I'm unaware of. But for others who don't know: https://web-platform-tests.org/ https://web-platform-tests.org/ https://github.com/web-platform-tests/wpt/tree/master/css/CSS2 https://github.com/web-platform-tests/wpt/tree/master/css/CS...
- andrewmcwatters 5y ago
  Yes, this is the closest thing we have today, but it is built in such a way that implies you have a fully functioning browser already, which sort of defeats one aspect of the need for such a suite. They’re also tests that don’t actually point anything out. Take the CSS2 tests for example, https://wpt.live/css/CSS2/ https://wpt.live/css/CSS2/, I can’t do basically anything with these. They don’t tell me whether or not a particular part of the implementation has failed. They’re all basically arbitrary tests that sort of cover the appropriate properties, sometimes, maybe. The biggest glaring issue is that the test runners run in the browser. Well, if my user agent doesn’t have JavaScript… I can’t “run” the tests. Further, there are some parts of the standard that can’t be automated simply because it’s up to user agents to decide how the final rasterization turns out. You can’t just compare rasterization output or even reliably look for element heights for verification based on the differences between font rasterization. This is also why a part of the suite requires manual verification. What I really need as an implementor is the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat. With, of course, the caveat being that you can only really do this for what is well-defined in the standards. But you can’t do that easily with the wpts.
  - jefftk 5y ago
    That seems unrelated to whether WPT could cover this bug?
    - andrewmcwatters 5y ago
      This subthread isn’t discussing IndexedDB.

hu3 5y ago

Are you speaking from experience? Because a cursory glance at Chromium src tells me they do have automated tests for IndexedDB: https://source.chromium.org/chromium/chromium/src/+/main:content/browser/indexed_db/indexed_db_tombstone_sweeper_unittest.cc?q=indexdb&ss=chromium https://source.chromium.org/chromium/chromium/src/+/main:con...

andrewmcwatters 5y ago
Yes, I am. I'm an implementor. Some of the most fundamental parts of the web don't have comprehensive test suites. IndexedDB is relatively much more modern. However, if you do stumble on some tests for basic things like the box model, I'd love to know where they are! Tests like those would allow other vendors to know whether or not they're adhering to specification, programmatically. Earlier standards are very descriptive, rather than telling you how to implement the actual web technologies. That's left as an exercise for the reader. Seriously!
- hu3 5y ago
  Interesting. This makes it even more bafling to me that Safari doesn't have basic unit tests for a modern feature such as IndexedDb. It's certainly not for a lack of funds. I guess applications living in the browser don't align well with Apple's interest in pushing their app store.
  - andrewmcwatters 5y ago
    A part of it is interest, too. Simply because there is a exceedingly small number of vendors, building comprehensive test suites that benefit newcomers is just not in most organizations' best interests. There actually are some partial tests from the W3C regarding basic web technologies, but they're not really automated ones in the way you'd think.
    - shawnz 5y ago
      I don't think the other poster was talking about automated tests for conformance to the specification, just basic unit tests indicating that the feature works at all by any reasonable definition.
      - lostmsu 5y ago
        That kind of an issue is generally tested against by integration tests. Depending on the root cause unit tests for the entry point code might not have caught it.
        shawnz 5y ago
        You are right, an integration test would probably be better for demonstrating that this feature works. I only mean to say that you don't need a whole compliance testing suite to catch a major bug like this.
- onion2k 5y ago
  However, if you do stumble on some tests for basic things like the box model, I'd love to know where they are! Its not really a test suite, but back in the early days of CSS we had https://en.m.wikipedia.org/wiki/Acid2 https://en.m.wikipedia.org/wiki/Acid2 It'd be easy to write a set of tests based on Acid2. There's also https://en.m.wikipedia.org/wiki/Acid3 https://en.m.wikipedia.org/wiki/Acid3 but that doesn't quite test things 'properly'.
  - andrewmcwatters 5y ago
    Yes, I fondly remember the Acid tests. Unfortunately, they aren’t very useful for implementors as they don’t cover the entire breadth of the specifications in isolation. In addition the specs have changed somewhat so now CSS 2.1 and the modules that existed at that point in time have matured beyond what the tests were written for, so a browser matching the spec today would fail. They were very fun for end users to see whether or not a browser was in compliance, though!
    - exikyut 5y ago
      I wonder if the path of least resistance would be to slowly convince the people that like writing Acid* tests to take things up a notch and rearchitect the next version(s) so that success is a measurement of collective+isolated implementational correctness. Such an effort would probably involve a higher level of ongoing support from the vendor side of the fence (in terms of motivation and iterative discussion etc), but relative to other options might be, as I said, the path of least resistance. </armchair_idea>
ec109685 5y ago
Seems like Safari could just adopt these tests…
gok 5y ago
So does WebKit/Safari. Hundreds of automated tests for IndexedDB, in fact. https://github.com/WebKit/WebKit/tree/main/LayoutTests/storage/indexeddb https://github.com/WebKit/WebKit/tree/main/LayoutTests/stora...

mh- 5y ago

I assume you're aware of the WPT project (contributed to by all of the major browser vendors these days), and its CSS 2.1 coverage is lacking in a way I'm unaware of. But for others who don't know: https://web-platform-tests.org/ https://web-platform-tests.org/ https://github.com/web-platform-tests/wpt/tree/master/css/CSS2 https://github.com/web-platform-tests/wpt/tree/master/css/CS...

andrewmcwatters 5y ago
Yes, this is the closest thing we have today, but it is built in such a way that implies you have a fully functioning browser already, which sort of defeats one aspect of the need for such a suite. They’re also tests that don’t actually point anything out. Take the CSS2 tests for example, https://wpt.live/css/CSS2/ https://wpt.live/css/CSS2/, I can’t do basically anything with these. They don’t tell me whether or not a particular part of the implementation has failed. They’re all basically arbitrary tests that sort of cover the appropriate properties, sometimes, maybe. The biggest glaring issue is that the test runners run in the browser. Well, if my user agent doesn’t have JavaScript… I can’t “run” the tests. Further, there are some parts of the standard that can’t be automated simply because it’s up to user agents to decide how the final rasterization turns out. You can’t just compare rasterization output or even reliably look for element heights for verification based on the differences between font rasterization. This is also why a part of the suite requires manual verification. What I really need as an implementor is the ability to point my software at some .html files, get some failures, add more implementation, get less failures, and repeat. With, of course, the caveat being that you can only really do this for what is well-defined in the standards. But you can’t do that easily with the wpts.
- jefftk 5y ago
  That seems unrelated to whether WPT could cover this bug?
  - andrewmcwatters 5y ago
    This subthread isn’t discussing IndexedDB.