3 ms·

From the second link: > We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the so

From the second link: > We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the source tree. Anybody can download the source code and run my test and run all those. They don’t provide 100% test coverage but they do test all the features very thoroughly. The 100% MCD tests, that’s called TH3. That’s proprietary. I had the idea that we would sell those tests to avionics manufacturers and make money that way. We’ve sold exactly zero copies of that so that didn’t really work out. It did work out really well for us in that it keeps our product really solid and it enables us to turn around new features and new bug fixes very fast.

ec109685 4y ago
It also makes forking SQLite infeasible since any new changes will be woefully under tested.
- deleted 4y ago
  [deleted]
- morelisp 4y ago
  SQLite's free test suite is far in excess of most other projects'. What do you think the test coverage is in the Linux kernel, GCC, or Emacs?
  - The_Colonel 4y ago
    (Embedded) database has stricter requirements than a text editor or a compiler.
    - morelisp 4y ago
      Obviously not, if nobody was willing to buy the test suite for their internal forks.
      - The_Colonel 4y ago
        That's a non-sequitur argument.
        morelisp 4y ago
        It follows fine, but you want to debate instead of think. If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Nobody bought it, so it's not that important to maintaining a fork compared to the regular test suite even for such environments. A test suite which, to go back to my first comment, is still leagues ahead of the dozen other pieces of lynchpin software most companies have no problem depending on. Meanwhile, for the 99.9% of us out here not building aircraft and merely shipping a billion browsers or phones...
        The_Colonel 4y ago
        > If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Again, that's a non-sequitur (or perhaps strawman, you can choose), because I wasn't addressing the proprietary test set, merely the comparison between a text editor and a database, which is completely absurd since the tolerance for failures is drastically different.
        morelisp 4y ago
        > I wasn't addressing the proprietary test set Then perhaps you're in the wrong thread to be saying anything at all.
  - ec109685 4y ago
    Right, but it makes the fork strictly worse from a reliability perspective than SQLite given it will be less tested. If there wasn't a competitive advantage given it has no sales, wouldn't they have open sourced it by now?
    - morelisp 4y ago
      > wouldn't they have open sourced it by now? If there's minimal value in it, why put in the work to open source an extremely complex test environment?
      - ec109685 4y ago
        Just for completeness sake, they do offer a SQLite Consortium Membership for $120k, which I guess includes all their test suites as a selling point: https://www.sqlite.org/prosupport.html https://www.sqlite.org/prosupport.html
bigDinosaur 4y ago
What was the incentive to buy these tests if they've already been used to improve the product?
- bluGill 4y ago
  You can then audit the tests to ensure they test some condition that you are concerned about. In reality nobody audits source code like that (see heartbleed for an unrelated example of critical code that didn't get proper audits from people who should have cared)

ec109685 4y ago

It also makes forking SQLite infeasible since any new changes will be woefully under tested.

deleted 4y ago
[deleted]
morelisp 4y ago
SQLite's free test suite is far in excess of most other projects'. What do you think the test coverage is in the Linux kernel, GCC, or Emacs?
- The_Colonel 4y ago
  (Embedded) database has stricter requirements than a text editor or a compiler.
  - morelisp 4y ago
    Obviously not, if nobody was willing to buy the test suite for their internal forks.
    - The_Colonel 4y ago
      That's a non-sequitur argument.
      - morelisp 4y ago
        It follows fine, but you want to debate instead of think. If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Nobody bought it, so it's not that important to maintaining a fork compared to the regular test suite even for such environments. A test suite which, to go back to my first comment, is still leagues ahead of the dozen other pieces of lynchpin software most companies have no problem depending on. Meanwhile, for the 99.9% of us out here not building aircraft and merely shipping a billion browsers or phones...
        The_Colonel 4y ago
        > If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Again, that's a non-sequitur (or perhaps strawman, you can choose), because I wasn't addressing the proprietary test set, merely the comparison between a text editor and a database, which is completely absurd since the tolerance for failures is drastically different.
        morelisp 4y ago
        > I wasn't addressing the proprietary test set Then perhaps you're in the wrong thread to be saying anything at all.
- ec109685 4y ago
  Right, but it makes the fork strictly worse from a reliability perspective than SQLite given it will be less tested. If there wasn't a competitive advantage given it has no sales, wouldn't they have open sourced it by now?
  - morelisp 4y ago
    > wouldn't they have open sourced it by now? If there's minimal value in it, why put in the work to open source an extremely complex test environment?
    - ec109685 4y ago
      Just for completeness sake, they do offer a SQLite Consortium Membership for $120k, which I guess includes all their test suites as a selling point: https://www.sqlite.org/prosupport.html https://www.sqlite.org/prosupport.html

bigDinosaur 4y ago

What was the incentive to buy these tests if they've already been used to improve the product?

bluGill 4y ago
You can then audit the tests to ensure they test some condition that you are concerned about. In reality nobody audits source code like that (see heartbleed for an unrelated example of critical code that didn't get proper audits from people who should have cared)