6 ms·

There was an interesting comment a few months ago [1] > Not mentioned is that the full test sqlite test suite is proprietary and you need a super expensive sql

There was an interesting comment a few months ago [1] > Not mentioned is that the full test sqlite test suite is proprietary and you need a super expensive sqlite foundation membership to get access to it. According to Dr Hipp [2], no one bought the test suite. So there are definitely deficiencies in the test suite which may have been better addressed if the full test suite was open. [1] https://news.ycombinator.com/item?id=33346661 https://news.ycombinator.com/item?id=33346661 [2] https://corecursive.com/066-sqlite-with-richard-hipp/#billions-of-tests https://corecursive.com/066-sqlite-with-richard-hipp/#billio...

notRobot 4y ago
From the second link: > We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the source tree. Anybody can download the source code and run my test and run all those. They don’t provide 100% test coverage but they do test all the features very thoroughly. The 100% MCD tests, that’s called TH3. That’s proprietary. I had the idea that we would sell those tests to avionics manufacturers and make money that way. We’ve sold exactly zero copies of that so that didn’t really work out. It did work out really well for us in that it keeps our product really solid and it enables us to turn around new features and new bug fixes very fast.
- ec109685 4y ago
  It also makes forking SQLite infeasible since any new changes will be woefully under tested.
  - deleted 4y ago
    [deleted]
  - morelisp 4y ago
    SQLite's free test suite is far in excess of most other projects'. What do you think the test coverage is in the Linux kernel, GCC, or Emacs?
    - The_Colonel 4y ago
      (Embedded) database has stricter requirements than a text editor or a compiler.
      - morelisp 4y ago
        Obviously not, if nobody was willing to buy the test suite for their internal forks.
        The_Colonel 4y ago
        That's a non-sequitur argument.
        morelisp 4y ago
        It follows fine, but you want to debate instead of think. If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Nobody bought it, so it's not that important to maintaining a fork compared to the regular test suite even for such environments. A test suite which, to go back to my first comment, is still leagues ahead of the dozen other pieces of lynchpin software most companies have no problem depending on. Meanwhile, for the 99.9% of us out here not building aircraft and merely shipping a billion browsers or phones...
        The_Colonel 4y ago
        > If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Again, that's a non-sequitur (or perhaps strawman, you can choose), because I wasn't addressing the proprietary test set, merely the comparison between a text editor and a database, which is completely absurd since the tolerance for failures is drastically different.
        morelisp 4y ago
        > I wasn't addressing the proprietary test set Then perhaps you're in the wrong thread to be saying anything at all.
- bigDinosaur 4y ago
  What was the incentive to buy these tests if they've already been used to improve the product?
  - bluGill 4y ago
    You can then audit the tests to ensure they test some condition that you are concerned about. In reality nobody audits source code like that (see heartbleed for an unrelated example of critical code that didn't get proper audits from people who should have cared)
cryptonector 4y ago
But the testsuite being proprietary is part of their business model.

notRobot 4y ago

From the second link: > We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the source tree. Anybody can download the source code and run my test and run all those. They don’t provide 100% test coverage but they do test all the features very thoroughly. The 100% MCD tests, that’s called TH3. That’s proprietary. I had the idea that we would sell those tests to avionics manufacturers and make money that way. We’ve sold exactly zero copies of that so that didn’t really work out. It did work out really well for us in that it keeps our product really solid and it enables us to turn around new features and new bug fixes very fast.

ec109685 4y ago
It also makes forking SQLite infeasible since any new changes will be woefully under tested.
- deleted 4y ago
  [deleted]
- morelisp 4y ago
  SQLite's free test suite is far in excess of most other projects'. What do you think the test coverage is in the Linux kernel, GCC, or Emacs?
  - The_Colonel 4y ago
    (Embedded) database has stricter requirements than a text editor or a compiler.
    - morelisp 4y ago
      Obviously not, if nobody was willing to buy the test suite for their internal forks.
      - The_Colonel 4y ago
        That's a non-sequitur argument.
        morelisp 4y ago
        It follows fine, but you want to debate instead of think. If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Nobody bought it, so it's not that important to maintaining a fork compared to the regular test suite even for such environments. A test suite which, to go back to my first comment, is still leagues ahead of the dozen other pieces of lynchpin software most companies have no problem depending on. Meanwhile, for the 99.9% of us out here not building aircraft and merely shipping a billion browsers or phones...
        The_Colonel 4y ago
        > If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it. Again, that's a non-sequitur (or perhaps strawman, you can choose), because I wasn't addressing the proprietary test set, merely the comparison between a text editor and a database, which is completely absurd since the tolerance for failures is drastically different.
        morelisp 4y ago
        > I wasn't addressing the proprietary test set Then perhaps you're in the wrong thread to be saying anything at all.
bigDinosaur 4y ago
What was the incentive to buy these tests if they've already been used to improve the product?
- bluGill 4y ago
  You can then audit the tests to ensure they test some condition that you are concerned about. In reality nobody audits source code like that (see heartbleed for an unrelated example of critical code that didn't get proper audits from people who should have cared)

cryptonector 4y ago

But the testsuite being proprietary is part of their business model.