7 ms·

Anything specific about their tests? I found this doc: http://www.sqlite.org/testing.html http://www.sqlite.org/testing.html

jncraton 10y ago
SQLite is frequently cited for having incredible robustness, and this is certainly related to incredible test coverage. Many of the stats on that page are impressive, but the one that always gets me is that for 122 thousand lines of production code, the project has 90 million lines of tests.
- chrisweekly 10y ago
  Tests are like guard rails. Nobody's saying they shouldn't be there, but they're safety nets. [Rich Hickey's _amazing_ talk on simple vs easy](https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi...)
- sAbakumoff 10y ago
  Though I am a huge fan of SQLite, I am not sure if the incredible test coverage necessarily leads to success : "Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software-development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't just test more; develop better."(McConnell, Steve (2009-11-30). Code Complete (Kindle Location 16276). Microsoft Press. Kindle Edition) Another unique aspect of SQLite code base is total sticking to KISS(http://www.jarchitect.com/Blog/?p=2392 http://www.jarchitect.com/Blog/?p=2392)
  - austinpray 10y ago
    That is not an apt analogy. Unit tests improve code quality: code that is easily testable has high cohesion and low coupling. Tests can also serve as valuable documentation.
    - brutuscat 10y ago
      Not quite a lot apparently.... https://www.scribd.com/document/225906827/Software-Defect-Origins-and-Removal-Methods2013 https://www.scribd.com/document/225906827/Software-Defect-Or... https://theholyjava.wordpress.com/2015/01/26/challenging-myself-with-copliens-why-most-unit-testing-is-waste/ https://theholyjava.wordpress.com/2015/01/26/challenging-mys...
    - sAbakumoff 10y ago
      That is analogy I copied from McConnell, Steve (2009-11-30). Code Complete (Kindle Location 16276). Microsoft Press. Kindle Edition. I tend to rely on what I written on this book
      - sigzero 10y ago
        That's fine. It's a good book. That doesn't mean the analogy works in every situation. I happen to think it doesn't in this one. Dr. Hipp is a great coder. Telling him to "develop better" from that analogy just falls apart. Tests are there for a reason and Dr. Hipp uses them to great effect on the quality of SQLite.
        sAbakumoff 10y ago
        Yeah I agree. Actually the whole point of the original comment was about SQLite's robustness as the result of multiple development practices they use(including testing of course), but testing coverage can't lead to success itself. That's it)
    - Someone 10y ago
      High cohesion and low coupling are good, and easily testable correlates with that, but having lots of unit tests doesn't imply that the code is easily testable, and not having unit tests doesn't imply that code has low cohesion or high coupling. It's even possible code needs many lines of testing partly because it lacks in ease of testing. In the case of SQLite I think it mostly is because of hard work to fulfill the ambition to deliver a robust project and because of the existence of fuzzing, which can automate test generation.
  - andrewflnr 10y ago
    If you want to lose weight, you need to be able to actually measure your weight. Codebases with inadequate tests can't do this accurately, so they don't even know what changes to their diet/development process help.
    - sAbakumoff 10y ago
      >If you want to lose weight, you need to be able to actually measure your weight. That's not true. There are multiple ways to track the progress of losing weight, starting from simple "look at the mirror" ending with fat percent measure.
    - sAbakumoff 10y ago
      Also, the quote does not say that you have to avoid testing, it says that trying to improve software quality by increasing the amount of testing does not make much sense.
  - hvidgaard 10y ago
    More unittests would be like weighting how much bone, muscle, fat, organ, water, ect. you have in your body, instead of just one combined weight. You can test known edge case input along with the general input, and it really does give you more robustness.
  - pdw 10y ago
    Yes. Despite all their tests, they still have monthly bugfix releases, often fixing years-old bugs.
    - adrianN 10y ago
      Do you think they would have fewer bugs if they had worse test coverage?
  - sAbakumoff 10y ago
    I was wondering why did this comment receive multiple downvotes?
  - adrianN 10y ago
    It is pretty well demonstrated that testing your software more reduces the number of bugs. That's why all safety critical software is required by industry standards to have complete test coverage. Above a certain amount of tests it is maybe not the most cost-effective way to reduce defects, but arguing that testing is not the way to go seems misguided to me.
    - sAbakumoff 10y ago
      Well, no one can argue that testing is imperative. It's just the numbers that SQLite exposed might suggest that their test coverage is above any reasonable limits ) > the SQLite library consists of approximately 122.9 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 745 times as much test code and test scripts - 91596.1 KSLOC. Though again, I admire SQLite and use it my projects all the time.
bdarnell 10y ago
The `sqllogictest` suite (https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki) is a great resource that is not SQLite-specific. It's a huge collection of queries and expected responses (in an easily-parseable format) that mostly sticks to the SQL standard so it can be run against any database backend. We're using it in CockroachDB.

jncraton 10y ago

SQLite is frequently cited for having incredible robustness, and this is certainly related to incredible test coverage. Many of the stats on that page are impressive, but the one that always gets me is that for 122 thousand lines of production code, the project has 90 million lines of tests.

chrisweekly 10y ago
Tests are like guard rails. Nobody's saying they shouldn't be there, but they're safety nets. [Rich Hickey's _amazing_ talk on simple vs easy](https://github.com/matthiasn/talk-transcripts/blob/master/Hickey_Rich/SimpleMadeEasy.md https://github.com/matthiasn/talk-transcripts/blob/master/Hi...)
sAbakumoff 10y ago
Though I am a huge fan of SQLite, I am not sure if the incredible test coverage necessarily leads to success : "Trying to improve software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software-development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't just test more; develop better."(McConnell, Steve (2009-11-30). Code Complete (Kindle Location 16276). Microsoft Press. Kindle Edition) Another unique aspect of SQLite code base is total sticking to KISS(http://www.jarchitect.com/Blog/?p=2392 http://www.jarchitect.com/Blog/?p=2392)
- austinpray 10y ago
  That is not an apt analogy. Unit tests improve code quality: code that is easily testable has high cohesion and low coupling. Tests can also serve as valuable documentation.
  - brutuscat 10y ago
    Not quite a lot apparently.... https://www.scribd.com/document/225906827/Software-Defect-Origins-and-Removal-Methods2013 https://www.scribd.com/document/225906827/Software-Defect-Or... https://theholyjava.wordpress.com/2015/01/26/challenging-myself-with-copliens-why-most-unit-testing-is-waste/ https://theholyjava.wordpress.com/2015/01/26/challenging-mys...
  - sAbakumoff 10y ago
    That is analogy I copied from McConnell, Steve (2009-11-30). Code Complete (Kindle Location 16276). Microsoft Press. Kindle Edition. I tend to rely on what I written on this book
    - sigzero 10y ago
      That's fine. It's a good book. That doesn't mean the analogy works in every situation. I happen to think it doesn't in this one. Dr. Hipp is a great coder. Telling him to "develop better" from that analogy just falls apart. Tests are there for a reason and Dr. Hipp uses them to great effect on the quality of SQLite.
      - sAbakumoff 10y ago
        Yeah I agree. Actually the whole point of the original comment was about SQLite's robustness as the result of multiple development practices they use(including testing of course), but testing coverage can't lead to success itself. That's it)
  - Someone 10y ago
    High cohesion and low coupling are good, and easily testable correlates with that, but having lots of unit tests doesn't imply that the code is easily testable, and not having unit tests doesn't imply that code has low cohesion or high coupling. It's even possible code needs many lines of testing partly because it lacks in ease of testing. In the case of SQLite I think it mostly is because of hard work to fulfill the ambition to deliver a robust project and because of the existence of fuzzing, which can automate test generation.
- andrewflnr 10y ago
  If you want to lose weight, you need to be able to actually measure your weight. Codebases with inadequate tests can't do this accurately, so they don't even know what changes to their diet/development process help.
  - sAbakumoff 10y ago
    >If you want to lose weight, you need to be able to actually measure your weight. That's not true. There are multiple ways to track the progress of losing weight, starting from simple "look at the mirror" ending with fat percent measure.
  - sAbakumoff 10y ago
    Also, the quote does not say that you have to avoid testing, it says that trying to improve software quality by increasing the amount of testing does not make much sense.
- hvidgaard 10y ago
  More unittests would be like weighting how much bone, muscle, fat, organ, water, ect. you have in your body, instead of just one combined weight. You can test known edge case input along with the general input, and it really does give you more robustness.
- pdw 10y ago
  Yes. Despite all their tests, they still have monthly bugfix releases, often fixing years-old bugs.
  - adrianN 10y ago
    Do you think they would have fewer bugs if they had worse test coverage?
- sAbakumoff 10y ago
  I was wondering why did this comment receive multiple downvotes?
- adrianN 10y ago
  It is pretty well demonstrated that testing your software more reduces the number of bugs. That's why all safety critical software is required by industry standards to have complete test coverage. Above a certain amount of tests it is maybe not the most cost-effective way to reduce defects, but arguing that testing is not the way to go seems misguided to me.
  - sAbakumoff 10y ago
    Well, no one can argue that testing is imperative. It's just the numbers that SQLite exposed might suggest that their test coverage is above any reasonable limits ) > the SQLite library consists of approximately 122.9 KSLOC of C code. (KSLOC means thousands of "Source Lines Of Code" or, in other words, lines of code excluding blank lines and comments.) By comparison, the project has 745 times as much test code and test scripts - 91596.1 KSLOC. Though again, I admire SQLite and use it my projects all the time.

bdarnell 10y ago

The `sqllogictest` suite (https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki https://www.sqlite.org/sqllogictest/doc/trunk/about.wiki) is a great resource that is not SQLite-specific. It's a huge collection of queries and expected responses (in an easily-parseable format) that mostly sticks to the SQL standard so it can be run against any database backend. We're using it in CockroachDB.