8 ms·

How SQLite Is Tested

200 points by asymmetric 10y ago

antonios 10y ago
Welcome, most reliable karma generator on HN.
sgt 10y ago
asymmetric currently holding a karma level of 282. Let's all observe how this improves throughout the day. Will he reach 500? Even a 1000? There's only one way to find out - wait and see!
panic 10y ago
The section on static analysis is somewhat damning: Static analysis has not proven to be especially helpful in finding bugs in SQLite. Static analysis has found a few bugs in SQLite, but those are the exceptions. More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis.
- masklinn 10y ago
  That's only "damning" if like SQLite your project has several orders of magnitude more test code than tested code, which I'm reasonably certain is not the case.
  - fit2rule 10y ago
    Static analysis is rarely helpful after the fact. Its usefulness is directly related to its entry into the project timeline. Just like "handle all warnings, always" is a generally good mantra, having static analysis is supposed to keep you ahead of the curve. If you don't do it early, and often, you're not doing it properly. Also, having these tools, but no coding rules by code contributors, enforced with real code-reviews and reviewer(s), will also deliver 'not worth it' results. Its a process, not a chisel.
    - mikeash 10y ago
      That doesn't match what I've seen. When Apple added support for the clang static analyzer to Xcode, a ton of Objective-C programmers started running it on their code bases, and among those I know, it found a ton of bugs that had gone dormant for years.
      - fit2rule 10y ago
        Sure, of course it'll find bugs after the fact. But the value to the project in doing static analysis is in not adding bugs in the first place, because you caught them with static analysis as a standard part of your development process. Who knows what those unknown bugs cost those developers over the years ..
        mikeash 10y ago
        Are you implying that there is no value in finding bugs after the fact? Sure, it's better to catch them up front. But catching them later is still extremely valuable.
        fit2rule 10y ago
        Catching them before they're deployed anywhere is the point. Deploying quality software from 1.0 is a different order of value than 'experience improved for the 1.1 user because: bugs fixed'. Both circumstances have value: I meant only to state that static analysis should be up-front, to get the absolute most value out of it, because it will massively improve your software quality to have it there.
  - deleted 10y ago
    [deleted]
- TickleSteve 10y ago
  SQLite is already a mature codebase. You would hope (expect?) that most issues that a Static Analyser would pick up would be purely stylistic ones in this case. This speaks more to the maturity and robustness of the SQLite codebase than it does of static analysis as a type of tool. Static Analysis is very useful in the development phase of a project where you can treat them effectively as compiler warnings if your build system is setup that way. ...and that isn't to say that its not worth still using SA on mature codebases, just because a codebase is mature does not indicate its quality.
  - CodexArcanum 10y ago
    I've found static analysis (and related concepts e.g. type systems) to be extremely useful for safely refactoring code. Much more so than tests, which tend to just break and require rewriting. But as you say, once the code base is mature that kind of large scale code shuffling is less likely. (Not to speak poorly of testing, I'm a big fan of that too. Why not use every tool at one's disposal to ensure quality?)
    - TickleSteve 10y ago
      true, though there is a difference.... SA will capture issues related to the implementation but in itself it has no knowledge of application-specific behaviour. Tests on the other hand do have application-specific knowledge and this is where their value lies.
- nickpsecurity 10y ago
  TickleSteve made one good point. I'll add to that one two things: the analyzer; the point of them. The analyzer itself sucks compared to a number of others. Most commercial work tries to use several given each invariably misses things. The other point is that many use static analysis to reduce the testing burden by catching problems early. As in, they might not have needed all those tests on undefined behavior and such if they coded in a way that passed on static analysis.
  - throwawaysocks 10y ago
    Exactly! The point of static analysis is avoiding the need for "millions and millions" of tests.
- junke 10y ago
  Related talk about Tis-interpreter and SQLite: http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/JourneesNationales/GPL2016/sqlite-in-tis-interpreter.pdf http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/Jour... Note also: > We want verification to be about proving absence of bugs, not about finding bugs
- pcwalton 10y ago
  Well, if your static analysis consists of compiler warnings and a few clang passes while your dynamic analysis is avionics/military grade, of course the dynamic analysis is going to be more effective. Would you expect anything else? A more interesting comparison would be between the most powerful/restrictive static analyses vs. the most powerful dynamic analyses.
- adekok 10y ago
  Once a code base gets mature (i.e. tested mature), static analysis becomes less useful. In my experience, though, it's worth still running it, just to catch issues with new code. Every release of FreeRADIUS (http://freeradius.org http://freeradius.org) gets run through three different static analysis tools. They each find slightly different things. At this point, the bulk of issues they find are false positives. Addressing those doesn't create bugs, and I'm not sure why it would.
  - psaccounts 10y ago
    Could you share which 3 static analysis tools you use? We found Coverity to be prohibitively expensive; but unfortunately there isn't any real alternative!
  - adrianratnapala 10y ago
    Addressing those doesn't create bugs, and I'm not sure why it would. Anything that changes behaviour could create bugs. Here is an example signed bar = something_possibly_negative(); if(some_case) { bar = calc_returning_unsigned_where_MSB_is_an_error_flag(); if(bar < 0) return ERROR; } // now bar is valid, whether negative or positive. This is a correct - if dodgy - way to test the MSB. But now if you get a warning about the signed integer not having the full range of the unsigned integer you might "fix" it by changing the first line to long long bar = something_possibly_negative(); Now the signed bar covers the full range of the unsigned calculation - including the garbage in the error case. For more in this vein see: http://ithare.com/best-practices-vs-witch-hunts/ http://ithare.com/best-practices-vs-witch-hunts/
- thesz 10y ago
  http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-easy-way.html http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-ea... Fuzzing found real actionable bugs in SQLite. As I consider fuzzing a variant of static checking (it executes a subset of symbolic evaluation space), I consider the result above as a resounding success of static methods. Also please look at the date when the phrase was added: https://web.archive.org/web/20090205181900/http://sqlite.org/testing.html https://web.archive.org/web/20090205181900/http://sqlite.org... - it's 2009, the very first version of the page! And they didn't change it a bit since. Basically you base your damnation on an outdated information.
  - emeraldd 10y ago
    How does fuzzing fit into static analysis? By my understanding, Static Analysis is defined as examining the code without executing it and fuzzing would inherently require executing the code ...
    - thesz 10y ago
      Executing code with single input value is a subset of (symbolic) execution of the same code with all values. You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. I am sure I sound contrary to "common definitions" - so be it. I also think that dynamic typing is a subset of static typing, more or less for the same reason.
      - emeraldd 10y ago
        > You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. It is entirely possible to break a stream of text down into symbols and examine the relationship of those symbols without evaluating (i.e. executing) them. It is not really necessary or possible to know their meaning/definition at all, only their "shape" or the "part of speech" that they represent. In the simplest form, static analyzers are a set of heuristic pattern matchers that a stream of text is sent through. It might get fancy and build an AST first, but that still does not constitute execution or evaluation in the technical sense.
chris_wot 10y ago
In 2010 I stumbled upn a bug that segfaulted Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=581946 https://bugzilla.mozilla.org/show_bug.cgi?id=581946 The SQLite guy were ridiculously responsive and added all sorts of tests around the fix. It's not every day you can say you found and helped troubleshoot a bug that literally could be affected 100s of millions of users!
- mSparks 10y ago
  Big fan of sqlite. But have to say I'm surprised to see how much testing it gets. While it's great when it works sqlite databases are definitely temperamental. In fact those test suites being so large is possibly part of the problem. Sounds like they are being overly prescriptive in their testing, which is why it's so damn fussy about the order commands are executed without causing it to leak memory left right and center.
  - TickleSteve 10y ago
    regarding being temperamental.... You're going to have to justify that a bit more, that would not be the overwhelming experience around these parts. Not to say its perfect but its the weight of evidence is in SQLites favour here.
    - mSparks 10y ago
      Meh, like I said, big fan of SQLite. But any use of it in a "serious" application will undoubtably run into: Error: database disk image is malformed On a fairly regular basis. Google gives:About 57,400 results So I'm definitely not the only one.
      - TickleSteve 10y ago
        http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption (Just so you know)
deleted 10y ago
[deleted]
cyphar 10y ago
> The TH3 test harness is a set of proprietary tests, [...] TH3 is free to SQLite Consortium members and is available by license to others. I think it's ridiculous that people make proprietary test suites. In my opinion, test suites shouldn't even be copyrightable let alone made proprietary. A test suite is used to define what the precise interface of an API is, it's just a piece of code rather than a standards document. EDIT: To be clear, the actual test framework might be copyrightable. But the actual test cases should not be IMO. It's also a shame that a free software project requires using proprietary software in order to practically test it.
- Tomte 10y ago
  If your API is defined by a test suite you're doing something very, very wrong. Additionally, there are three major test suites for SQLite, and two of them (with a test coverage that still leaves most other open source projects in the dust) are open source and free. I can live with paying members getting additional testing possibilities.
  - mlvljr 10y ago
    Question is, what defines the SQLITE API (if not elaborate tests)? The documentations is a bit scarce, that is :)
- dalke 10y ago
  Facts aren't copywriteable, and in the US a collection of facts is also not copyrwriteable. A test case is structured around an assertion of facts. To that extent, it is not copywriteable. But other parts of a test case, which have creative thought, are copywriteable. This can include the test name and any comments. If you manage to get ahold of the test suite, strip out all of the assertions, and create a new test system which does the same tests, then you might get a lawyer to agree that it's not copyright infringement. Even then, a test case might use something which is under copyright. For example, it may contain a poem, and check if the length of the poem is as expected. That said, if there were no copyright (let's say it's the 1960s, before copyright was extended to software), it could still be covered as a trade secret, and protected under confidentiality agreements. That is, copyright affects what you can do once you get the software, but you might not be able to get it in the first place. Therefore, while I understand and even empathize with your argument, I don't think you should based the argument in copywriteability. Instead, I think you should look to Stallman's argument for why documentation for a free software project should itself be free. If you modify the code, you should be able to modify the documentation instead of starting from scratch. https://www.gnu.org/software/gsl/manual/html_node/Free-Software-Needs-Free-Documentation.html https://www.gnu.org/software/gsl/manual/html_node/Free-Softw... By analogy, if you modify the code, you should be modify the test suite instead of starting from scratch. One obvious solution is to start your own GNU-licensed SQLite test suite, to, ahem, fight against the software hoarders. ;)
youdontknowtho 10y ago
I get that Valgrind is great, but in their opinion Linux is the only platform that people should use for development because of Valgrind support? Do statements like that ever do anything but make people, if not angry, annoyed? Maybe, I'm wrong. Is Valgrind so good that development on any other platform is bound to be more bug ridden? If it is that good why not just throw resources at it so that you could use it for everything? The author says that it approximates the difference in performance between a desktop and phone. What would you need to get it to performance that could be used all the time? Or was this just hyperbole?
jesseryoung 10y ago
This seems like an extreme amount of tests, I would be very interested in hearing how they are managed. Without some sort of system around them I would imagine that many tests would lose their meaning.
rodionos 10y ago
> Millions and millions of test cases I wasn't sure how to interpret it, but then they mention that it's 40100 distinct test cases, parameterized. I would have rephrased it as "40K of parameterized test cases".
onderkalaci 10y ago
The title says "how SQLite is tester", well I don't think the article talks about "how". It only talks about the different types of tests that they apply. I'm curious about "how" it is done. Do you guys run a single "test" button and all the tests are executed? Or, all the tests are done independently? On which platforms do you test?
- SQLite 10y ago
  There is a testing checklist (https://www.sqlite.org/checklists/3130000/index https://www.sqlite.org/checklists/3130000/index is an example). Each item on the checklist is usually just a single "button push" (really a shell command). But we have to push that same button on lots of different platforms.
64bitbrain 10y ago
Did I miss something, because I didn't see performance testing. Given some specific test bed, which has X amount of RAM, How much Insertion Per Second(IPS) I can get?
- Cthulhu_ 10y ago
  > In addition to the three major test harnesses, there several other small programs that implement specialized tests. > 4. The "speedtest1.c" program estimates the performance of SQLite under a typical workload. > 5. The "mptester.c" program is a stress test for multiple processes concurrently reading and writing a single database. > 6. The "threadtest3.c" program is a stress test for multiple threads using SQLite simultaneously.
  - 64bitbrain 10y ago
    Thanks! I dont know how I missed that.
dang 10y ago
Many prior discussions: https://hn.algolia.com/?query=How%20SQLite%20Is%20Tested&sort=byDate&dateRange=all&type=story&storyText=false&prefix&page=0 https://hn.algolia.com/?query=How%20SQLite%20Is%20Tested&sor....

200 points by asymmetric 10y ago

Welcome, most reliable karma generator on HN.

sgt 10y ago

asymmetric currently holding a karma level of 282. Let's all observe how this improves throughout the day. Will he reach 500? Even a 1000? There's only one way to find out - wait and see!

panic 10y ago

The section on static analysis is somewhat damning: Static analysis has not proven to be especially helpful in finding bugs in SQLite. Static analysis has found a few bugs in SQLite, but those are the exceptions. More bugs have been introduced into SQLite while trying to get it to compile without warnings than have been found by static analysis.

masklinn 10y ago
That's only "damning" if like SQLite your project has several orders of magnitude more test code than tested code, which I'm reasonably certain is not the case.
- fit2rule 10y ago
  Static analysis is rarely helpful after the fact. Its usefulness is directly related to its entry into the project timeline. Just like "handle all warnings, always" is a generally good mantra, having static analysis is supposed to keep you ahead of the curve. If you don't do it early, and often, you're not doing it properly. Also, having these tools, but no coding rules by code contributors, enforced with real code-reviews and reviewer(s), will also deliver 'not worth it' results. Its a process, not a chisel.
  - mikeash 10y ago
    That doesn't match what I've seen. When Apple added support for the clang static analyzer to Xcode, a ton of Objective-C programmers started running it on their code bases, and among those I know, it found a ton of bugs that had gone dormant for years.
    - fit2rule 10y ago
      Sure, of course it'll find bugs after the fact. But the value to the project in doing static analysis is in not adding bugs in the first place, because you caught them with static analysis as a standard part of your development process. Who knows what those unknown bugs cost those developers over the years ..
      - mikeash 10y ago
        Are you implying that there is no value in finding bugs after the fact? Sure, it's better to catch them up front. But catching them later is still extremely valuable.
        fit2rule 10y ago
        Catching them before they're deployed anywhere is the point. Deploying quality software from 1.0 is a different order of value than 'experience improved for the 1.1 user because: bugs fixed'. Both circumstances have value: I meant only to state that static analysis should be up-front, to get the absolute most value out of it, because it will massively improve your software quality to have it there.
- deleted 10y ago
  [deleted]
TickleSteve 10y ago
SQLite is already a mature codebase. You would hope (expect?) that most issues that a Static Analyser would pick up would be purely stylistic ones in this case. This speaks more to the maturity and robustness of the SQLite codebase than it does of static analysis as a type of tool. Static Analysis is very useful in the development phase of a project where you can treat them effectively as compiler warnings if your build system is setup that way. ...and that isn't to say that its not worth still using SA on mature codebases, just because a codebase is mature does not indicate its quality.
- CodexArcanum 10y ago
  I've found static analysis (and related concepts e.g. type systems) to be extremely useful for safely refactoring code. Much more so than tests, which tend to just break and require rewriting. But as you say, once the code base is mature that kind of large scale code shuffling is less likely. (Not to speak poorly of testing, I'm a big fan of that too. Why not use every tool at one's disposal to ensure quality?)
  - TickleSteve 10y ago
    true, though there is a difference.... SA will capture issues related to the implementation but in itself it has no knowledge of application-specific behaviour. Tests on the other hand do have application-specific knowledge and this is where their value lies.
nickpsecurity 10y ago
TickleSteve made one good point. I'll add to that one two things: the analyzer; the point of them. The analyzer itself sucks compared to a number of others. Most commercial work tries to use several given each invariably misses things. The other point is that many use static analysis to reduce the testing burden by catching problems early. As in, they might not have needed all those tests on undefined behavior and such if they coded in a way that passed on static analysis.
- throwawaysocks 10y ago
  Exactly! The point of static analysis is avoiding the need for "millions and millions" of tests.
junke 10y ago
Related talk about Tis-interpreter and SQLite: http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/JourneesNationales/GPL2016/sqlite-in-tis-interpreter.pdf http://gdr-gpl.cnrs.fr/sites/default/files/documentsGPL/Jour... Note also: > We want verification to be about proving absence of bugs, not about finding bugs
pcwalton 10y ago
Well, if your static analysis consists of compiler warnings and a few clang passes while your dynamic analysis is avionics/military grade, of course the dynamic analysis is going to be more effective. Would you expect anything else? A more interesting comparison would be between the most powerful/restrictive static analyses vs. the most powerful dynamic analyses.
adekok 10y ago
Once a code base gets mature (i.e. tested mature), static analysis becomes less useful. In my experience, though, it's worth still running it, just to catch issues with new code. Every release of FreeRADIUS (http://freeradius.org http://freeradius.org) gets run through three different static analysis tools. They each find slightly different things. At this point, the bulk of issues they find are false positives. Addressing those doesn't create bugs, and I'm not sure why it would.
- psaccounts 10y ago
  Could you share which 3 static analysis tools you use? We found Coverity to be prohibitively expensive; but unfortunately there isn't any real alternative!
- adrianratnapala 10y ago
  Addressing those doesn't create bugs, and I'm not sure why it would. Anything that changes behaviour could create bugs. Here is an example signed bar = something_possibly_negative(); if(some_case) { bar = calc_returning_unsigned_where_MSB_is_an_error_flag(); if(bar < 0) return ERROR; } // now bar is valid, whether negative or positive. This is a correct - if dodgy - way to test the MSB. But now if you get a warning about the signed integer not having the full range of the unsigned integer you might "fix" it by changing the first line to long long bar = something_possibly_negative(); Now the signed bar covers the full range of the unsigned calculation - including the garbage in the error case. For more in this vein see: http://ithare.com/best-practices-vs-witch-hunts/ http://ithare.com/best-practices-vs-witch-hunts/
thesz 10y ago
http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-easy-way.html http://lcamtuf.blogspot.ru/2015/04/finding-bugs-in-sqlite-ea... Fuzzing found real actionable bugs in SQLite. As I consider fuzzing a variant of static checking (it executes a subset of symbolic evaluation space), I consider the result above as a resounding success of static methods. Also please look at the date when the phrase was added: https://web.archive.org/web/20090205181900/http://sqlite.org/testing.html https://web.archive.org/web/20090205181900/http://sqlite.org... - it's 2009, the very first version of the page! And they didn't change it a bit since. Basically you base your damnation on an outdated information.
- emeraldd 10y ago
  How does fuzzing fit into static analysis? By my understanding, Static Analysis is defined as examining the code without executing it and fuzzing would inherently require executing the code ...
  - thesz 10y ago
    Executing code with single input value is a subset of (symbolic) execution of the same code with all values. You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. I am sure I sound contrary to "common definitions" - so be it. I also think that dynamic typing is a subset of static typing, more or less for the same reason.
    - emeraldd 10y ago
      > You cannot "examine the code" without executing it, partially or in full, with some input or intermediate values or (symbolically) with all of them. It is entirely possible to break a stream of text down into symbols and examine the relationship of those symbols without evaluating (i.e. executing) them. It is not really necessary or possible to know their meaning/definition at all, only their "shape" or the "part of speech" that they represent. In the simplest form, static analyzers are a set of heuristic pattern matchers that a stream of text is sent through. It might get fancy and build an AST first, but that still does not constitute execution or evaluation in the technical sense.

chris_wot 10y ago

In 2010 I stumbled upn a bug that segfaulted Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=581946 https://bugzilla.mozilla.org/show_bug.cgi?id=581946 The SQLite guy were ridiculously responsive and added all sorts of tests around the fix. It's not every day you can say you found and helped troubleshoot a bug that literally could be affected 100s of millions of users!

mSparks 10y ago
Big fan of sqlite. But have to say I'm surprised to see how much testing it gets. While it's great when it works sqlite databases are definitely temperamental. In fact those test suites being so large is possibly part of the problem. Sounds like they are being overly prescriptive in their testing, which is why it's so damn fussy about the order commands are executed without causing it to leak memory left right and center.
- TickleSteve 10y ago
  regarding being temperamental.... You're going to have to justify that a bit more, that would not be the overwhelming experience around these parts. Not to say its perfect but its the weight of evidence is in SQLites favour here.
  - mSparks 10y ago
    Meh, like I said, big fan of SQLite. But any use of it in a "serious" application will undoubtably run into: Error: database disk image is malformed On a fairly regular basis. Google gives:About 57,400 results So I'm definitely not the only one.
    - TickleSteve 10y ago
      http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption (Just so you know)

deleted 10y ago

[deleted]

cyphar 10y ago

> The TH3 test harness is a set of proprietary tests, [...] TH3 is free to SQLite Consortium members and is available by license to others. I think it's ridiculous that people make proprietary test suites. In my opinion, test suites shouldn't even be copyrightable let alone made proprietary. A test suite is used to define what the precise interface of an API is, it's just a piece of code rather than a standards document. EDIT: To be clear, the actual test framework might be copyrightable. But the actual test cases should not be IMO. It's also a shame that a free software project requires using proprietary software in order to practically test it.

Tomte 10y ago
If your API is defined by a test suite you're doing something very, very wrong. Additionally, there are three major test suites for SQLite, and two of them (with a test coverage that still leaves most other open source projects in the dust) are open source and free. I can live with paying members getting additional testing possibilities.
- mlvljr 10y ago
  Question is, what defines the SQLITE API (if not elaborate tests)? The documentations is a bit scarce, that is :)
dalke 10y ago
Facts aren't copywriteable, and in the US a collection of facts is also not copyrwriteable. A test case is structured around an assertion of facts. To that extent, it is not copywriteable. But other parts of a test case, which have creative thought, are copywriteable. This can include the test name and any comments. If you manage to get ahold of the test suite, strip out all of the assertions, and create a new test system which does the same tests, then you might get a lawyer to agree that it's not copyright infringement. Even then, a test case might use something which is under copyright. For example, it may contain a poem, and check if the length of the poem is as expected. That said, if there were no copyright (let's say it's the 1960s, before copyright was extended to software), it could still be covered as a trade secret, and protected under confidentiality agreements. That is, copyright affects what you can do once you get the software, but you might not be able to get it in the first place. Therefore, while I understand and even empathize with your argument, I don't think you should based the argument in copywriteability. Instead, I think you should look to Stallman's argument for why documentation for a free software project should itself be free. If you modify the code, you should be able to modify the documentation instead of starting from scratch. https://www.gnu.org/software/gsl/manual/html_node/Free-Software-Needs-Free-Documentation.html https://www.gnu.org/software/gsl/manual/html_node/Free-Softw... By analogy, if you modify the code, you should be modify the test suite instead of starting from scratch. One obvious solution is to start your own GNU-licensed SQLite test suite, to, ahem, fight against the software hoarders. ;)

youdontknowtho 10y ago

I get that Valgrind is great, but in their opinion Linux is the only platform that people should use for development because of Valgrind support? Do statements like that ever do anything but make people, if not angry, annoyed? Maybe, I'm wrong. Is Valgrind so good that development on any other platform is bound to be more bug ridden? If it is that good why not just throw resources at it so that you could use it for everything? The author says that it approximates the difference in performance between a desktop and phone. What would you need to get it to performance that could be used all the time? Or was this just hyperbole?

jesseryoung 10y ago

This seems like an extreme amount of tests, I would be very interested in hearing how they are managed. Without some sort of system around them I would imagine that many tests would lose their meaning.

rodionos 10y ago

> Millions and millions of test cases I wasn't sure how to interpret it, but then they mention that it's 40100 distinct test cases, parameterized. I would have rephrased it as "40K of parameterized test cases".

onderkalaci 10y ago

The title says "how SQLite is tester", well I don't think the article talks about "how". It only talks about the different types of tests that they apply. I'm curious about "how" it is done. Do you guys run a single "test" button and all the tests are executed? Or, all the tests are done independently? On which platforms do you test?

SQLite 10y ago
There is a testing checklist (https://www.sqlite.org/checklists/3130000/index https://www.sqlite.org/checklists/3130000/index is an example). Each item on the checklist is usually just a single "button push" (really a shell command). But we have to push that same button on lots of different platforms.

64bitbrain 10y ago

Did I miss something, because I didn't see performance testing. Given some specific test bed, which has X amount of RAM, How much Insertion Per Second(IPS) I can get?

Cthulhu_ 10y ago
> In addition to the three major test harnesses, there several other small programs that implement specialized tests. > 4. The "speedtest1.c" program estimates the performance of SQLite under a typical workload. > 5. The "mptester.c" program is a stress test for multiple processes concurrently reading and writing a single database. > 6. The "threadtest3.c" program is a stress test for multiple threads using SQLite simultaneously.
- 64bitbrain 10y ago
  Thanks! I dont know how I missed that.

dang 10y ago

Many prior discussions: https://hn.algolia.com/?query=How%20SQLite%20Is%20Tested&sort=byDate&dateRange=all&type=story&storyText=false&prefix&page=0 https://hn.algolia.com/?query=How%20SQLite%20Is%20Tested&sor....