15 ms·

I found a bug in SQLite

584 points by otoolep 4y ago

habibur 4y ago
Even though SQLite bugs are rare, those can be found occasionally. I remember a bug finder took the sqlite documentation off their website. Collected all their keywords, made up millions of jumbled up queries of random combination between keywords and then ran those overnight to find 10 bugs where the engine crashed. And yes those were also reported and fixed quickly.
- markerz 4y ago
  Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! I did something like this in college when I was implementing a compiler and it caught a lot of really weird bugs in a lot of peoples codes. The beautiful part of some fuzzers though is they can automatically simplify their complicated test cases to something you can actually look at and reason about.
  - carlmr 4y ago
    >Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! QuickCheck style testing is maybe also worth a mention here. Instead of using any possible inputs, like in fuzzing, you restrict yourself to legal inputs, like the keywords here, to get maybe less random crashes, but more likely to find useful corner cases because of the restriction on the search space.
- return_to_monke 4y ago
  so basically, fuzzing. https://en.m.wikipedia.org/wiki/Fuzzing https://en.m.wikipedia.org/wiki/Fuzzing
  - rsaxvc 4y ago
    Sounds more like smart Monkey testing, much older, less feedback. https://en.m.wikipedia.org/wiki/Monkey_testing https://en.m.wikipedia.org/wiki/Monkey_testing Edit: looks like some consider these the same nowadays.
    - robocat 4y ago
      I wonder if ChatGPT would be good at generating various fuzz testing queries. Maybe too slow? Maybe too repetitive? Maybe too unvarying? Sorry for mentioning ChatGPT (to those who are sick of the infection).
      - mattygee 4y ago
        Actually, with one of the problems being faced in that space at present, especially for applications involving factual answers, being "hallucinations" ( ie. essentially as I understand it the level of "creativity" in responses) such "creativity" may well be quite suited to finding those various unusual edge cases.
        TeMPOraL 4y ago
        You know this common trope in fiction and jokes, that advanced computers & AIs can be easily defeated by feeding them a logical inconsistency? Guess what, we now have a chatbot AI that not only doesn't mind working with nonsense input, it will happily produce logically-inconsistent statements on its own, and can do it so sneakily and convincingly, that it's the human operator who could end up believing logically inconsistent statements, without even realizing it, and possibly end up in serious trouble some time later.
- YetAnotherNick 4y ago
  SQLite is continuosly fuzz tested by at least 2-3 professional teams[0], I am really amazed you could find 10 bugs in one day. Would love to read your write up in this! [0]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
  - deleted 4y ago
    [deleted]
techn00 4y ago
It's impressive to find a bug in something like sqlite, which is famous for its larger than the codebase test suite
- capableweb 4y ago
  > larger than the codebase test suite This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. I think what they are famous for is the quality of the testing suite, rather than the amount.
  - markerz 4y ago
    I think a good way of looking at the quality of their tests is how they categorize their tests, which focuses on the different types of problems they want to solve: power loss, concurrency, API issues, logic, etc. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
  - loeg 4y ago
    This is a definition of “serious projects” that excludes almost all serious projects.
    - rat9988 4y ago
      I'm not sure, all of mines at big companies have followed this definition.
  - valray 4y ago
    > This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. Reading this comment, I was thinking "Oh that must mean the test code is 2x or maybe even 3x the amount of source code" Going to the SQLite web site, I was surprised to find that the test code is 600x larger than the source code. Impressive. Is this 600:1 ratio typical for other projects? The ones that I have seen are more like 1x or 2x, but I have not worked with many open source systems.
password4321 4y ago
Long long ago I lost my SMS database on my first Android phone to corruption. The worst part was if the app encountered an error opening the database, it just deleted it and started over -- no chance of repair to rescue any of the data. I don't think this is done this way anymore. After that I have installed SMS Backup+ first thing on every new phone.
- password4321 4y ago
  https://stackoverflow.com/questions/7764943/what-can-be-done-about-the-fact-that-android-automatically-deletes-corrupt-sqlit https://stackoverflow.com/questions/7764943/what-can-be-done...
mort96 4y ago
I'd be interested to read some discussion on why this wasn't caught earlier. When you have some thing with supports read + write and supports access from multiple threads/processes, I'd expect "read as fast as possible from one thread, write as fast as possible from another" to be one of the most obvious tests to write.
- loeg 4y ago
  Seems like it was particularly the lock upgrade path in the memdb vfs — so a little weird case that must have not been covered adequately. https://www.sqlite.org/src/info/15f0be8a640e7bfa https://www.sqlite.org/src/info/15f0be8a640e7bfa
  - deleted 4y ago
    [deleted]
  - morelisp 4y ago
    I vaguely recall some variant of MC/DC coverage that also treats each bit in any bit flag checks as independent boolean conditions. It seems like there could be a similar variant that requires checking each value of such a "leveled" enum independently; if your enum says you can take on values 0-5 and the logic checks <= 2 and <= 5, nonetheless there should be a test case for 0, 1, 3, and 4.
- YetAnotherNick 4y ago
  Also this is not even a silent bug, so I would have expected that this would have been caught by someone in production.
- xk3 4y ago
  I've experienced this a few times before but I just ignored it because I never had access issues after waiting for the other process to finish writing. I wonder what the patch will really do, is it just a better error message or will SQLITE show only committed data?
jeroen79 4y ago
still just submit a bugreport instead of making a fuss of it.
- 0cf8612b2e1e 4y ago
  People are allowed to be proud of their accomplishments. SQLite is extremely high quality software. Identifying a mistake is practically equivalent to receiving a Knuth check. Maybe this post will inspire others on how to locate other bugs, improving the world for the rest of us.
- wistlo 4y ago
  I found a bug in mySQL and after the mySQL team fixed it, I included a summary of it in my annual employee review. Bug fixing widely-used software is not part of my role, but the company recognized it as an accomplishment.
- bawolff 4y ago
  It is impressive for very popular software to find bugs. But even if it wasnt, its still a blog post. The entire point is to talk about what you have been doing. Personally, My blog is super inane.
- sigjuice 4y ago
  This is all rather tasteful. Making a fuss would be registering a domain name and making a whole freaking website just for the bug.
jacquesm 4y ago
No such thing as software without bugs, but given the incredibly widespread use that SQLite sees the quality as evidenced by the fact that finding a bug is news by itself is extremely high. Something to strive for.
- The_Colonel 4y ago
  It's news only because the author wrote a blogpost about it. There are bugs fixed in every SQLite release.
MrWiffles 4y ago
Props to the author, this was well written. Clear and concise, it was easy to follow. Not like my ratings and ravings! ;-)
- otoolep 4y ago
  Glad you enjoyed it -- thanks.
orf 4y ago
Lots of people saying that SQLite is super high quality and finding a bug is so rare. It’s not. I found one on a simple query utilising WHERE EXISTS [1]. Reporting it to a weird forum was also a horrible experience. It’s high quality software, don’t get me wrong, but the infamous 100% test coverage doesn’t make it somehow immune to issues, or imply that the issues you do find are of a certain level of complexity. Nothing is back and white like that. 1. https://sqlite.org/forum/forumpost/452888d3b1?t=c&unf https://sqlite.org/forum/forumpost/452888d3b1?t=c&unf
- qbasic_forever 4y ago
  What exactly was a 'horrible experience' with the report you filed? Richard himself promptly replied in less than 24 hours that the issue was resolved and previously seen by Firefox too. I am at a genuine loss, what was horrible here??
  - orf 4y ago
    It was a while ago, but if I recall I ended up having to sign up several times, the post failed to be created for some reason and something else. There was a captcha involved I think. I don’t remember the specifics, but I do remember coming away from it with a feeling of “wow, that was an atrocious experience. I wonder what the drop off rate is”
    - devwastaken 4y ago
      Sounds about right, most forum software is buggy and fails, but those failures don't get sent to the maintainers. It just waits until someone whom knows the maintainers says "uhhh it's broke.".
    - zoomablemind 4y ago
      Perhaps a while ago... For quite some time SQLite official forum has been running its own version of Forum software (part of Fossil). It allows Anonymous login, with captcha (again, Fossil handles), yeah, but this is nothing unexpected. So in general, the path to Forum presence is fairly unimpeded, well, it is moderated. Reasonably stated issues seem to get attention. Though, some issues indeed need a push to be recognized as such, as it's a public forum, so other users may express their "other" opinions... All in all it's Freedom of Reasonable speech in action. I believe there's a different channel for reporting security-related issues. Again, it's through the Forum, but there's a private message feature for signed-in users.
de6u99er 4y ago
Did you have to get baptized first to be allowed reporting a bug?
- kanonieer 4y ago
  Obviously. This is a consecrated codebase[1]. [1]: https://www.sqlite.org/codeofethics.html https://www.sqlite.org/codeofethics.html
lifeisstillgood 4y ago
I understand the comment about "intellectual honesty" about those pesky transient bugs. It's really hard to know where a hard to reproduce bug is on the cost benefit spectrum - and that is the crux - not knowing enough about the bug to determine it's negative weight means you are essentially guessing both sides of the equation. It's probably not the best idea, it waiting till users find it at east gives a good idea of the prior
- quickthrower2 4y ago
  Yeah the old "boss, I have spent 2 days investigating the bug, and I think it is best overall if we don't fix it. maybe show a better message".
einpoklum 4y ago
> I found a bug in SQLite Well, good thing it wasn't a bug in the C compiler you were building sqlite with... even those can come up occasionally.
ChuckMcM 4y ago
The lesson here is never ignore your own "Huh, that's weird." experience when using software. :-)
- jicea 4y ago
  A French proverb is « Il n’y a pas de fumée sans feu », means there is no smoke without fire. I use it a lot in my day work when I try not to watch a suspicious behavior…
  - bewaretheirs 4y ago
    The usual form of that proverb in English is "When there's smoke, there's fire".
    - aidos 4y ago
      Is it? Interestingly I’ve only heard it as “no smoke without fire”
      - rgbrgb 4y ago
        “Where there’s smoke, there’s fire” is what I’ve heard (from fire heavy Southern California).
      - thedrexster 4y ago
        "Where there's smoke, there's fire" is how I've heard that proverb
      - berny222 4y ago
        This is how we phrase it in Australia.
        brokenmachine 4y ago
        I'm Australian and have only heard, "where there's smoke, there's fire".
fastaguy88 4y ago
I have seen similar error messages of the "database is corrupted ..." type with MariaDB when I simply typed in my SQL incorrectly.
nodesocket 4y ago
My first time coming across rqlite. Looks awesome. I just finished writing a poor man’s centralized store of SQLite writing the DB file to Minio (s3 compliant) storage but of course has race conditions all over the place. Gonna take a look at replacing with rqlite.
adius 4y ago
Just a few days ago I found a serious security issue in SQLite: https://sqlite.org/forum/forumpost/07beac8056151b2f https://sqlite.org/forum/forumpost/07beac8056151b2f It was also promptly fixed, but it makes me feel like the millions of tests sound better than they are in reality …
- cube00 4y ago
  At least the tests help avoid regressions, you can bet these bugs won't come back again.
- TAForObvReasons 4y ago
  There was an interesting comment a few months ago [1] > Not mentioned is that the full test sqlite test suite is proprietary and you need a super expensive sqlite foundation membership to get access to it. According to Dr Hipp [2], no one bought the test suite. So there are definitely deficiencies in the test suite which may have been better addressed if the full test suite was open. [1] https://news.ycombinator.com/item?id=33346661 https://news.ycombinator.com/item?id=33346661 [2] https://corecursive.com/066-sqlite-with-richard-hipp/#billions-of-tests https://corecursive.com/066-sqlite-with-richard-hipp/#billio...
  - notRobot 4y ago
    From the second link: > We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the source tree. Anybody can download the source code and run my test and run all those. They don’t provide 100% test coverage but they do test all the features very thoroughly. The 100% MCD tests, that’s called TH3. That’s proprietary. I had the idea that we would sell those tests to avionics manufacturers and make money that way. We’ve sold exactly zero copies of that so that didn’t really work out. It did work out really well for us in that it keeps our product really solid and it enables us to turn around new features and new bug fixes very fast.
    - ec109685 4y ago
      It also makes forking SQLite infeasible since any new changes will be woefully under tested.
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
neya 4y ago
The top comment is...sad yet funny: Euphorbium December 11, 2022 at 2:31 pm I think I hit the same bug in django, and it took them 5 years to fix it. Django tagline is “webframework for perfectionists with deadlines”. I was fired because of this bug.

584 points by otoolep 4y ago

Even though SQLite bugs are rare, those can be found occasionally. I remember a bug finder took the sqlite documentation off their website. Collected all their keywords, made up millions of jumbled up queries of random combination between keywords and then ran those overnight to find 10 bugs where the engine crashed. And yes those were also reported and fixed quickly.

markerz 4y ago
Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! I did something like this in college when I was implementing a compiler and it caught a lot of really weird bugs in a lot of peoples codes. The beautiful part of some fuzzers though is they can automatically simplify their complicated test cases to something you can actually look at and reason about.
- carlmr 4y ago
  >Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! QuickCheck style testing is maybe also worth a mention here. Instead of using any possible inputs, like in fuzzing, you restrict yourself to legal inputs, like the keywords here, to get maybe less random crashes, but more likely to find useful corner cases because of the restriction on the search space.
return_to_monke 4y ago
so basically, fuzzing. https://en.m.wikipedia.org/wiki/Fuzzing https://en.m.wikipedia.org/wiki/Fuzzing
- rsaxvc 4y ago
  Sounds more like smart Monkey testing, much older, less feedback. https://en.m.wikipedia.org/wiki/Monkey_testing https://en.m.wikipedia.org/wiki/Monkey_testing Edit: looks like some consider these the same nowadays.
  - robocat 4y ago
    I wonder if ChatGPT would be good at generating various fuzz testing queries. Maybe too slow? Maybe too repetitive? Maybe too unvarying? Sorry for mentioning ChatGPT (to those who are sick of the infection).
    - mattygee 4y ago
      Actually, with one of the problems being faced in that space at present, especially for applications involving factual answers, being "hallucinations" ( ie. essentially as I understand it the level of "creativity" in responses) such "creativity" may well be quite suited to finding those various unusual edge cases.
      - TeMPOraL 4y ago
        You know this common trope in fiction and jokes, that advanced computers & AIs can be easily defeated by feeding them a logical inconsistency? Guess what, we now have a chatbot AI that not only doesn't mind working with nonsense input, it will happily produce logically-inconsistent statements on its own, and can do it so sneakily and convincingly, that it's the human operator who could end up believing logically inconsistent statements, without even realizing it, and possibly end up in serious trouble some time later.
YetAnotherNick 4y ago
SQLite is continuosly fuzz tested by at least 2-3 professional teams[0], I am really amazed you could find 10 bugs in one day. Would love to read your write up in this! [0]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- deleted 4y ago
  [deleted]

techn00 4y ago

It's impressive to find a bug in something like sqlite, which is famous for its larger than the codebase test suite

capableweb 4y ago
> larger than the codebase test suite This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. I think what they are famous for is the quality of the testing suite, rather than the amount.
- markerz 4y ago
  I think a good way of looking at the quality of their tests is how they categorize their tests, which focuses on the different types of problems they want to solve: power loss, concurrency, API issues, logic, etc. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- loeg 4y ago
  This is a definition of “serious projects” that excludes almost all serious projects.
  - rat9988 4y ago
    I'm not sure, all of mines at big companies have followed this definition.
- valray 4y ago
  > This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. Reading this comment, I was thinking "Oh that must mean the test code is 2x or maybe even 3x the amount of source code" Going to the SQLite web site, I was surprised to find that the test code is 600x larger than the source code. Impressive. Is this 600:1 ratio typical for other projects? The ones that I have seen are more like 1x or 2x, but I have not worked with many open source systems.

password4321 4y ago

Long long ago I lost my SMS database on my first Android phone to corruption. The worst part was if the app encountered an error opening the database, it just deleted it and started over -- no chance of repair to rescue any of the data. I don't think this is done this way anymore. After that I have installed SMS Backup+ first thing on every new phone.

password4321 4y ago
https://stackoverflow.com/questions/7764943/what-can-be-done-about-the-fact-that-android-automatically-deletes-corrupt-sqlit https://stackoverflow.com/questions/7764943/what-can-be-done...

mort96 4y ago

I'd be interested to read some discussion on why this wasn't caught earlier. When you have some thing with supports read + write and supports access from multiple threads/processes, I'd expect "read as fast as possible from one thread, write as fast as possible from another" to be one of the most obvious tests to write.

loeg 4y ago
Seems like it was particularly the lock upgrade path in the memdb vfs — so a little weird case that must have not been covered adequately. https://www.sqlite.org/src/info/15f0be8a640e7bfa https://www.sqlite.org/src/info/15f0be8a640e7bfa
- deleted 4y ago
  [deleted]
- morelisp 4y ago
  I vaguely recall some variant of MC/DC coverage that also treats each bit in any bit flag checks as independent boolean conditions. It seems like there could be a similar variant that requires checking each value of such a "leveled" enum independently; if your enum says you can take on values 0-5 and the logic checks <= 2 and <= 5, nonetheless there should be a test case for 0, 1, 3, and 4.
YetAnotherNick 4y ago
Also this is not even a silent bug, so I would have expected that this would have been caught by someone in production.
xk3 4y ago
I've experienced this a few times before but I just ignored it because I never had access issues after waiting for the other process to finish writing. I wonder what the patch will really do, is it just a better error message or will SQLITE show only committed data?

jeroen79 4y ago

still just submit a bugreport instead of making a fuss of it.

0cf8612b2e1e 4y ago
People are allowed to be proud of their accomplishments. SQLite is extremely high quality software. Identifying a mistake is practically equivalent to receiving a Knuth check. Maybe this post will inspire others on how to locate other bugs, improving the world for the rest of us.
wistlo 4y ago
I found a bug in mySQL and after the mySQL team fixed it, I included a summary of it in my annual employee review. Bug fixing widely-used software is not part of my role, but the company recognized it as an accomplishment.
bawolff 4y ago
It is impressive for very popular software to find bugs. But even if it wasnt, its still a blog post. The entire point is to talk about what you have been doing. Personally, My blog is super inane.
sigjuice 4y ago
This is all rather tasteful. Making a fuss would be registering a domain name and making a whole freaking website just for the bug.

jacquesm 4y ago

No such thing as software without bugs, but given the incredibly widespread use that SQLite sees the quality as evidenced by the fact that finding a bug is news by itself is extremely high. Something to strive for.

The_Colonel 4y ago
It's news only because the author wrote a blogpost about it. There are bugs fixed in every SQLite release.

MrWiffles 4y ago

Props to the author, this was well written. Clear and concise, it was easy to follow. Not like my ratings and ravings! ;-)

otoolep 4y ago
Glad you enjoyed it -- thanks.

orf 4y ago

Lots of people saying that SQLite is super high quality and finding a bug is so rare. It’s not. I found one on a simple query utilising WHERE EXISTS [1]. Reporting it to a weird forum was also a horrible experience. It’s high quality software, don’t get me wrong, but the infamous 100% test coverage doesn’t make it somehow immune to issues, or imply that the issues you do find are of a certain level of complexity. Nothing is back and white like that. 1. https://sqlite.org/forum/forumpost/452888d3b1?t=c&unf https://sqlite.org/forum/forumpost/452888d3b1?t=c&unf

qbasic_forever 4y ago
What exactly was a 'horrible experience' with the report you filed? Richard himself promptly replied in less than 24 hours that the issue was resolved and previously seen by Firefox too. I am at a genuine loss, what was horrible here??
- orf 4y ago
  It was a while ago, but if I recall I ended up having to sign up several times, the post failed to be created for some reason and something else. There was a captcha involved I think. I don’t remember the specifics, but I do remember coming away from it with a feeling of “wow, that was an atrocious experience. I wonder what the drop off rate is”
  - devwastaken 4y ago
    Sounds about right, most forum software is buggy and fails, but those failures don't get sent to the maintainers. It just waits until someone whom knows the maintainers says "uhhh it's broke.".
  - zoomablemind 4y ago
    Perhaps a while ago... For quite some time SQLite official forum has been running its own version of Forum software (part of Fossil). It allows Anonymous login, with captcha (again, Fossil handles), yeah, but this is nothing unexpected. So in general, the path to Forum presence is fairly unimpeded, well, it is moderated. Reasonably stated issues seem to get attention. Though, some issues indeed need a push to be recognized as such, as it's a public forum, so other users may express their "other" opinions... All in all it's Freedom of Reasonable speech in action. I believe there's a different channel for reporting security-related issues. Again, it's through the Forum, but there's a private message feature for signed-in users.

de6u99er 4y ago

Did you have to get baptized first to be allowed reporting a bug?

kanonieer 4y ago
Obviously. This is a consecrated codebase[1]. [1]: https://www.sqlite.org/codeofethics.html https://www.sqlite.org/codeofethics.html

lifeisstillgood 4y ago

I understand the comment about "intellectual honesty" about those pesky transient bugs. It's really hard to know where a hard to reproduce bug is on the cost benefit spectrum - and that is the crux - not knowing enough about the bug to determine it's negative weight means you are essentially guessing both sides of the equation. It's probably not the best idea, it waiting till users find it at east gives a good idea of the prior

quickthrower2 4y ago
Yeah the old "boss, I have spent 2 days investigating the bug, and I think it is best overall if we don't fix it. maybe show a better message".

einpoklum 4y ago

> I found a bug in SQLite Well, good thing it wasn't a bug in the C compiler you were building sqlite with... even those can come up occasionally.

ChuckMcM 4y ago

The lesson here is never ignore your own "Huh, that's weird." experience when using software. :-)

jicea 4y ago
A French proverb is « Il n’y a pas de fumée sans feu », means there is no smoke without fire. I use it a lot in my day work when I try not to watch a suspicious behavior…
- bewaretheirs 4y ago
  The usual form of that proverb in English is "When there's smoke, there's fire".
  - aidos 4y ago
    Is it? Interestingly I’ve only heard it as “no smoke without fire”
    - rgbrgb 4y ago
      “Where there’s smoke, there’s fire” is what I’ve heard (from fire heavy Southern California).
    - thedrexster 4y ago
      "Where there's smoke, there's fire" is how I've heard that proverb
    - berny222 4y ago
      This is how we phrase it in Australia.
      - brokenmachine 4y ago
        I'm Australian and have only heard, "where there's smoke, there's fire".

fastaguy88 4y ago

I have seen similar error messages of the "database is corrupted ..." type with MariaDB when I simply typed in my SQL incorrectly.

nodesocket 4y ago

My first time coming across rqlite. Looks awesome. I just finished writing a poor man’s centralized store of SQLite writing the DB file to Minio (s3 compliant) storage but of course has race conditions all over the place. Gonna take a look at replacing with rqlite.

adius 4y ago

Just a few days ago I found a serious security issue in SQLite: https://sqlite.org/forum/forumpost/07beac8056151b2f https://sqlite.org/forum/forumpost/07beac8056151b2f It was also promptly fixed, but it makes me feel like the millions of tests sound better than they are in reality …

cube00 4y ago
At least the tests help avoid regressions, you can bet these bugs won't come back again.
TAForObvReasons 4y ago
There was an interesting comment a few months ago [1] > Not mentioned is that the full test sqlite test suite is proprietary and you need a super expensive sqlite foundation membership to get access to it. According to Dr Hipp [2], no one bought the test suite. So there are definitely deficiencies in the test suite which may have been better addressed if the full test suite was open. [1] https://news.ycombinator.com/item?id=33346661 https://news.ycombinator.com/item?id=33346661 [2] https://corecursive.com/066-sqlite-with-richard-hipp/#billions-of-tests https://corecursive.com/066-sqlite-with-richard-hipp/#billio...
- notRobot 4y ago
  From the second link: > We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the source tree. Anybody can download the source code and run my test and run all those. They don’t provide 100% test coverage but they do test all the features very thoroughly. The 100% MCD tests, that’s called TH3. That’s proprietary. I had the idea that we would sell those tests to avionics manufacturers and make money that way. We’ve sold exactly zero copies of that so that didn’t really work out. It did work out really well for us in that it keeps our product really solid and it enables us to turn around new features and new bug fixes very fast.
  - ec109685 4y ago
    It also makes forking SQLite infeasible since any new changes will be woefully under tested.

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

neya 4y ago

The top comment is...sad yet funny: Euphorbium December 11, 2022 at 2:31 pm I think I hit the same bug in django, and it took them 5 years to fix it. Django tagline is “webframework for perfectionists with deadlines”. I was fired because of this bug.