5 ms·

It's impressive to find a bug in something like sqlite, which is famous for its larger than the codebase test suite

capableweb 4y ago
> larger than the codebase test suite This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. I think what they are famous for is the quality of the testing suite, rather than the amount.
- markerz 4y ago
  I think a good way of looking at the quality of their tests is how they categorize their tests, which focuses on the different types of problems they want to solve: power loss, concurrency, API issues, logic, etc. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- loeg 4y ago
  This is a definition of “serious projects” that excludes almost all serious projects.
  - rat9988 4y ago
    I'm not sure, all of mines at big companies have followed this definition.
- valray 4y ago
  > This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. Reading this comment, I was thinking "Oh that must mean the test code is 2x or maybe even 3x the amount of source code" Going to the SQLite web site, I was surprised to find that the test code is 600x larger than the source code. Impressive. Is this 600:1 ratio typical for other projects? The ones that I have seen are more like 1x or 2x, but I have not worked with many open source systems.
  - baq 4y ago
    They count post-codegen test LOC, the checked in ratio is much smaller - used to be something like 5:1. Still super impressive!
- civopsec 4y ago
  Test to regular code ratio shouldn’t be impressive by itself anyway, considering how repetitive, duplicated, and setup-heavy test code can be.
zoomablemind 4y ago
>... larger than the codebase test suite I wonder if they directly test the concurrent use? It appears that the fix [1] of the OP bug did not lead to any addition/changes in resp. tests. [1]:https://www.sqlite.org/src/info/15f0be8a640e7bfa https://www.sqlite.org/src/info/15f0be8a640e7bfa P.S. looks like Fossil still has issues with content scrolling and wrapping to screen size (mobile).
- otoolep 4y ago
  Actually, they added a unit test in a follow-on change: https://www.sqlite.org/src/info/dc7dd2d3e50e7cc4 https://www.sqlite.org/src/info/dc7dd2d3e50e7cc4
  - zoomablemind 4y ago
    Indeed, thanks for pointing this one out. The test seems to test a shared access in rather a serial order. I wonder if underneath this is actually running as concurrent processes?
    - otoolep 4y ago
      I may not be following your point, but I don't believe the root cause of this issue was a race condition, or anything that might be related to concurrency. The fundamental cause AFAIK was a SQLite connection was attempting to make a state transition (from one type of locking state to another) which shouldn't be allowed under certain circumstances, but the implementation didn't actually enforce this rule. So the added test really does test the root cause.

> larger than the codebase test suite This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. I think what they are famous for is the quality of the testing suite, rather than the amount.

markerz 4y ago
I think a good way of looking at the quality of their tests is how they categorize their tests, which focuses on the different types of problems they want to solve: power loss, concurrency, API issues, logic, etc. https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
loeg 4y ago
This is a definition of “serious projects” that excludes almost all serious projects.
- rat9988 4y ago
  I'm not sure, all of mines at big companies have followed this definition.
valray 4y ago
> This tend to be true for most serious projects, that the amount of test code is greater than that of the code that is being exercised. Reading this comment, I was thinking "Oh that must mean the test code is 2x or maybe even 3x the amount of source code" Going to the SQLite web site, I was surprised to find that the test code is 600x larger than the source code. Impressive. Is this 600:1 ratio typical for other projects? The ones that I have seen are more like 1x or 2x, but I have not worked with many open source systems.
- baq 4y ago
  They count post-codegen test LOC, the checked in ratio is much smaller - used to be something like 5:1. Still super impressive!
civopsec 4y ago
Test to regular code ratio shouldn’t be impressive by itself anyway, considering how repetitive, duplicated, and setup-heavy test code can be.

zoomablemind 4y ago

>... larger than the codebase test suite I wonder if they directly test the concurrent use? It appears that the fix [1] of the OP bug did not lead to any addition/changes in resp. tests. [1]:https://www.sqlite.org/src/info/15f0be8a640e7bfa https://www.sqlite.org/src/info/15f0be8a640e7bfa P.S. looks like Fossil still has issues with content scrolling and wrapping to screen size (mobile).

otoolep 4y ago
Actually, they added a unit test in a follow-on change: https://www.sqlite.org/src/info/dc7dd2d3e50e7cc4 https://www.sqlite.org/src/info/dc7dd2d3e50e7cc4
- zoomablemind 4y ago
  Indeed, thanks for pointing this one out. The test seems to test a shared access in rather a serial order. I wonder if underneath this is actually running as concurrent processes?
  - otoolep 4y ago
    I may not be following your point, but I don't believe the root cause of this issue was a race condition, or anything that might be related to concurrency. The fundamental cause AFAIK was a SQLite connection was attempting to make a state transition (from one type of locking state to another) which shouldn't be allowed under certain circumstances, but the implementation didn't actually enforce this rule. So the added test really does test the root cause.