6 ms·

In 2010 I stumbled upn a bug that segfaulted Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=581946 https://bugzilla.mozilla.org/show_bug.cgi?id=581946

In 2010 I stumbled upn a bug that segfaulted Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=581946 https://bugzilla.mozilla.org/show_bug.cgi?id=581946 The SQLite guy were ridiculously responsive and added all sorts of tests around the fix. It's not every day you can say you found and helped troubleshoot a bug that literally could be affected 100s of millions of users!

mSparks 10y ago
Big fan of sqlite. But have to say I'm surprised to see how much testing it gets. While it's great when it works sqlite databases are definitely temperamental. In fact those test suites being so large is possibly part of the problem. Sounds like they are being overly prescriptive in their testing, which is why it's so damn fussy about the order commands are executed without causing it to leak memory left right and center.
- TickleSteve 10y ago
  regarding being temperamental.... You're going to have to justify that a bit more, that would not be the overwhelming experience around these parts. Not to say its perfect but its the weight of evidence is in SQLites favour here.
  - mSparks 10y ago
    Meh, like I said, big fan of SQLite. But any use of it in a "serious" application will undoubtably run into: Error: database disk image is malformed On a fairly regular basis. Google gives:About 57,400 results So I'm definitely not the only one.
    - TickleSteve 10y ago
      http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption (Just so you know)
      - olavgg 10y ago
        This is so true, some consumer SSD's are so evil that they do not even honor ATA_CMD_FLUSH. Bad things will happen then.
      - mSparks 10y ago
        Yeah. There's that. But there are also numerous memory leaks that have to be carefully managed. Opening and closing the database regularly is usually enough. And if you do manage to let it run long enough to use all the system ram, that will malform the database to. Transactions are particularly bad. One bug I ran into earlier in the year would gobble up 12gb of system ram trying to search and insert a few 10s of mb of text, just because it was leaving prepared statements laying around without telling me.
        TickleSteve 10y ago
        Sounds like a pretty serious bug that would probably affect millions of people.... ... thanks for reporting it. (https://www.sqlite.org/malloc.html https://www.sqlite.org/malloc.html) "No memory leaks. The application is responsible for destroying any objects it allocates." "Robust against allocation failures. If a memory allocation ever fails (that is to say, if malloc() or realloc() ever return NULL) then SQLite will recover gracefully."
        mSparks 10y ago
        Fine. But in practice. By time malloc returns null. It's blown up anything on the system that doesn't recover gracefully. possibly the file system. Which is possibly how the database becomes corrupt. And I see plenty of reported examples and work arounds. Even yum has been suffering from it for years. http://forums.fedoraforum.org/showthread.php?t=260195 http://forums.fedoraforum.org/showthread.php?t=260195 http://www.pvangsgaard.com/2016/03/16/yum-error-database-disk-image-is-malformed/ http://www.pvangsgaard.com/2016/03/16/yum-error-database-dis...
        TickleSteve 10y ago
- aexaey 10y ago
  ...care to give an example for sqlite being "temperamental"?
- has2k1 10y ago
  The testing is only excessive if you do not know the issues(s) they are trying to avoid[1]. [1] https://www.sqlite.org/howtocorrupt.html https://www.sqlite.org/howtocorrupt.html
  - mSparks 10y ago
    I didn't say "excessive" I said "prescriptive". i.e. of the nature (pseudocode): testCommand1(); testCommand2(); testCommand3(); testCommand4(); testCommand5(); testCommand6(); rather than: for 1 to 1000 testCommand(random 1 to 6) 6 lines instead of 2, and less robust testing.
yAnonymous 10y ago
>It's not every day you can say you found and helped troubleshoot a bug that literally could be affected 100s of millions of users! With today's mindset of letting users do most of the testing, it is actually quite common to find bugs like that. However, really nice job of the SQLite devs to follow through so fast.
- dalke 10y ago
  You write "it is actually quite common to find bugs like that" Your definition of "common" is quite different than mine. There are only a handful of packages which are used by "100s of millions of users." If finding and helping to troubleshoot a bug is "common" then that's, what, 1 million user-reported bugs? Mozilla's bug tracker has only 1.3M entries. Now, I've submitted a bug to Mozilla. But I've submitted far more bugs to smaller packages, which don't have 100s of millions of users. So I think it's not that "common to find bugs like that."
  - yAnonymous 10y ago
    >Mozilla's bug tracker has only 1.3M entries Your definition of "only" is quite different than mine. :) Dozens of these bugs have been reported by me (same for Chromium) and I'm not even that much of a bug hunter. So yeah, I'd say it's common.
    - dalke 10y ago
      The "only" is as a ratio. If there are >100M users and only 1.3M bugs then that implies <76 users per reported bug. Given that some of those are developers, or people like you, who report dozens of bugs, the odds of a user reporting a bug is well less than 1:100. I submit that it's far less. The odds of someone finding and helping to troubleshoot the bug are even lower. By comparison, most of the bug report I've sent in are to packages with fewer than 10,000 users.
    - chris_wot 10y ago
      It wasn't that I reported it, it was more that I had a stab at troubleshooting it :-) If I hadn't traced out the SQL statement that was causing the segfaults I think it might not have been found!

mSparks 10y ago

Big fan of sqlite. But have to say I'm surprised to see how much testing it gets. While it's great when it works sqlite databases are definitely temperamental. In fact those test suites being so large is possibly part of the problem. Sounds like they are being overly prescriptive in their testing, which is why it's so damn fussy about the order commands are executed without causing it to leak memory left right and center.

TickleSteve 10y ago
regarding being temperamental.... You're going to have to justify that a bit more, that would not be the overwhelming experience around these parts. Not to say its perfect but its the weight of evidence is in SQLites favour here.
- mSparks 10y ago
  Meh, like I said, big fan of SQLite. But any use of it in a "serious" application will undoubtably run into: Error: database disk image is malformed On a fairly regular basis. Google gives:About 57,400 results So I'm definitely not the only one.
  - TickleSteve 10y ago
    http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/lockingv3.html#how_to_corrupt http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption http://www.sqlite.org/cvstrac/wiki?p=DatabaseCorruption (Just so you know)
    - olavgg 10y ago
      This is so true, some consumer SSD's are so evil that they do not even honor ATA_CMD_FLUSH. Bad things will happen then.
    - mSparks 10y ago
      Yeah. There's that. But there are also numerous memory leaks that have to be carefully managed. Opening and closing the database regularly is usually enough. And if you do manage to let it run long enough to use all the system ram, that will malform the database to. Transactions are particularly bad. One bug I ran into earlier in the year would gobble up 12gb of system ram trying to search and insert a few 10s of mb of text, just because it was leaving prepared statements laying around without telling me.
      - TickleSteve 10y ago
        Sounds like a pretty serious bug that would probably affect millions of people.... ... thanks for reporting it. (https://www.sqlite.org/malloc.html https://www.sqlite.org/malloc.html) "No memory leaks. The application is responsible for destroying any objects it allocates." "Robust against allocation failures. If a memory allocation ever fails (that is to say, if malloc() or realloc() ever return NULL) then SQLite will recover gracefully."
        mSparks 10y ago
        Fine. But in practice. By time malloc returns null. It's blown up anything on the system that doesn't recover gracefully. possibly the file system. Which is possibly how the database becomes corrupt. And I see plenty of reported examples and work arounds. Even yum has been suffering from it for years. http://forums.fedoraforum.org/showthread.php?t=260195 http://forums.fedoraforum.org/showthread.php?t=260195 http://www.pvangsgaard.com/2016/03/16/yum-error-database-disk-image-is-malformed/ http://www.pvangsgaard.com/2016/03/16/yum-error-database-dis...
        TickleSteve 10y ago
aexaey 10y ago
...care to give an example for sqlite being "temperamental"?
has2k1 10y ago
The testing is only excessive if you do not know the issues(s) they are trying to avoid[1]. [1] https://www.sqlite.org/howtocorrupt.html https://www.sqlite.org/howtocorrupt.html
- mSparks 10y ago
  I didn't say "excessive" I said "prescriptive". i.e. of the nature (pseudocode): testCommand1(); testCommand2(); testCommand3(); testCommand4(); testCommand5(); testCommand6(); rather than: for 1 to 1000 testCommand(random 1 to 6) 6 lines instead of 2, and less robust testing.

yAnonymous 10y ago

>It's not every day you can say you found and helped troubleshoot a bug that literally could be affected 100s of millions of users! With today's mindset of letting users do most of the testing, it is actually quite common to find bugs like that. However, really nice job of the SQLite devs to follow through so fast.

dalke 10y ago
You write "it is actually quite common to find bugs like that" Your definition of "common" is quite different than mine. There are only a handful of packages which are used by "100s of millions of users." If finding and helping to troubleshoot a bug is "common" then that's, what, 1 million user-reported bugs? Mozilla's bug tracker has only 1.3M entries. Now, I've submitted a bug to Mozilla. But I've submitted far more bugs to smaller packages, which don't have 100s of millions of users. So I think it's not that "common to find bugs like that."
- yAnonymous 10y ago
  >Mozilla's bug tracker has only 1.3M entries Your definition of "only" is quite different than mine. :) Dozens of these bugs have been reported by me (same for Chromium) and I'm not even that much of a bug hunter. So yeah, I'd say it's common.
  - dalke 10y ago
    The "only" is as a ratio. If there are >100M users and only 1.3M bugs then that implies <76 users per reported bug. Given that some of those are developers, or people like you, who report dozens of bugs, the odds of a user reporting a bug is well less than 1:100. I submit that it's far less. The odds of someone finding and helping to troubleshoot the bug are even lower. By comparison, most of the bug report I've sent in are to packages with fewer than 10,000 users.
  - chris_wot 10y ago
    It wasn't that I reported it, it was more that I had a stab at troubleshooting it :-) If I hadn't traced out the SQL statement that was causing the segfaults I think it might not have been found!