3 ms·

A list of bugs found using this approach is here: https://www.manuelrigger.at/dbms-bugs/ https://www.manuelrigger.at/dbms-bugs/

by j4mie 6y ago

amadvance 6y ago
Finding 179 bugs in SQLite in less than a year is astounding!
- petters 6y ago
  It really is astounding. Given the amount of testing sqlite receives, one cannot help but wondering whether writing a database is beyond the reach of our current development tools.
  - WJW 6y ago
    Clearly writing databases is not beyond the reach of of current development, since we have a bunch of them and they have been doing pretty well at serving production workloads for literally decades. Writing 100% bug-free databases is probably not possible with current development tools, but it might also not be required. Every extra "nine" of reliability costs exponentially more, whether you're looking at uptime, durability or any other metric. At some point the extra costs just don't measure up to the extra correctness gained.
    - ableal 6y ago
      This work seems to have made much cheaper a couple of those "nines" for the logic correctness metric.
  - mrigger 6y ago
    Thank you! SQLite was indeed a challenging target. We could find and report many bugs in it only because all of our bugs were addressed very quickly (typically within a day). For some of the other DBMS that we tested, bugs were fixed slowly, and for some we also saw many unresolved bugs in their bug tracker, which is why we did/could not comprehensively test these DBMS. Before we proposed our testing approaches, the state of the art of finding logic bugs was writing unit tests, which does not scale well, since it requires much manual effort. We believe that we provide the first practical approaches to automatically finding logic bugs. Companies have already started adopting them. For example, PingCAP implemented all three approaches that we proposed in https://github.com/chaos-mesh/go-sqlancer https://github.com/chaos-mesh/go-sqlancer to test their DBMS TiDB. A number of tools are available to test other aspects, like performance or for crash bugs. SQLsmith, for example, is one effective tool that allows finding crash bugs (see https://github.com/anse1/sqlsmith https://github.com/anse1/sqlsmith).
- cheez 6y ago
  Finding ONLY 179 bugs is astounding. If I had written this database.... 179 bugs per line is more likely.

A list of bugs found using this approach is here: https://www.manuelrigger.at/dbms-bugs/ https://www.manuelrigger.at/dbms-bugs/

by j4mie 6y ago

A list of bugs found using this approach is here: https://www.manuelrigger.at/dbms-bugs/ https://www.manuelrigger.at/dbms-bugs/

amadvance 6y ago

Finding 179 bugs in SQLite in less than a year is astounding!

petters 6y ago
It really is astounding. Given the amount of testing sqlite receives, one cannot help but wondering whether writing a database is beyond the reach of our current development tools.
- WJW 6y ago
  Clearly writing databases is not beyond the reach of of current development, since we have a bunch of them and they have been doing pretty well at serving production workloads for literally decades. Writing 100% bug-free databases is probably not possible with current development tools, but it might also not be required. Every extra "nine" of reliability costs exponentially more, whether you're looking at uptime, durability or any other metric. At some point the extra costs just don't measure up to the extra correctness gained.
  - ableal 6y ago
    This work seems to have made much cheaper a couple of those "nines" for the logic correctness metric.
- mrigger 6y ago
  Thank you! SQLite was indeed a challenging target. We could find and report many bugs in it only because all of our bugs were addressed very quickly (typically within a day). For some of the other DBMS that we tested, bugs were fixed slowly, and for some we also saw many unresolved bugs in their bug tracker, which is why we did/could not comprehensively test these DBMS. Before we proposed our testing approaches, the state of the art of finding logic bugs was writing unit tests, which does not scale well, since it requires much manual effort. We believe that we provide the first practical approaches to automatically finding logic bugs. Companies have already started adopting them. For example, PingCAP implemented all three approaches that we proposed in https://github.com/chaos-mesh/go-sqlancer https://github.com/chaos-mesh/go-sqlancer to test their DBMS TiDB. A number of tools are available to test other aspects, like performance or for crash bugs. SQLsmith, for example, is one effective tool that allows finding crash bugs (see https://github.com/anse1/sqlsmith https://github.com/anse1/sqlsmith).
cheez 6y ago
Finding ONLY 179 bugs is astounding. If I had written this database.... 179 bugs per line is more likely.