5 ms·

Even though SQLite bugs are rare, those can be found occasionally. I remember a bug finder took the sqlite documentation off their website. Collected all their

Even though SQLite bugs are rare, those can be found occasionally. I remember a bug finder took the sqlite documentation off their website. Collected all their keywords, made up millions of jumbled up queries of random combination between keywords and then ran those overnight to find 10 bugs where the engine crashed. And yes those were also reported and fixed quickly.

markerz 4y ago
Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! I did something like this in college when I was implementing a compiler and it caught a lot of really weird bugs in a lot of peoples codes. The beautiful part of some fuzzers though is they can automatically simplify their complicated test cases to something you can actually look at and reason about.
- carlmr 4y ago
  >Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! QuickCheck style testing is maybe also worth a mention here. Instead of using any possible inputs, like in fuzzing, you restrict yourself to legal inputs, like the keywords here, to get maybe less random crashes, but more likely to find useful corner cases because of the restriction on the search space.
return_to_monke 4y ago
so basically, fuzzing. https://en.m.wikipedia.org/wiki/Fuzzing https://en.m.wikipedia.org/wiki/Fuzzing
- rsaxvc 4y ago
  Sounds more like smart Monkey testing, much older, less feedback. https://en.m.wikipedia.org/wiki/Monkey_testing https://en.m.wikipedia.org/wiki/Monkey_testing Edit: looks like some consider these the same nowadays.
  - robocat 4y ago
    I wonder if ChatGPT would be good at generating various fuzz testing queries. Maybe too slow? Maybe too repetitive? Maybe too unvarying? Sorry for mentioning ChatGPT (to those who are sick of the infection).
    - mattygee 4y ago
      Actually, with one of the problems being faced in that space at present, especially for applications involving factual answers, being "hallucinations" ( ie. essentially as I understand it the level of "creativity" in responses) such "creativity" may well be quite suited to finding those various unusual edge cases.
      - TeMPOraL 4y ago
        You know this common trope in fiction and jokes, that advanced computers & AIs can be easily defeated by feeding them a logical inconsistency? Guess what, we now have a chatbot AI that not only doesn't mind working with nonsense input, it will happily produce logically-inconsistent statements on its own, and can do it so sneakily and convincingly, that it's the human operator who could end up believing logically inconsistent statements, without even realizing it, and possibly end up in serious trouble some time later.
YetAnotherNick 4y ago
SQLite is continuosly fuzz tested by at least 2-3 professional teams[0], I am really amazed you could find 10 bugs in one day. Would love to read your write up in this! [0]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- deleted 4y ago
  [deleted]

markerz 4y ago

Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! I did something like this in college when I was implementing a compiler and it caught a lot of really weird bugs in a lot of peoples codes. The beautiful part of some fuzzers though is they can automatically simplify their complicated test cases to something you can actually look at and reason about.

carlmr 4y ago
>Sounds a lot like fuzzing, which is a really good way to find really esoteric bugs against a spec! QuickCheck style testing is maybe also worth a mention here. Instead of using any possible inputs, like in fuzzing, you restrict yourself to legal inputs, like the keywords here, to get maybe less random crashes, but more likely to find useful corner cases because of the restriction on the search space.

return_to_monke 4y ago

so basically, fuzzing. https://en.m.wikipedia.org/wiki/Fuzzing https://en.m.wikipedia.org/wiki/Fuzzing

rsaxvc 4y ago
Sounds more like smart Monkey testing, much older, less feedback. https://en.m.wikipedia.org/wiki/Monkey_testing https://en.m.wikipedia.org/wiki/Monkey_testing Edit: looks like some consider these the same nowadays.
- robocat 4y ago
  I wonder if ChatGPT would be good at generating various fuzz testing queries. Maybe too slow? Maybe too repetitive? Maybe too unvarying? Sorry for mentioning ChatGPT (to those who are sick of the infection).
  - mattygee 4y ago
    Actually, with one of the problems being faced in that space at present, especially for applications involving factual answers, being "hallucinations" ( ie. essentially as I understand it the level of "creativity" in responses) such "creativity" may well be quite suited to finding those various unusual edge cases.
    - TeMPOraL 4y ago
      You know this common trope in fiction and jokes, that advanced computers & AIs can be easily defeated by feeding them a logical inconsistency? Guess what, we now have a chatbot AI that not only doesn't mind working with nonsense input, it will happily produce logically-inconsistent statements on its own, and can do it so sneakily and convincingly, that it's the human operator who could end up believing logically inconsistent statements, without even realizing it, and possibly end up in serious trouble some time later.

YetAnotherNick 4y ago

SQLite is continuosly fuzz tested by at least 2-3 professional teams[0], I am really amazed you could find 10 bugs in one day. Would love to read your write up in this! [0]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

deleted 4y ago
[deleted]