4 ms·

I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the

I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload. With a skeptical lens, unclear.

deleted 2mo ago
[deleted]
gamegoblin 2mo ago
Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351 https://news.ycombinator.com/item?id=49278351
- biorach 2mo ago
  looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesishq:sqlite:3.51.2-instrumented https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream
  - carlsverre 2mo ago
    Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.
    - biorach 2mo ago
      oh! damn! I did not notice that. That does answer my question.
  - mook 2mo ago
    Hmm, shouldn't the diff start at the tag? That is, it seems like it should be https://github.com/sqlite/sqlite/compare/version-3.51.2...antithesishq:sqlite:3.51.2-instrumented https://github.com/sqlite/sqlite/compare/version-3.51.2...an... Which says they made two commits to make instrumentation work. Also, no fixes (which seems reasonable to me; need to figure out what's wrong first) Presumably any bugs found are being sent upstream to fix.
carlsverre 2mo ago
Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing other WAL checkpoint-style systems: by putting them under stress. The workload itself does not know about the bug. My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test. [1]: https://github.com/antithesishq/antithesis-skills https://github.com/antithesishq/antithesis-skills
- minimaltom 2mo ago
  Thanks for clarifying! It would be really interesting if Antithesis finds the bug when: 1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used
  - carlsverre 2mo ago
    This is a solid idea. However, something that is maybe similar to this idea is this: https://antithesis.com/blog/2026/finding-bugs-in-raft-implementations/ https://antithesis.com/blog/2026/finding-bugs-in-raft-implem... Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.
    - minimaltom 2mo ago
      Yeah 100%! And I'm sorry if I sound a little more critical and less eager, its just thats theres a world of difference between a priori finding the bug, and reproducing it, and the impression of the article (from my read) was the former. But please keep writing, I know its super hard to put yourself out there and make content!
      - 2mo ago

deleted 2mo ago

[deleted]

gamegoblin 2mo ago

Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351 https://news.ycombinator.com/item?id=49278351

biorach 2mo ago
looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesishq:sqlite:3.51.2-instrumented https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream
- carlsverre 2mo ago
  Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.
  - biorach 2mo ago
    oh! damn! I did not notice that. That does answer my question.
- mook 2mo ago
  Hmm, shouldn't the diff start at the tag? That is, it seems like it should be https://github.com/sqlite/sqlite/compare/version-3.51.2...antithesishq:sqlite:3.51.2-instrumented https://github.com/sqlite/sqlite/compare/version-3.51.2...an... Which says they made two commits to make instrumentation work. Also, no fixes (which seems reasonable to me; need to figure out what's wrong first) Presumably any bugs found are being sent upstream to fix.

carlsverre 2mo ago

Hey, author here! Good question. The prompt was to use the Antithesis skills[1] to write a workload that exercises SQLite's insertion and checkpoint machinery across multiple processes using a specific version of SQLite (3.51.2). The agent who did the work was aware of the bug and knew I wanted to see if Antithesis could reproduce it. However, I specifically designed the workload the same way we approach testing other WAL checkpoint-style systems: by putting them under stress. The workload itself does not know about the bug. My goal, which I believe I accomplished, was to determine if Antithesis would have found this bug if we had put the SQLite WAL under test. [1]: https://github.com/antithesishq/antithesis-skills https://github.com/antithesishq/antithesis-skills

minimaltom 2mo ago
Thanks for clarifying! It would be really interesting if Antithesis finds the bug when: 1. The specific bug isnt mentioned 2. (If youre game) a model with a knowledge-cutoff date before the report is used
- carlsverre 2mo ago
  This is a solid idea. However, something that is maybe similar to this idea is this: https://antithesis.com/blog/2026/finding-bugs-in-raft-implementations/ https://antithesis.com/blog/2026/finding-bugs-in-raft-implem... Which is: "Given the knowledge of a kind of bug that can happen in one Raft implementation, can a similar workload find similar bugs in other Raft implementations?" Turns out it can.
  - minimaltom 2mo ago
    Yeah 100%! And I'm sorry if I sound a little more critical and less eager, its just thats theres a world of difference between a priori finding the bug, and reproducing it, and the impression of the article (from my read) was the former. But please keep writing, I know its super hard to put yourself out there and make content!
    - 2mo ago