4 ms·

Breaking the WAL

162 points by wwilson 2mo ago

john_strinlai 2mo ago
related: Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug (tailscale.com) https://news.ycombinator.com/item?id=49272832 https://news.ycombinator.com/item?id=49272832
wwilson 2mo ago
We wanted to publish this ASAP since people are talking about this bug today. In a follow-up, we will show how our automated causality analysis could have saved Tailscale and SQLite from 6 months of root cause analysis. Stay tuned!
- deleted 2mo ago
  [deleted]
grebc 2mo ago
[flagged]
- wwilson 2mo ago
  As the post (and linked repo) pretty clearly indicate, zero info about the existence of the bug went into this work. Oh btw, we also found some other ones… Stay tuned!
  - grebc 2mo ago
    So you found this before SQLite published the fix?
- carlsverre 2mo ago
  Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.
  - grebc 2mo ago
    I read your ad. It states you replicated the bug once the SQLite team fixed it, and published it. Not sure what’s difficult about replicating behaviour when it’s spelled out for you.
    - biorach 2mo ago
      this is a deliberate mischaracterisation of the content of the article
      - grebc 2mo ago
        You’re mischaracterising an ad, that’s also funnily enough full of it.
  - returningfory2 2mo ago
    I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built. My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense. Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.
    - carlsverre 2mo ago
      I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach: https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521 https://news.ycombinator.com/item?id=49278521 Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.
      - 2mo ago
carlsverre 2mo ago
Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs. If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading! [1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumented/antithesis/workload.c https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
- MPSimmons 2mo ago
  How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.
  - carlsverre 2mo ago
    Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis. The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time. [1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumented/antithesis/workload.c https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
deleted 2mo ago
[deleted]
minimaltom 2mo ago
I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload. With a skeptical lens, unclear.
- deleted 2mo ago
  [deleted]
- gamegoblin 2mo ago
  Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351 https://news.ycombinator.com/item?id=49278351
  - biorach 2mo ago
    looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesishq:sqlite:3.51.2-instrumented https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream
    - carlsverre 2mo ago
      Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.
      - biorach 2mo ago
        oh! damn! I did not notice that. That does answer my question.
deleted 2mo ago
[deleted]
uhohherewegoaga 2mo ago
as a long time lurker who usually enjoys antithesis posts, this was a pretty existentially sad read. we all know that company blog posts here always tend to be ads at their core, but historically afaicr antithesis generally toes the line well of keeping the "thing we sell is really good" as a side dish to the "i (author) did very interesting deep thinking and/or hard work to accomplish something cool" entree. this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.
- another_twist 2mo ago
  I am okay with companies shilling their tools. Ultimately I'd want my employer to buy good tools that make our lives easier specifically wrt debugging which I genuinely hate. However this post could have been a better walk through about what Claude found, what was the exact root cause and how it can be fixed instead of the generic Anthithesis found it in 15m.
  - uhohherewegoaga 2mo ago
    yeah im fine with shilling that has actual interesting substance beyond the ad. imo this is an ad + AI handwaving and nothing else. this is a pretty wide trend for blog content nowadays so im not suprised overall, i just commented because previous antithesis blogs were better than this to that criteria. just take a look at the other older write ups on their site compared to this one and you’ll see a very clear difference in depth and effort of content.
Ozzie-D 2mo ago
[flagged]
archseer 2mo ago
Related: finding the same bug with TLA+ https://news.ycombinator.com/item?id=48730953 https://news.ycombinator.com/item?id=48730953
LoganDark 2mo ago
I love hearing about this product, I hope one day I'll get to use anything like it.
nemothekid 2mo ago
I understand the author feels he missed an opportunity to catch a bug in Sqlite - but this still feels like catching something in hindsight. Just knowing the bug the is in Sqlite's concurrency engine is a huge win. But reading over tailscale's blogpost, I'm not sure how I would connect the dots to "this issue I'm seeing in production" to "load this up in antithesis". It seemed half the battle was figuring out if the bug was in Sqlite, Tailscale, or Linux, and in that situation if you gave Antithesis all the context, would it reliably find the bug?
- stavros 2mo ago
  Maybe it wouldn't, but being able to cheaply test SQLite within a few minutes is still very valuable, even if just to rule it out.
moomin 2mo ago
I feel like we’re in a weird P=NP age, where after someone finds an issue, someone points out a cheaper way to find the same issue, which inevitably involves knowing the issue is there in the first place and it’s approximate location.
78787 2mo ago
[dead]
xyzsparetimexyz 2mo ago
Omg the one thing I hate more than ai art is ai pixelart where the pixels are all different sizes

162 points by wwilson 2mo ago

related: Tailscale Traces Database Corruption to 16y/o SQLite WAL-Reset Bug (tailscale.com) https://news.ycombinator.com/item?id=49272832 https://news.ycombinator.com/item?id=49272832

wwilson 2mo ago

We wanted to publish this ASAP since people are talking about this bug today. In a follow-up, we will show how our automated causality analysis could have saved Tailscale and SQLite from 6 months of root cause analysis. Stay tuned!

deleted 2mo ago
[deleted]

grebc 2mo ago

[flagged]

wwilson 2mo ago
As the post (and linked repo) pretty clearly indicate, zero info about the existence of the bug went into this work. Oh btw, we also found some other ones… Stay tuned!
- grebc 2mo ago
  So you found this before SQLite published the fix?
carlsverre 2mo ago
Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.
- grebc 2mo ago
  I read your ad. It states you replicated the bug once the SQLite team fixed it, and published it. Not sure what’s difficult about replicating behaviour when it’s spelled out for you.
  - biorach 2mo ago
    this is a deliberate mischaracterisation of the content of the article
    - grebc 2mo ago
      You’re mischaracterising an ad, that’s also funnily enough full of it.
- returningfory2 2mo ago
  I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built. My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense. Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.
  - carlsverre 2mo ago
    I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach: https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521 https://news.ycombinator.com/item?id=49278521 Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.
    - 2mo ago

carlsverre 2mo ago

Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug https://tailscale.com/blog/sqlite-wal-reset-bug I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs. If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading! [1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumented/antithesis/workload.c https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

MPSimmons 2mo ago
How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.
- carlsverre 2mo ago
  Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis. The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!). Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time. [1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumented/antithesis/workload.c https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...

deleted 2mo ago

[deleted]

minimaltom 2mo ago

I went clicking through to see if I could find the prompt they fed the AI to locate the issue / write the test suite. I couldn't find it, so its unclear if the prompt was completely "make a test suite" or was lead towards finding it in the first place, which wouldn't be a fair test. The closest I mention of the prompt I could find was: > Then I asked it to write a simple workload which exercised the WAL insert and checkpoint code. Notably, this is a completely generic workload. With a skeptical lens, unclear.

deleted 2mo ago
[deleted]
gamegoblin 2mo ago
Their founder guy says[1] they found some other bugs while doing this that they will report soon, so if that is true, seems more plausibly like a pretty generic thing. Looking forward to seeing the other bugs they found. 1. https://news.ycombinator.com/item?id=49278351 https://news.ycombinator.com/item?id=49278351
- biorach 2mo ago
  looks like the bug fixes are in individual commits here: https://github.com/sqlite/sqlite/compare/master...antithesishq:sqlite:3.51.2-instrumented https://github.com/sqlite/sqlite/compare/master...antithesis... I'd be interested in seeing what the SQLite people have to say about them if/when they are submitted upstream
  - carlsverre 2mo ago
    Those are the commits from upstream that resolved the bug. Notice that most of them were committed by Dr. Richard Hipp himself.
    - biorach 2mo ago
      oh! damn! I did not notice that. That does answer my question.

deleted 2mo ago

[deleted]

uhohherewegoaga 2mo ago

as a long time lurker who usually enjoys antithesis posts, this was a pretty existentially sad read. we all know that company blog posts here always tend to be ads at their core, but historically afaicr antithesis generally toes the line well of keeping the "thing we sell is really good" as a side dish to the "i (author) did very interesting deep thinking and/or hard work to accomplish something cool" entree. this one falls very flat on that front imo. the "antithesis finds rare bugs" is the overwhelming majority of the actual content here, where as the usual meat of the blog, the hard work, the deep thinking, really distills down to "typed this into claude mobile, typed that into claude mobile". yes, its a statement to the power of the technologies of today that's all it takes to find this bug, but man it's a sad signal for those of us looking for compelling technical reads instead of b2b sales pitches.

another_twist 2mo ago
I am okay with companies shilling their tools. Ultimately I'd want my employer to buy good tools that make our lives easier specifically wrt debugging which I genuinely hate. However this post could have been a better walk through about what Claude found, what was the exact root cause and how it can be fixed instead of the generic Anthithesis found it in 15m.
- uhohherewegoaga 2mo ago
  yeah im fine with shilling that has actual interesting substance beyond the ad. imo this is an ad + AI handwaving and nothing else. this is a pretty wide trend for blog content nowadays so im not suprised overall, i just commented because previous antithesis blogs were better than this to that criteria. just take a look at the other older write ups on their site compared to this one and you’ll see a very clear difference in depth and effort of content.

Ozzie-D 2mo ago

[flagged]

archseer 2mo ago

Related: finding the same bug with TLA+ https://news.ycombinator.com/item?id=48730953 https://news.ycombinator.com/item?id=48730953

LoganDark 2mo ago

I love hearing about this product, I hope one day I'll get to use anything like it.

nemothekid 2mo ago

I understand the author feels he missed an opportunity to catch a bug in Sqlite - but this still feels like catching something in hindsight. Just knowing the bug the is in Sqlite's concurrency engine is a huge win. But reading over tailscale's blogpost, I'm not sure how I would connect the dots to "this issue I'm seeing in production" to "load this up in antithesis". It seemed half the battle was figuring out if the bug was in Sqlite, Tailscale, or Linux, and in that situation if you gave Antithesis all the context, would it reliably find the bug?

stavros 2mo ago
Maybe it wouldn't, but being able to cheaply test SQLite within a few minutes is still very valuable, even if just to rule it out.

moomin 2mo ago

I feel like we’re in a weird P=NP age, where after someone finds an issue, someone points out a cheaper way to find the same issue, which inevitably involves knowing the issue is there in the first place and it’s approximate location.

78787 2mo ago

[dead]

xyzsparetimexyz 2mo ago

Omg the one thing I hate more than ai art is ai pixelart where the pixels are all different sizes