4 ms·
This is the actual problem: "Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both c
by chasil 6mo ago
This is the actual problem:
"Kamal runs blue-green deploys — it starts a new container, health-checks it, then stops the old one. During the switchover, both containers are running. Both mount ultrathink_storage. Both have the SQLite files open."
WAL mode requires shared access to System V IPC mapped memory. This is unlikely to work across containers.
In case anybody needs a refresher:
https://en.wikipedia.org/wiki/Shared_memory https://en.wikipedia.org/wiki/Shared_memory
https://en.wikipedia.org/wiki/CB_UNIX https://en.wikipedia.org/wiki/CB_UNIX
https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-system-v-interprocess-communication-ipc https://www.ibm.com/docs/en/aix/7.1.0?topic=operations-syste...
- Retr0id 6mo ago> This is unlikely to work across containers. Why not?
- simonw 6mo agoThanks for this, the anecdote with the lost data was very concerning to me. I think you're exactly right about the WAL shared memory not crossing the container boundary. EDIT: It looks like WAL works fine across Docker boundaries, see https://news.ycombinator.com/item?id=47637353#47677163 https://news.ycombinator.com/item?id=47637353#47677163 I don't know much about Kamal but I'd look into ways of "pausing" traffic during a deploy - the trick where a proxy pretends that a request is taking another second to finish when it's actually held in the proxy while the two containers switch over. From https://kamal-deploy.org/docs/upgrading/proxy-changes/ https://kamal-deploy.org/docs/upgrading/proxy-changes/ it looks like Kamal 2's new proxy doesn't have this yet, they list "Pausing requests" as "coming soon".
- chasil 6mo agoYou might consider taking the database(s) out of WAL mode during a migration. That would eliminate the need for shared memory.
- Retr0id 6mo ago> I think you're exactly right about the WAL shared memory not crossing the container boundary. I don't, fwiw (so long as all containers are bind mounting the same underlying fs).
- hedora 6mo agoIt would explain the corruption: https://sqlite.org/wal.html https://sqlite.org/wal.html The containers would need to use a path on a shared FS to setup the SHM handle, and, even then, this sounds like the sort of thing you could probably break via arcane misconfiguration. I agree shm should work in principle though.
- PunchyHamster 6mo agoNot how SQLite works (any more) > The wal-index is implemented using an ordinary file that is mmapped for robustness. Early (pre-release) implementations of WAL mode stored the wal-index in volatile shared-memory, such as files created in /dev/shm on Linux or /tmp on other unix systems. The problem with that approach is that processes with a different root directory (changed via chroot) will see different files and hence use different shared memory areas, leading to database corruption. Other methods for creating nameless shared memory blocks are not portable across the various flavors of unix. And we could not find any method to create nameless shared memory blocks on windows. The only way we have found to guarantee that all processes accessing the same database file use the same shared memory is to create the shared memory by mmapping a file in the same directory as the database itself.
- simonw 6mo agoI just tried an experiment and you're right, WAL mode worked fine across two Docker containers running on the same (macOS) host: https://github.com/simonw/research/tree/main/sqlite-wal-docker-containers#readme https://github.com/simonw/research/tree/main/sqlite-wal-dock... Could the two containers in the OP have been running on separate filesystems, perhaps?
- Retr0id 6mo ago
- hedora 6mo agoPausing requests then running two sqlites momentarily probably won’t prevent corruption. It might make it less likely and harder to catch in testing. The easiest approach is to kill sqlite, then start the new one. I’d use a unix lockfile as a last-resort mechanism (assuming the container environment doesn’t somehow break those).
- simonw 6mo agoI'm saying you pause requests, shut down one of the SQLite containers, start up the other one and un-pause.
- gcr 6mo agoThe SQLite documentation says in strong terms not to do this. https://sqlite.org/howtocorrupt.html#_filesystems_with_broken_or_missing_lock_implementations https://sqlite.org/howtocorrupt.html#_filesystems_with_broke... See more: https://sqlite.org/wal.html#concurrency https://sqlite.org/wal.html#concurrency
- Retr0id 6mo agoThey tell you to use a proper FS, which is largely orthogonal to containerization.
- jmull 6mo agoWAL relies on shared memory, so while a proper FS is necessary, it isn't going to help in this case.
- fauigerzigerk 6mo agoWhy does it not help if both containers can mmap the same -shm file?
- merb 6mo agobtw nfs that is mentioned here is fine in sync mode. However that is slow.
- PunchyHamster 6mo ago> WAL mode requires shared access to System V IPC mapped memory. Incorrect. It requires access to mmap() "The wal-index is implemented using an ordinary file that is mmapped for robustness. Early (pre-release) implementations of WAL mode stored the wal-index in volatile shared-memory, such as files created in /dev/shm on Linux or /tmp on other unix systems. The problem with that approach is that processes with a different root directory (changed via chroot) will see different files and hence use different shared memory areas, leading to database corruption." > This is unlikely to work across containers. I'd imagine sqlite code would fail if that was the case; in case of k8s at least mounting same storage to 2 containers in most configurations causes K8S to co-locate both pods on same node so it should be fine. It is far more likely they just fucked up the code and lost data that way...
- voidfunc 6mo agoOoh new historical Unix variant I had never heard of.. neat!
- chasil 6mo agoAIX is still supported and sold, so quite current? Some that I used that are gone... Ultrix (MIPS), Clix, Irix, SunOS 4, SCO OpenServer, TI System V. https://en.wikipedia.org/wiki/Ultrix https://en.wikipedia.org/wiki/Ultrix https://en.wikipedia.org/wiki/Intergraph https://en.wikipedia.org/wiki/Intergraph
- nxobject 6mo agoNeXTstep? (Leaving aside fun spitballing about whether Tahoe is morally OPENSTEP 26, and whether it was NeXT that actually bought Apple for negative $400 million...)
- chasil 6mo agoAlas, I never had access to any of the Next environments, until PPC MacOS. I did hold a copy in my hands for 486-class machines in the college bookstore.
- ncruces 6mo agoThis thread in the SQLite forum should be instructive: https://sqlite.org/forum/forumpost/90d6805c7cec827f https://sqlite.org/forum/forumpost/90d6805c7cec827f