ncruces

35 ms·

721.
by ncruces 2y ago
You're right that write reordering might cause this (an invalid frame followed by a few valid frames). That shouldn't happen if you use PRAGMA synchronous=FULL, but in WAL mode it's very common to use NORMAL. Not sure what&#x
722.
by ncruces 2y ago
Yeah, I've double checked. The checksum VFS shim does nothing to fix the (silent) data loss a random bit flip in the WAL causes. If you want long term durability, checkpoint your WAL. Then the checksum VFS will help.
723.
by ncruces 2y ago
There are benefits to having a single writer connection (and multiple readers). One is that you can get fairer and more efficient locking. The (interruptible) file lock APIs that SQLite uses lead to a lot of sleeping and retrying that waste
724.
by ncruces 2y ago
Also, I'm not 100% certain what the interaction is between the checksums of the VFS shim and the WAL checksums during checkpointing w.r.t. silent durability loss. The shim's checksums are explicitly ignored during checkpointing so
725.
by ncruces 2y ago
It was committed successfully, if there's a valid commit frame in the WAL. It simply wasn't checkpointed at this point. If the WAL has valid data frames and a valid commit frame after a corrupted frame, there's a strong
726.
by ncruces 2y ago
My Go driver adds support for this by default, and I provide a simple API to enable it. I'm of the opinion that the VFS shim should ship by default, certainly on the CLI, because it still needs to be enabled for each DB, and all it tak
727.
by ncruces 2y ago
Try splitting Georgia, where Harris wins a few populous counties with a 30 to 70 pp margin, and Trump leads the lump of smaller counties with a 30 to 70 pp margin. They reelected the DA that's prosecuting Trump on one of the populous c
728.
by ncruces 2y ago
I'm the developer of the Wasm driver mentioned in the article: https://github.com/ncruces/go-sqlite3 The v0.20.x series (I just released v0.20.1) should improve this significantly . The driver was reserving lots
729.
by ncruces 2y ago
I meant in Go. This is not a pissing match. I personally like it better than exceptions (even if it's much noisier for the 95% common case as another poster put it), both of which I've used enough to appreciate the pros/c
730.
by ncruces 2y ago
Yeah, sure. But ignoring by mistake gets caught by linters, in practice. And then doing it on purpose is almost as noisy as bubbling it, and sure to raise an eyebrow in code review. My experience is with exceptions in Java/C#, Go error
731.
by ncruces 2y ago
There is no better way. The general idea is that you should pay the cost of handling an error right there where you receive one, even if you're just going to return it. This reduces the incremental cost of actually doing somethin
732.
by ncruces 2y ago
I particularly like the part where it suggests changes to pasted code. When I copy and paste code, very often it needs some small changes (like changing all xs to ys and at the same time widths to heights). It's very good at this, and
733.
by ncruces 2y ago
Particularly Galicia (Santiago), which should really use the Portuguese timezone. Or not: you cross the border and don't “fix” your watch, because mealtimes, etc, are all shifted an hour in the opposite direction.
734.
by ncruces 2y ago
That stops when husband can legitimately claim he does more than wife.
735.
by ncruces 2y ago
Yeah, it might still be possible to implement a mutex based on just the existence of an object, but it'll be harder to add expiration/liveness which I find essential.
736.
by ncruces 2y ago
Google Cloud Storage is good enough to implement locks all by itself: https://reddit.com/r/golang/comments/t52d4f/gmutex_a_global_... Doesn't S3 provide primitives to do the same? At least since mov
737.
by ncruces 2y ago
Yep. The Go build system runs under your current architecture, cross-compiling tests to your target architecture. Then, the Go test runner also runs under your current architecture, orchestrating running your cross compiled test binaries.
738.
by ncruces 2y ago
It gets even more amazing than that. Look at this bit from my GitHub Actions: https://github.com/ncruces/go-sqlite3/blob/fefee692dbfad39f6... I install QEMU (I have the same setup locally), then it's one
739.
by ncruces 2y ago
Litestream is out of process, requires shared memory WAL, doesn't work with exclusive mode WAL, wouldn't work work on Wasm. LiteFS didn't support WAL mode initially, now does. The Wasm version doesn't support WAL mode at
740.
by ncruces 2y ago
I'm chill bro. Still waiting for your productive couple of weekends. But that'll require more than looking at docs reading the words "lock" and "shared" and assuming you know what those mean.
741.
by ncruces 2y ago
Yeah, I wonder why no one thought of that.
742.
by ncruces 2y ago
Seriously? Litestream works out-of-process. It needs shared memory WAL and file locking to work exactly like on "desktop" SQLite for it to even function and produce this "shadow WAL." No ifs, no buts, no inbetweens
743.
by ncruces 2y ago
Then you don't need WAL. I didn't say replication isn't possible, I said "that's not how it works." A bunch of different people solved this already (though none in a couple of weekends); common to all of them:
744.
by ncruces 2y ago
Yep, that doesn't use WAL mode at all, and it's a decent amount of work. It uses a custom VFS, memory journal mode, and bypasses SQLite for optimistic concurrency handling. I dunno how it handles crash safety, or if it even uses t
745.
by ncruces 2y ago
I… really don't get this. The people on SQLite, employed to work on this full time for over a year, have this to say (on the link I posted above): “Because the WASM build does not have shared memory APIs, activating WAL requires that
746.
by ncruces 2y ago
Yes. Each connection lives in its own isolated sandbox, and only communicates with other connections through the “file system” (which is a virtual abstraction, actually). WAL mode is the “exception”: a few pages of the sandbox's memory
747.
by ncruces 2y ago
It's turning out great, I guess. The current approach is not portable to Windows, but it works fine on Linux, macOS, BSD and illumos. In general, portability is hindered more by file locking (I hate POSIX locks) than mmap. The currentl
748.
by ncruces 2y ago
Not really, or it depends. That complexity is dealt with by SQLite. The complexity for you comes from trying to reuse their (battle tested) implementation when (as I wrote above) the primitives they depend upon were not meant to make port
749.
by ncruces 2y ago
File backed SQLite in a browser? Do you mean like OPFS? https://sqlite.org/wasm/doc/trunk/persistence.md#opfs-wal Again, just because the all the Lego pieces sound like they should all just fit together, does
750.
by ncruces 2y ago
I don't know much about DuckDB's architecture. Wasm is fine for compute (though concurrency is still a somewhat open question). To have Wasm talk to the outside world, you need “host calls” where the guest calls the host. On a bro

More ›