7 ms·
> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read re
by infogulch 4y ago
> To improve latency, we're aiming at a scale-out model that works similarly to Fly Postgres. That's to say: writes get forwarded to the primary and all read requests get served from their local copies.
How can you ensure that a client that just performed a forwarded write will be able to read that back on their local replica on subsequent reads?
- sk55 4y agoThey don't have ACID guarantees with this setup.
- benbjohnson 4y agoACID is typically used to describe single database instances and from that perspective, LiteFS doesn't change the ACID semantics that SQLite has. For distributed consistency models[1], it gets more complicated as LiteFS currently just supports async replication. We'll introduce stronger guarantees in the future although we currently provide some information such as transaction IDs so you can manage replication lag if you need stronger consistency. [1]: https://jepsen.io/consistency https://jepsen.io/consistency
- hinkley 4y agoThat's a fun one. A couple years ago someone posted a solution to that here. I'm not sure if it works for SQLite, but it worked for Postgres. The basics of it were that each replica was aware of the latest transaction ID it had seen. On a normal read you'd deal with the usual set of eventually consistent issues. But on a read-after-write, you would select a replica that was ahead of the write transaction. Ultimately that's a very odd flavor of sharding. What I don't recall is how they propagated that shard information efficiently, since with sharding and consistent hashing the whole idea is that the lookup algorithm is deterministic, and therefore can be run anywhere and get the same result. WAL lag information is dynamic, so... Raft?
- simonw 4y agoAnother solution that I've used successfully in the past for web apps is to set a 10s cookie every time a user performs a write, and then route their read requests to the lead server until that cookie expires. That way they're sure to see the impact of the write they just made.
- hinkley 4y agoIf write traffic is low enough I could see that working, but 10s is an awful long time. Anything less than a 1000:1 read:write ratio would start running into capacity problems wouldn't it?
- simonw 4y agoThat's never been a problem on any system I've worked on. If you assume a web application with logged in users who are occasionally updating their data, the percentage of users who will have performed a write within the last 10s is always going to be absolutely tiny.
- hinkley 4y agoThe absolute rate is the problem. You can't shed that load to other machines so the percentage of users or percentage of traffic doesn't matter. This is basically a variant of Little's Law, where server count has a ceiling of 1. 100 edits a minute from distinct sessions is 1000 sessions pinned at any moment. If they read anything in that interval it comes from the primary. The only question is what's the frequency and interval of reads after a write.
- infogulch 4y agoYeah a transaction ID solution would work pretty well. Every request would return the highest transaction ID seen by the replica; writes forwarded to the primary would also return the transaction ID after that write; any request with a transaction ID higher than the replica would be forwarded to the primary.