6 ms·
The way this is implemented, with the ability for an application server attached to a replica to say "error: this needs to perform a write - hey CDN, replay thi
by simonw 5y ago
The way this is implemented, with the ability for an application server attached to a replica to say "error: this needs to perform a write - hey CDN, replay this request against the region with the database leader in it" is SO clever.
- zozbot234 5y agoThere's probably a way to do this in native postgres by abusing the FDW facility. No real need to involve the CDN or replay the request.
- mrkurt 5y agoWe tried running something like pgpool's split write setup, where the app or a proxy in front of the DB knows where to send transactions: https://www.pgpool.net/docs/latest/en/html/runtime-config-load-balancing.html https://www.pgpool.net/docs/latest/en/html/runtime-config-lo... It didn't work cross region because no one builds an app expecting latency between their app server code and database. What we'd see on write requests was something like: 1. Query for data from read replica, perhaps for validation (0ms) 2. Do a write to the primary in a different region (20-400ms) 3. Query primary for consistency (20-400ms) 4. More queries against primary for consistency (20-400ms) 5. Maybe another write (20-400ms) 6. repeat You can actually use our postgres this way if you want! But it breaks for most apps. It is much, much faster to ship the whole HTTP request where it needs to be than the move the database away from an app instance.
- namibj 5y agoThe problem is that most postgres clients can't handle multiple parametric queries in a single connection (necessary for them to be in a single transaction). This came to light for me as I was trying to build an app on CockroachDB, and the cluster nature of that (at least for geo-redundant setups) implies a non-negligible latency in the low single digits.
- rad_gruchalski 5y agoAnd probably adds quite a lot of latency under load.
- simonw 5y agoThe demo at https://fly-global-rails.fly.dev/ https://fly-global-rails.fly.dev/ lets you see replay times. I'm in SJC (San Jose / Silicon Valley) and https://fly-global-rails.fly.dev/regions/syd https://fly-global-rails.fly.dev/regions/syd shows that my replay time to their region in Sydney Australia is between 25 and 45ms. That's pretty quick!
- rad_gruchalski 5y agoThe try / repeat pattern inevitably needs to do a full round trip multiple times in different regions. What would be awesome to have is something like this but built on top of envoy postgres proxy or similar where it would know where to send the query to based on the table + column value / pk. But one then rebuilds Yugabyte.
- zozbot234 5y ago> postgres proxy or similar where it would know where to send the query to based on the table + column value / pk. That sounds like sharding, which Postgres supports already.
- rad_gruchalski 5y agoIt does, doesn’t it? Every database can be sharded. But you have to do it yourself. This is how it works in Yugabyte: https://docs.yugabyte.com/latest/explore/multi-region-deployments/ https://docs.yugabyte.com/latest/explore/multi-region-deploy....
- mrkurt 5y agoWe need to optimize this. Right now, the edge proxy handles the replay request. So if you hit SJC and a server in LAX handles the request (which is probably what's happening for you), you pay the SJC -> LAX latency penalty before the replay happens. There's no reason we can't replay from LAX, though, just need to build it.
- yashap 5y agoClever, but only useful if your “pre-DB-write” work is cheap. For example, I work at a company where part of what we do is matching riders to drivers. This can be very expensive, and a typical flow is: 1) Read current state of riders/drivers from DB (slightly expensive) 2) Solve a vehicle routing problem with the new rider request added to the current state (can be VERY expensive) 3) If there’s a good new solution, commit the changes that the VRP solutions suggest (this is the DB write, and it’s only slightly expensive) The approach in this blog post would have us duplicating the most expensive thing our app does (step 2 above), for most requests - not good. Much more load on our system, and these already slow writes would take ~twice as long. I’d hope Fly also lets you configure the load balancer - i.e. have a way to send certain requests to the “writer” nodes by default, vs. in a retry.
- simonw 5y agoRather than using their suggested "catch writes and throw an exception" mechanism, I would instead write my own fast application logic to identify if something is likely to be a write. For most of the applications I build the HTTP verb is good enough for this - so I would add a tiny piece of Django middleware which looks for a POST to a non-primary region and sends fly-replay straight away at that point.
- chachra 5y agowould one do it in the custom db router itself instead?
- mrkurt 5y agoThis is a good way to do it. Catching errors lets us reliably ship a library that makes this work for almost everyone, but it's not right for all apps: https://github.com/soupedup/fly-rails/blob/main/lib/fly-rails/rails.rb#L6-L9 https://github.com/soupedup/fly-rails/blob/main/lib/fly-rail...
- kenrose 5y agoCries in GraphQL (where you generally POST, even for queries).
- foobarbazetc 5y agoI think most database engineers who think about ACID/isolation levels/etc would look at this and go "uhm... jumps out a window" but for a subset of users I guess it kind of works? I don't really see how a transaction that performs reads and writes can be replayed like this with any sort of guarantee about anything, though. And there's no opportunity to build any sort of conflict resolution in there because the replay is automatic. Maybe I missed something though?
- simonw 5y agoThe replay is only automatic if you opt for the version where you catch write errors and turn them into replay requests. My experience of building web apps, where most business logic runs within the scope of a single HTTP request, suggests that a quite impressive number of common cases could be served by the replay-request-against-the-leader pattern.
- tptacek 5y agoI mean: just to be clear: you can't write to a read replica. There's no way to introduce a conflict that way. Also just for what it's worth: we agree with 'foobarbazetc. There's a section about that in the post. If you're saying there are important classes of applications this doesn't work well for, that's true.
- foobarbazetc 5y agoJust to be clear, I'm not trying to say it's bad or whatever. It's super cool! I'm a big fan of fly.io. I just don't see how something like this actually works in a way where you can reason about transaction ordering without global serialisation of requests against the writable database, which I presume isn't happening behind the scenes? Basically, this would seem to work great if every request did INSERTs only and you based no logic to run INSERTs on anything you SELECT'ed from the read only replicas. But you could have situations where, e.g., you DELETE or UPDATE something in a request to one region, and it goes to replay that against the writable region, and in "replay gap" another request modifies the same rows or objects, and based on various factors such as latency etc, a DELETE ... WHERE or UPDATE ... WHERE clause might no longer hold. Or you UPDATE the wrong objects, or DELETE the wrong data, etc. I did read the post, but I guess I need to re-read it. I have written a globally distributed database before so I'm always intrigued by how these work. :)
- atom_arranger 5y agoI think it could be done better by marking requests as either queries or mutations like GraphQL does, then you can just check which it is and forward accordingly. Although even with REST you should be able to do the same thing, assume GETS can go to the read replicas, should work if you aren’t doing anything weird.