5 ms·

From what I understand, that should be possible with Fly.io, using the fly-replay header to denote requests that need to be run against the primary database. I'

From what I understand, that should be possible with Fly.io, using the fly-replay header to denote requests that need to be run against the primary database. I'm not sure if that's still their recommended approach, though. EDIT: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/ talks about how this can work. The approach it suggests for implementing this is to try running all database queries locally; if there's an error from the local database being a read-only replica, add the fly-replay header pointing to the primary region and return an HTTP 409, and Fly's proxy will rerun the request in the specified region. There's also some commentary on handling consistency issues to at least get read-your-own-writes.

tptacek 3y ago
Yes, it is. It's how we think about multi-region replication in our own tooling. It's not the only way to handle it (Cockroach would be an example of another approach; so would rqlite). We're not evangelists for multi-reader single-writer --- it's just the most common approach big Postgres shops tend to use, so we worked on making it effective. But the other approaches should work fine on Fly.io too. What we especially like about multi-reader single-writer clusters is that you can mostly keep all the pieces in your head at the same time. If you understand Postgres or SQLite, and can get your head around the `fly-replay` header (which is just "replay this request over there please"), you've got the whole system.
- DylanSp 3y ago
  Thanks for confirming this!
  - tptacek 3y ago
    We do new stuff all the time but `fly-replay` is unlikely to go away, since I think it's Kurt's favorite thing we've ever come up with. :)
- otoolep 3y ago
  rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, but it is an option. "rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes." [1] https://rqlite.io/ https://rqlite.io/ [2] https://rqlite.io/docs/clustering/read-only-nodes/ https://rqlite.io/docs/clustering/read-only-nodes/

tptacek 3y ago

Yes, it is. It's how we think about multi-region replication in our own tooling. It's not the only way to handle it (Cockroach would be an example of another approach; so would rqlite). We're not evangelists for multi-reader single-writer --- it's just the most common approach big Postgres shops tend to use, so we worked on making it effective. But the other approaches should work fine on Fly.io too. What we especially like about multi-reader single-writer clusters is that you can mostly keep all the pieces in your head at the same time. If you understand Postgres or SQLite, and can get your head around the `fly-replay` header (which is just "replay this request over there please"), you've got the whole system.

DylanSp 3y ago
Thanks for confirming this!
- tptacek 3y ago
  We do new stuff all the time but `fly-replay` is unlikely to go away, since I think it's Kurt's favorite thing we've ever come up with. :)
otoolep 3y ago
rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, but it is an option. "rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes." [1] https://rqlite.io/ https://rqlite.io/ [2] https://rqlite.io/docs/clustering/read-only-nodes/ https://rqlite.io/docs/clustering/read-only-nodes/