4 ms·

What about local read only replicas of the db in each region, and one primary in your primary region? Or a write through cache in each region.

zknill 3y ago
This design is really similar to the approach taken by litefs[1] and turso[2]. Writes are forwarded to the primary (like you would with a write-through cache) and a local copy of the data is held in each region. These local copies are only guaranteed to be up-to-date with the most recent write forwarded by that local copy. [1] https://fly.io/docs/litefs/proxy/ https://fly.io/docs/litefs/proxy/ [2] https://docs.turso.tech/reference/data-consistency#on-the-replicas https://docs.turso.tech/reference/data-consistency#on-the-re...
moffkalast 3y ago
Or just having a completely separate database for each region, much like the average MMO does it.
cj 3y ago
This is relatively easy to orchestrate with MongoDB replica sets (and I'm sure with other databases, too). In a typical MongoDB deployment you want at least 3 nodes (1 primary, 2 or more secondaries). You can put one of the secondaries on the other side of the world. By default, all reads are sent to the primary. But you can easily specify "nearest" as your "read preference" on a query-by-query basis which would allow the app to read from the nreplica with the lowest latency (whether it be the primary or one of the secondaries). The ability to specify read preference at the query level lets you have 2 categories of queries, 1) queries that must always be served with most recent writes, sent to primary at the expense , and 2) queries that are allowed to occasionally send stale data but benefit from latency improvements by reading from the nearest node.
donmcronald 3y ago
> Or a write through cache in each region. This is something I want to play around with using Cloudflare. I wonder if there’s enough of a guarantee for a user to “stick” to a node that you could use Workers KV as an eventually consistent write through cache for user unique data and a Durable Object as a strongly consistent write through cache. I feel like that would give a pretty solid balance, wouldn’t it?
- tracker1 3y ago
  Cloudflare has Durable Objects/workers for consistent write-through... You can combine them. You can also consider D1, which might be the better option all around for many use cases.
_ben_ 3y ago
PolyScale [1] focuses on many of these issues. It provides a globally distributed database cache at the edge. Writes pass through to the database and reads are cached locally to the app tier. The Smart Invalidation feature inspects updates/deletes/inserts and invalidates just the changed data from the cache, globally. 1. https://www.polyscale.ai/ https://www.polyscale.ai/
DylanSp 3y ago
From what I understand, that should be possible with Fly.io, using the fly-replay header to denote requests that need to be run against the primary database. I'm not sure if that's still their recommended approach, though. EDIT: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/ talks about how this can work. The approach it suggests for implementing this is to try running all database queries locally; if there's an error from the local database being a read-only replica, add the fly-replay header pointing to the primary region and return an HTTP 409, and Fly's proxy will rerun the request in the specified region. There's also some commentary on handling consistency issues to at least get read-your-own-writes.
- tptacek 3y ago
  Yes, it is. It's how we think about multi-region replication in our own tooling. It's not the only way to handle it (Cockroach would be an example of another approach; so would rqlite). We're not evangelists for multi-reader single-writer --- it's just the most common approach big Postgres shops tend to use, so we worked on making it effective. But the other approaches should work fine on Fly.io too. What we especially like about multi-reader single-writer clusters is that you can mostly keep all the pieces in your head at the same time. If you understand Postgres or SQLite, and can get your head around the `fly-replay` header (which is just "replay this request over there please"), you've got the whole system.
  - DylanSp 3y ago
    Thanks for confirming this!
    - tptacek 3y ago
      We do new stuff all the time but `fly-replay` is unlikely to go away, since I think it's Kurt's favorite thing we've ever come up with. :)
  - otoolep 3y ago
    rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, but it is an option. "rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes." [1] https://rqlite.io/ https://rqlite.io/ [2] https://rqlite.io/docs/clustering/read-only-nodes/ https://rqlite.io/docs/clustering/read-only-nodes/

zknill 3y ago

This design is really similar to the approach taken by litefs[1] and turso[2]. Writes are forwarded to the primary (like you would with a write-through cache) and a local copy of the data is held in each region. These local copies are only guaranteed to be up-to-date with the most recent write forwarded by that local copy. [1] https://fly.io/docs/litefs/proxy/ https://fly.io/docs/litefs/proxy/ [2] https://docs.turso.tech/reference/data-consistency#on-the-replicas https://docs.turso.tech/reference/data-consistency#on-the-re...

moffkalast 3y ago

Or just having a completely separate database for each region, much like the average MMO does it.

cj 3y ago

This is relatively easy to orchestrate with MongoDB replica sets (and I'm sure with other databases, too). In a typical MongoDB deployment you want at least 3 nodes (1 primary, 2 or more secondaries). You can put one of the secondaries on the other side of the world. By default, all reads are sent to the primary. But you can easily specify "nearest" as your "read preference" on a query-by-query basis which would allow the app to read from the nreplica with the lowest latency (whether it be the primary or one of the secondaries). The ability to specify read preference at the query level lets you have 2 categories of queries, 1) queries that must always be served with most recent writes, sent to primary at the expense , and 2) queries that are allowed to occasionally send stale data but benefit from latency improvements by reading from the nearest node.

donmcronald 3y ago

> Or a write through cache in each region. This is something I want to play around with using Cloudflare. I wonder if there’s enough of a guarantee for a user to “stick” to a node that you could use Workers KV as an eventually consistent write through cache for user unique data and a Durable Object as a strongly consistent write through cache. I feel like that would give a pretty solid balance, wouldn’t it?

tracker1 3y ago
Cloudflare has Durable Objects/workers for consistent write-through... You can combine them. You can also consider D1, which might be the better option all around for many use cases.

_ben_ 3y ago

PolyScale [1] focuses on many of these issues. It provides a globally distributed database cache at the edge. Writes pass through to the database and reads are cached locally to the app tier. The Smart Invalidation feature inspects updates/deletes/inserts and invalidates just the changed data from the cache, globally. 1. https://www.polyscale.ai/ https://www.polyscale.ai/

DylanSp 3y ago

From what I understand, that should be possible with Fly.io, using the fly-replay header to denote requests that need to be run against the primary database. I'm not sure if that's still their recommended approach, though. EDIT: https://fly.io/blog/globally-distributed-postgres/ https://fly.io/blog/globally-distributed-postgres/ talks about how this can work. The approach it suggests for implementing this is to try running all database queries locally; if there's an error from the local database being a read-only replica, add the fly-replay header pointing to the primary region and return an HTTP 409, and Fly's proxy will rerun the request in the specified region. There's also some commentary on handling consistency issues to at least get read-your-own-writes.

tptacek 3y ago
Yes, it is. It's how we think about multi-region replication in our own tooling. It's not the only way to handle it (Cockroach would be an example of another approach; so would rqlite). We're not evangelists for multi-reader single-writer --- it's just the most common approach big Postgres shops tend to use, so we worked on making it effective. But the other approaches should work fine on Fly.io too. What we especially like about multi-reader single-writer clusters is that you can mostly keep all the pieces in your head at the same time. If you understand Postgres or SQLite, and can get your head around the `fly-replay` header (which is just "replay this request over there please"), you've got the whole system.
- DylanSp 3y ago
  Thanks for confirming this!
  - tptacek 3y ago
    We do new stuff all the time but `fly-replay` is unlikely to go away, since I think it's Kurt's favorite thing we've ever come up with. :)
- otoolep 3y ago
  rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, but it is an option. "rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes." [1] https://rqlite.io/ https://rqlite.io/ [2] https://rqlite.io/docs/clustering/read-only-nodes/ https://rqlite.io/docs/clustering/read-only-nodes/