6 ms·

> Guess what? fly.io offers a turnkey distributed/replicated Postgres for just this reason. You use an HTTP header to route writes to the region hosting your pr

by nickjj 4y ago

> Guess what? fly.io offers a turnkey distributed/replicated Postgres for just this reason. You use an HTTP header to route writes to the region hosting your primary. Doesn't this take away a lot of the benefits of global distribution? For example if you pay Fly hundreds of dollars a month to distribute your small app in a few datacenters around the globe but your primary DB is in California then everyone from the EU is going to have about 150-200ms round trip latency every time you write to your DB because you can't get around the limitations of the speed of light. Now we're back to non-distributed latency times every time you want to write to the DB which is quite often in a lot of types of apps. If you want to cache mostly static read-only pages at the CDN level you can do this with a number of services. Fly has about 20 datacenters, hosting a small'ish web app that's distributed across them will be over $200 / month without counting extra storage or bandwidth just for the web app portion. Their pg pricing isn't clear but a fairly small cluster is $33.40 / month for 2GB of memory and 40GB of storage. Based on their pricing page it sounds like that's the cost for 1 datacenter, so if you wanted read-replicas in a bunch of other places it adds up. Before you know it you might be at $500 / month to host something that will have similar latency on DB writes as a $20 / month DigitalOcean server that you self manage, Fly also charges you $2 / month per Let's Encrypt wildcard cert where as that's free from Let's Encrypt directly.

deleted 4y ago
[deleted]
anamexis 4y ago
Yes, there are hundreds of different ways you could accomplish this. Fly.io is a convenient and easy to use one.
manmal 4y ago
You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Things like telemetry can be written in edge instances. Depends on your application of course, but in many cases that should be only a tiny fraction of all requests needing redirects to primary. And why would you get 20 instances, all around the world right out of the gate? 6-7 probably do the job quite well, but maybe you don’t even need that many. Depending on where most of your customers are, you could get good results with 3-4 for most users.
- nickjj 4y ago
  > You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Thanks, can you give an example of how that works? Did you write your own fork of Postgres or are you using a third party solution like BDR? Also do you have a few use cases where you'd want writes being dependent on another write? > 6-7 probably do the job quite well You could, let's call it 5. For a 2gb set up would that be about $50 for the web app, $50 for the background workers, $160ish for postgres and then $50 for Redis? We're still at $300+? I was thinking maybe 5 background workers wasn't necessary but frameworks like Rails will put a bunch of things through a background worker where you would want low latency even if they're happening in the background because it's not only things like sending an email where it doesn't matter if it's delayed for 2 seconds behind the scenes. It's performing various Hotwire Turbo actions which render templates and modify records where you'd want to see those things reflected in the web UI as soon as possible.
  - nicoburns 4y ago
    For low-latency workers like that it might make sense to just run them on the same instance as the web servers.
    - nickjj 4y ago
      Does Fly let you run multiple commands in separate Docker images? That's usually the pattern on how to run a web app + worker with Docker, as opposed to creating an init system in Docker and running (2) processes in 1 container (this goes against best practices). The Fly docs only mention the approach of using an init system inside of your image and also tries to talk you into running a separate VM[0] to keep your web app + worker isolated. In either case I think the price still doubles because both your web app and worker need memory for a bunch of common set ups like Rails + Sidekiq, Flask / Django + Celery, etc.. [0]: https://fly.io/docs/app-guides/multiple-processes/ https://fly.io/docs/app-guides/multiple-processes/
      - tptacek 4y ago
throwaway892238 4y ago
And in today's world all data needs to be federated between national borders anyway. Try doing business in China if the user data isn't stored in China. Or Russia. Or the EU. Modern designs need the data layer to be forked between regions, not replicated, with merges between the forks. On top of that, most replication systems are brittle and create logistical and administrative headaches. If you can get by with just rsync, do.
burner556 4y ago
Why are people tripping over $2/mo ? I don’t get this tight-ass mentality. It’s a rounding error.
- thorncorona 4y ago
  why spend money when you don’t have to?
- mathgladiator 4y ago
  One reason is that many people use that as a baseline for how they I'd multi-tenacy. It could be they just proxy resources from their customer down to the infrastructure.
- nickjj 4y ago
  It's not the $2/mo at face value. For me it's the idea of them pushing to make paying for SSL certificates the norm again after Let's Encrypt has put in a huge amount of effort to change that field. Not that there's anything wrong with charging for things but charging for a free service rubs me in a weird way, especially since they're using Let's Encrypt. Using the rounding error logic, how do you feel about companies adding $1.99 "convenience fees" or "administrative fees"?

by nickjj 4y ago

deleted 4y ago

[deleted]

anamexis 4y ago

Yes, there are hundreds of different ways you could accomplish this. Fly.io is a convenient and easy to use one.

manmal 4y ago

You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Things like telemetry can be written in edge instances. Depends on your application of course, but in many cases that should be only a tiny fraction of all requests needing redirects to primary. And why would you get 20 instances, all around the world right out of the gate? 6-7 probably do the job quite well, but maybe you don’t even need that many. Depending on where most of your customers are, you could get good results with 3-4 for most users.

nickjj 4y ago
> You don’t need to route every write to primary though, but only those writes that have dependencies on other writes. Thanks, can you give an example of how that works? Did you write your own fork of Postgres or are you using a third party solution like BDR? Also do you have a few use cases where you'd want writes being dependent on another write? > 6-7 probably do the job quite well You could, let's call it 5. For a 2gb set up would that be about $50 for the web app, $50 for the background workers, $160ish for postgres and then $50 for Redis? We're still at $300+? I was thinking maybe 5 background workers wasn't necessary but frameworks like Rails will put a bunch of things through a background worker where you would want low latency even if they're happening in the background because it's not only things like sending an email where it doesn't matter if it's delayed for 2 seconds behind the scenes. It's performing various Hotwire Turbo actions which render templates and modify records where you'd want to see those things reflected in the web UI as soon as possible.
- nicoburns 4y ago
  For low-latency workers like that it might make sense to just run them on the same instance as the web servers.
  - nickjj 4y ago
    Does Fly let you run multiple commands in separate Docker images? That's usually the pattern on how to run a web app + worker with Docker, as opposed to creating an init system in Docker and running (2) processes in 1 container (this goes against best practices). The Fly docs only mention the approach of using an init system inside of your image and also tries to talk you into running a separate VM[0] to keep your web app + worker isolated. In either case I think the price still doubles because both your web app and worker need memory for a bunch of common set ups like Rails + Sidekiq, Flask / Django + Celery, etc.. [0]: https://fly.io/docs/app-guides/multiple-processes/ https://fly.io/docs/app-guides/multiple-processes/
    - tptacek 4y ago

throwaway892238 4y ago

And in today's world all data needs to be federated between national borders anyway. Try doing business in China if the user data isn't stored in China. Or Russia. Or the EU. Modern designs need the data layer to be forked between regions, not replicated, with merges between the forks. On top of that, most replication systems are brittle and create logistical and administrative headaches. If you can get by with just rsync, do.

burner556 4y ago

Why are people tripping over $2/mo ? I don’t get this tight-ass mentality. It’s a rounding error.

thorncorona 4y ago
why spend money when you don’t have to?
mathgladiator 4y ago
One reason is that many people use that as a baseline for how they I'd multi-tenacy. It could be they just proxy resources from their customer down to the infrastructure.
nickjj 4y ago
It's not the $2/mo at face value. For me it's the idea of them pushing to make paying for SSL certificates the norm again after Let's Encrypt has put in a huge amount of effort to change that field. Not that there's anything wrong with charging for things but charging for a free service rubs me in a weird way, especially since they're using Let's Encrypt. Using the rounding error logic, how do you feel about companies adding $1.99 "convenience fees" or "administrative fees"?