3 ms·

Yeah, distributed systems at the global scale are very very difficult - at least with the Heroku style problem, you'd be looking at scaling in a single datacent

Yeah, distributed systems at the global scale are very very difficult - at least with the Heroku style problem, you'd be looking at scaling in a single datacenter I think - deployments to multiple datacenters wouldn't share dependencies. I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives. Simplicity is the key to reliability, but this isn't a simple product, so idk.

lmm 4y ago
> I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives. In my experience people who ran Postgres distributed across a WAN tended to use obscure third-party plugins at best, more often a pile of dodgy Perl scripts. Using something designed from the ground up to be clustered seems to have a much better chance of working out than trying to make something that's been built as a single-instance system for decades work across the internet.
- mattbillenstein 4y ago
  Yeah, point taken, I wasn't thinking to cluster across the WAN - more like an api wrapping postgres in a single DC. But you pay the price of read latency I guess... it's a hard problem no doubt.
zamnos 4y ago
Backend simplicity also means a more shallow moat. It makes it easier for Digital Ocean/Linode (Akamai)/Hetzner to offer a competing service with the same backend knobs to turn, should they decide they want to get into that market. The goal should be to make the backend as simple as possible, but no simplier. Complexity here leads to operational burden and toil. But that's why you hire good SREs and treat them well. What's more important is frontend complexity, aka how difficult it is for customers to use. Backend and frontend complexity aren't necessarily linked, which, imo, fly.io achieves, downtime aside.

lmm 4y ago

> I do wonder however if they'd be better off using less l33t tech - do almost everything on Postgres vs consul and vault, etc. Scaling, failover, consistency, etc is a more well-known problem and there are a lot of people who've ran other DBs at tremendous scale than the alternatives. In my experience people who ran Postgres distributed across a WAN tended to use obscure third-party plugins at best, more often a pile of dodgy Perl scripts. Using something designed from the ground up to be clustered seems to have a much better chance of working out than trying to make something that's been built as a single-instance system for decades work across the internet.

mattbillenstein 4y ago
Yeah, point taken, I wasn't thinking to cluster across the WAN - more like an api wrapping postgres in a single DC. But you pay the price of read latency I guess... it's a hard problem no doubt.

zamnos 4y ago

Backend simplicity also means a more shallow moat. It makes it easier for Digital Ocean/Linode (Akamai)/Hetzner to offer a competing service with the same backend knobs to turn, should they decide they want to get into that market. The goal should be to make the backend as simple as possible, but no simplier. Complexity here leads to operational burden and toil. But that's why you hire good SREs and treat them well. What's more important is frontend complexity, aka how difficult it is for customers to use. Backend and frontend complexity aren't necessarily linked, which, imo, fly.io achieves, downtime aside.