4 ms·

Fly is building everything in hard mode - since they are not layering on top of an existing cloud like pretty much everyone else (heroku, render, railway, ...).

by suryao 4y ago

Fly is building everything in hard mode - since they are not layering on top of an existing cloud like pretty much everyone else (heroku, render, railway, ...). It's either very smart (if they pull it off) because they will have a ginormous cost advantage or they fail. I'm personally of the opinion that the ux on top of aws/gcp/... is worse than a doo-doo in a shoe. However, they are as stable as can be (all complex systems go down once in a while). There are very few mature projects that do not rely on aws/gcp/... managed services anyway. Might as well put in the little bit of effort to set yourself up for the future instead of painful migrations. This obviously doesn't hold for hobby projects. In any case, I have a lot of respect for the engineering that fly does. Kudos.

candiddevmike 4y ago
Are they really building everything in hard mode or do they just have a bad architecture?
- mrkurt 4y ago
  Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".
  - lghh 4y ago
    I really appreciate the honesty here. I’m not a fly customer (no use case for it) but your transparency is admirable. Wish y’all the best.
  - 3np 4y ago
    As a fellow hashistack operator I'd love to hear what the bad decision was.
    - pcthrowaway 4y ago
      Hopefully they'll blog about it at some point. At this point I might pay just for their blog posts, even though I'm not using fly (yet?) because it's certainly a more cost-effective way to learn about big architectural mistakes than the way they're going about it, which involves actually paying the sunk costs.
    - mrkurt 4y ago
      We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/ https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal. The second problem was using some Consul features that forced us to keep it single region. What we actually need is a global view of a single service. Federated Consul doesn't quite give us that. Earlier versions of our infrastructure were using a bunch of Consul watches to update local state, so we couldn't really federate. Some of this I'd do very differently if we rewound. But we were also building an idea with no actual users. Nomad and Consul gave us a nice platform to experiment on. We just outgrew the "prototype" as we learned what people actually wanted from us.
      - HyperSane 4y ago
        One of the single best design decisions that AWS makes is to isolate each region as much as possible so that truly global outages are nearly impossible.
- suryao 4y ago
  They build everything from scratch - on bare metal, including sourcing hardware (though I'd presume they use a data center manager for it). Arch, from their engineering blogs, is pretty sound.
- faizshah 4y ago
  AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: https://m.youtube.com/watch?v=swQbA4zub20 https://m.youtube.com/watch?v=swQbA4zub20 AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.
  - HyperSane 4y ago
    I feel that this is one of the biggest advantages AWS has over Azure. AWS has never had truly global outages the way that Azure has had with Azure AD
    - olivermuty 4y ago
      Azures global outages have all been DNS more or less. And AWS has had a few of those in my 13-14 years I have used them :D Azure reliability sucks more for sure though. Especiallly networking. Edit: us-east-1 going down disrupts most of global AWS pretty severely fwiw.
      - faizshah 4y ago
        That actually brings up another aspect of AWS architecture, even for global services that depend on us-east-1, AWS separates the control plane (configuration of resources) from the data plane (usage of resources). These global services like IAM have regionalized data planes: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html#global-single-region-operations https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso... In a disaster scenario, the data plane operations can continue so customer workloads can still run while the control plane might experience downtime. This is another lesson where in the case of fly isolating the control plane (deployment of services) from the data plane (executing customer code) could have limited the blast radius of this fault instead of using a global cluster manager.

by suryao 4y ago

candiddevmike 4y ago

Are they really building everything in hard mode or do they just have a bad architecture?

mrkurt 4y ago
Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".
- lghh 4y ago
  I really appreciate the honesty here. I’m not a fly customer (no use case for it) but your transparency is admirable. Wish y’all the best.
- 3np 4y ago
  As a fellow hashistack operator I'd love to hear what the bad decision was.
  - pcthrowaway 4y ago
    Hopefully they'll blog about it at some point. At this point I might pay just for their blog posts, even though I'm not using fly (yet?) because it's certainly a more cost-effective way to learn about big architectural mistakes than the way they're going about it, which involves actually paying the sunk costs.
  - mrkurt 4y ago
    We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/ https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal. The second problem was using some Consul features that forced us to keep it single region. What we actually need is a global view of a single service. Federated Consul doesn't quite give us that. Earlier versions of our infrastructure were using a bunch of Consul watches to update local state, so we couldn't really federate. Some of this I'd do very differently if we rewound. But we were also building an idea with no actual users. Nomad and Consul gave us a nice platform to experiment on. We just outgrew the "prototype" as we learned what people actually wanted from us.
    - HyperSane 4y ago
      One of the single best design decisions that AWS makes is to isolate each region as much as possible so that truly global outages are nearly impossible.
suryao 4y ago
They build everything from scratch - on bare metal, including sourcing hardware (though I'd presume they use a data center manager for it). Arch, from their engineering blogs, is pretty sound.
faizshah 4y ago
AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: https://m.youtube.com/watch?v=swQbA4zub20 https://m.youtube.com/watch?v=swQbA4zub20 AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.
- HyperSane 4y ago
  I feel that this is one of the biggest advantages AWS has over Azure. AWS has never had truly global outages the way that Azure has had with Azure AD
  - olivermuty 4y ago
    Azures global outages have all been DNS more or less. And AWS has had a few of those in my 13-14 years I have used them :D Azure reliability sucks more for sure though. Especiallly networking. Edit: us-east-1 going down disrupts most of global AWS pretty severely fwiw.
    - faizshah 4y ago
      That actually brings up another aspect of AWS architecture, even for global services that depend on us-east-1, AWS separates the control plane (configuration of resources) from the data plane (usage of resources). These global services like IAM have regionalized data planes: https://docs.aws.amazon.com/whitepapers/latest/aws-fault-isolation-boundaries/global-services.html#global-single-region-operations https://docs.aws.amazon.com/whitepapers/latest/aws-fault-iso... In a disaster scenario, the data plane operations can continue so customer workloads can still run while the control plane might experience downtime. This is another lesson where in the case of fly isolating the control plane (deployment of services) from the data plane (executing customer code) could have limited the blast radius of this fault instead of using a global cluster manager.