4 ms·

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

by mrkurt 4y ago

lghh 4y ago
I really appreciate the honesty here. I’m not a fly customer (no use case for it) but your transparency is admirable. Wish y’all the best.
3np 4y ago
As a fellow hashistack operator I'd love to hear what the bad decision was.
- pcthrowaway 4y ago
  Hopefully they'll blog about it at some point. At this point I might pay just for their blog posts, even though I'm not using fly (yet?) because it's certainly a more cost-effective way to learn about big architectural mistakes than the way they're going about it, which involves actually paying the sunk costs.
- mrkurt 4y ago
  We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/ https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal. The second problem was using some Consul features that forced us to keep it single region. What we actually need is a global view of a single service. Federated Consul doesn't quite give us that. Earlier versions of our infrastructure were using a bunch of Consul watches to update local state, so we couldn't really federate. Some of this I'd do very differently if we rewound. But we were also building an idea with no actual users. Nomad and Consul gave us a nice platform to experiment on. We just outgrew the "prototype" as we learned what people actually wanted from us.
  - HyperSane 4y ago
    One of the single best design decisions that AWS makes is to isolate each region as much as possible so that truly global outages are nearly impossible.

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

by mrkurt 4y ago

Yes. Both. This outage was caused by a bad architectural decision. We had an incident a few weeks ago caused by "hard mode".

lghh 4y ago

I really appreciate the honesty here. I’m not a fly customer (no use case for it) but your transparency is admirable. Wish y’all the best.

3np 4y ago

As a fellow hashistack operator I'd love to hear what the bad decision was.

pcthrowaway 4y ago
Hopefully they'll blog about it at some point. At this point I might pay just for their blog posts, even though I'm not using fly (yet?) because it's certainly a more cost-effective way to learn about big architectural mistakes than the way they're going about it, which involves actually paying the sunk costs.
mrkurt 4y ago
We got ourselves into a bind because of Nomad. We outlined a bunch of it here: https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/ https://fly.io/blog/carving-the-scheduler-out-of-our-orchest... The short version is that "using Nomad and Consul for the type of global workloads we run is not a good choice". I do not believe we'd have the same problems with Nomad + Consul in a single region. But running a single, global cluster of each of these is suboptimal. The second problem was using some Consul features that forced us to keep it single region. What we actually need is a global view of a single service. Federated Consul doesn't quite give us that. Earlier versions of our infrastructure were using a bunch of Consul watches to update local state, so we couldn't really federate. Some of this I'd do very differently if we rewound. But we were also building an idea with no actual users. Nomad and Consul gave us a nice platform to experiment on. We just outgrew the "prototype" as we learned what people actually wanted from us.
- HyperSane 4y ago
  One of the single best design decisions that AWS makes is to isolate each region as much as possible so that truly global outages are nearly impossible.