6 ms·

in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" - you can formal m

in case people don't read all the way to the end, the important takeaway is "you simply can't afford to do instant global state distribution" - you can formal method and Rust and test and watchdog yourself as much as you want, but you simply have to stop doing that or the unknown unknowns will just keep taking you down.

tptacek 1y ago
I mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait.
- __turbobrew__ 1y ago
  > Proxies in Asia need to know about that, right away, and this time you can't afford to wait. Did you ever consider envoy xDS? There are a lot of really cool things in envoy like outlier detection, circuit breakers, load shedding, etc…
  - tptacek 1y ago
    Nope. Talk a little about how how Envoy's service discovery would scale to millions of apps in a global network? There's no way we found the only possible point in the solution space. Do they do something clever here? What we (think we) know won't work is a topologically centralized database that uses distributed consensus algorithms to synchronize. Running consensus transcontinentally is very painful, and keep the servers central, so that update proposals are local and the protocol can run quickly, subjects large portions of the network to partition risk. The natural response (what I think a lot of people do, in fact) is just to run multiple consensus clusters, but our UX includes a global namespace for customer workloads.
    - hedgehog 1y ago
      Is it actually necessary to run transcontinental consensus? Apps in a given location are not movable so it would seem for a given app it's known which part of the network writes can come from. That would require partitioning the namespace but, given that apps are not movable, does that matter? It feel like there are other areas like docs and tooling that would benefit from relatively higher prioritization.
      - tptacek 1y ago
        Apps in a given location are extremely movable! That's the point of the service!
        hedgehog 1y ago
        We unfortunately lost our location with not a whole lot of notice and the migration to a new one was not seamless, on top of things like the GitHub actions being out of date (only supporting the deprecated Postgres service, not the new one).
    - __turbobrew__ 1y ago
      I haven’t personally worked on envoy xds, but it is what I have seen several BigCo’s use for routing from the edge to internal applications. > Running consensus transcontinentally is very painful You don’t necessarily have to do that, you can keep your quorum nodes (lets assume we are talking about etcd) far enough apart to be in separate failure domains (fires, power loss, natural disasters) but close enough that network latency isn’t unbearably high between the replicas. I have seen the following scheme work for millions of workloads: 1. Etcd quorum across 3 close, but independent regions 2. On startup, the app registers itself under a prefix that all other app replicas register 3. All clients to that app issue etcd watches for that prefix and almost instantly will be notified when there is a change. This is baked as a plugin within grpc clients. 4. A custom grpc resolver is used to do lookups by service name
      - tptacek 1y ago
- vlovich123 1y ago
  > Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait. But they have to. Physically no solution will be instantaneous because that’s not how the speed of light nor relativity works - even two events next to each other cannot find out about each other instantaneously. So then the question is “how long can I wait for this information”. And that’s the part that I feel isn’t answered - eg if the app dies, the TCP connections die and in theory that information travels as quickly as anything else you send. It’s not reliably detectable but conceivably you could have an eBPF program monitoring death and notifying the proxies. Thats the part that’s really not explained in the article which is why you need to maintain an eventually consistent view of the connectivity. I get maybe why that could be useful but noticing app connectivity death seems wrong considering I believe you’re more tracking machine and cluster health right? Ie not noticing an app instance goes down but noticing all app instances on a given machine are gone and consensus deciding globally where the new app instance will be as quickly as possible?
  - tptacek 1y ago
    A request routed to a dead instance doesn't fall into a black hole: our proxies reroute it. But that's very slow; to deliver acceptable service quality you need to minimize the number of times that happens. So you can't accept a solution that leaves large windows of time within which every instance that has gone down has a stale entry. Remember: instances coming up and down happens all the time on this platform! It's part of the point.
- bananapub 1y ago
  apologies for misinterpreting it. that said, I would be very interested if in a couple of years you write a followup post about whether you have found that global ~instantaneous state is workable under the right circumstances or not.

tptacek 1y ago

I mean, the thing we're saying is that instant global state with database-style consensus is unworkable. Instant state distribution though is kind of just... necessary? for a platform like ours. You bring up an app in Europe, proxies in Asia need to know about it to route to it. So you say, "ok, well, they can wait a minute to learn about the app, not the end of the world". Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait.

__turbobrew__ 1y ago
> Proxies in Asia need to know about that, right away, and this time you can't afford to wait. Did you ever consider envoy xDS? There are a lot of really cool things in envoy like outlier detection, circuit breakers, load shedding, etc…
- tptacek 1y ago
  Nope. Talk a little about how how Envoy's service discovery would scale to millions of apps in a global network? There's no way we found the only possible point in the solution space. Do they do something clever here? What we (think we) know won't work is a topologically centralized database that uses distributed consensus algorithms to synchronize. Running consensus transcontinentally is very painful, and keep the servers central, so that update proposals are local and the protocol can run quickly, subjects large portions of the network to partition risk. The natural response (what I think a lot of people do, in fact) is just to run multiple consensus clusters, but our UX includes a global namespace for customer workloads.
  - hedgehog 1y ago
    Is it actually necessary to run transcontinental consensus? Apps in a given location are not movable so it would seem for a given app it's known which part of the network writes can come from. That would require partitioning the namespace but, given that apps are not movable, does that matter? It feel like there are other areas like docs and tooling that would benefit from relatively higher prioritization.
    - tptacek 1y ago
      Apps in a given location are extremely movable! That's the point of the service!
      - hedgehog 1y ago
        We unfortunately lost our location with not a whole lot of notice and the migration to a new one was not seamless, on top of things like the GitHub actions being out of date (only supporting the deprecated Postgres service, not the new one).
  - __turbobrew__ 1y ago
    I haven’t personally worked on envoy xds, but it is what I have seen several BigCo’s use for routing from the edge to internal applications. > Running consensus transcontinentally is very painful You don’t necessarily have to do that, you can keep your quorum nodes (lets assume we are talking about etcd) far enough apart to be in separate failure domains (fires, power loss, natural disasters) but close enough that network latency isn’t unbearably high between the replicas. I have seen the following scheme work for millions of workloads: 1. Etcd quorum across 3 close, but independent regions 2. On startup, the app registers itself under a prefix that all other app replicas register 3. All clients to that app issue etcd watches for that prefix and almost instantly will be notified when there is a change. This is baked as a plugin within grpc clients. 4. A custom grpc resolver is used to do lookups by service name
    - tptacek 1y ago
vlovich123 1y ago
> Now: that same European instance goes down. Proxies in Asia need to know about that, right away, and this time you can't afford to wait. But they have to. Physically no solution will be instantaneous because that’s not how the speed of light nor relativity works - even two events next to each other cannot find out about each other instantaneously. So then the question is “how long can I wait for this information”. And that’s the part that I feel isn’t answered - eg if the app dies, the TCP connections die and in theory that information travels as quickly as anything else you send. It’s not reliably detectable but conceivably you could have an eBPF program monitoring death and notifying the proxies. Thats the part that’s really not explained in the article which is why you need to maintain an eventually consistent view of the connectivity. I get maybe why that could be useful but noticing app connectivity death seems wrong considering I believe you’re more tracking machine and cluster health right? Ie not noticing an app instance goes down but noticing all app instances on a given machine are gone and consensus deciding globally where the new app instance will be as quickly as possible?
- tptacek 1y ago
  A request routed to a dead instance doesn't fall into a black hole: our proxies reroute it. But that's very slow; to deliver acceptable service quality you need to minimize the number of times that happens. So you can't accept a solution that leaves large windows of time within which every instance that has gone down has a stale entry. Remember: instances coming up and down happens all the time on this platform! It's part of the point.
bananapub 1y ago
apologies for misinterpreting it. that said, I would be very interested if in a couple of years you write a followup post about whether you have found that global ~instantaneous state is workable under the right circumstances or not.