4 ms·

How can a software company with $17 billion in revenue be down for more than 8 hours? It seems hard to understand how failover systems could not have been succe

How can a software company with $17 billion in revenue be down for more than 8 hours? It seems hard to understand how failover systems could not have been successfully spun up within a couple hours of a total system meltdown, maybe someone else has insight into this. Square seems like it'd be an order of magnitude less complex to keep running and swap to a failover, say compared with Youtube, Amazon.com, or other billion-dollar grossing software products (I could be wrong).

Swizec 3y ago
Facebook had that 12+ hour outage last year. Meltdowns like this often lead to cascading failures followed by queues so full of stuff you immediately die when trying to come back. There’s bound to be lots of very stressed engineers at Square right now
nine_k 3y ago
No, of course Square is harder to fail-over than YouTube. There's no such thing as a lost monetary transaction on YouTube (a lost ad click at most). From financial liability point of view, and also from PR point of view, it may be better to completely shut down and not accept payments than to take money from a buyer and fail to transfer it to the seller, or bill somebody twice, etc. Eventual consistency is OK, but final consistency should be guaranteed. Because of that, it's harder to run stuff like chaos monkeys, DiRT-style exercises, etc, in an institution that handles relatively large and well-tracked transactions. Hence it's harder to stay prepared to various disaster scenarios.
- cypherpunks01 3y ago
  That makes sense, thanks for the perspective!
- lopkeny12ko 3y ago
  Your answer to "why does the technology stack of a critical multi-billion dollar business not have a failover ability" is "because it's hard"?
  - nine_k 3y ago
    Yes. Because it's really hard. The expense to build the (extra) failover-ability for a case of rare catastrophic events is much larger than the fallout from one such event. It can only pay for itself over a long time, and would be built (maybe has Bern being built already) during a long time.
blamestross 3y ago
Management that refuses to fund or mandate any common infrastructure. Jack likes running his companies like "an economy of startups" but in practice this means nobody is allowed to do things without fast returns. Teams are encouraged to use whatever programming language and infrastructure they think fits them. It is the fast track of tragedy of the commons problems.
- tdeck 3y ago
  I worked at Square from 2014 to 2016 and this wasn't my experience. In fact, Square invested a lot in infrastructure that was pretty impressive compared to other companies of a similar size. We're talking 800-1200 people and they had a single common custom deployment system that had been built before Docker was a thing, and later replaced that with a new in house system (p2) based on containers while I was there. They had a common custom CI system that all services used, and that actually worked quite well (much nicer than Jenkins which was common at the time). And Square really only had 3 languages that were acceptable to use for services (Java, Ruby, and Golang; node.js being phased out). I remember interviewing people at other larger companies and finding they didn't have things we had at Square in 2014, like a common way of configuring and registering services, or an easy way to see what's running on a particular box, or service-to-service ACLs, or a tool to coordinate online schema changes. Square even built their own S2S RPC protocol based on protobufs, and went on to participate in the design of gRPC. Maybe it's changed since then, I don't know. But the culture was pretty solid when I was there. And the plateng team ran multiple DC failover tests of our service while I worked there.
  - dzader 3y ago
    [dead]
  - shams93 3y ago
    Those are all garbage collected languages so I am betting they are reaching the size where they're going to need to transition to stronger iron like rust or c++.
    - 3y ago
arberavdullahu 3y ago
Also the Incident history(https://www.issquareup.com/history https://www.issquareup.com/history) is wild in my opinion. During August there were 31 incidents, some of them lasted for hours

Swizec 3y ago

Facebook had that 12+ hour outage last year. Meltdowns like this often lead to cascading failures followed by queues so full of stuff you immediately die when trying to come back. There’s bound to be lots of very stressed engineers at Square right now

nine_k 3y ago

No, of course Square is harder to fail-over than YouTube. There's no such thing as a lost monetary transaction on YouTube (a lost ad click at most). From financial liability point of view, and also from PR point of view, it may be better to completely shut down and not accept payments than to take money from a buyer and fail to transfer it to the seller, or bill somebody twice, etc. Eventual consistency is OK, but final consistency should be guaranteed. Because of that, it's harder to run stuff like chaos monkeys, DiRT-style exercises, etc, in an institution that handles relatively large and well-tracked transactions. Hence it's harder to stay prepared to various disaster scenarios.

cypherpunks01 3y ago
That makes sense, thanks for the perspective!
lopkeny12ko 3y ago
Your answer to "why does the technology stack of a critical multi-billion dollar business not have a failover ability" is "because it's hard"?
- nine_k 3y ago
  Yes. Because it's really hard. The expense to build the (extra) failover-ability for a case of rare catastrophic events is much larger than the fallout from one such event. It can only pay for itself over a long time, and would be built (maybe has Bern being built already) during a long time.

blamestross 3y ago

Management that refuses to fund or mandate any common infrastructure. Jack likes running his companies like "an economy of startups" but in practice this means nobody is allowed to do things without fast returns. Teams are encouraged to use whatever programming language and infrastructure they think fits them. It is the fast track of tragedy of the commons problems.

tdeck 3y ago
I worked at Square from 2014 to 2016 and this wasn't my experience. In fact, Square invested a lot in infrastructure that was pretty impressive compared to other companies of a similar size. We're talking 800-1200 people and they had a single common custom deployment system that had been built before Docker was a thing, and later replaced that with a new in house system (p2) based on containers while I was there. They had a common custom CI system that all services used, and that actually worked quite well (much nicer than Jenkins which was common at the time). And Square really only had 3 languages that were acceptable to use for services (Java, Ruby, and Golang; node.js being phased out). I remember interviewing people at other larger companies and finding they didn't have things we had at Square in 2014, like a common way of configuring and registering services, or an easy way to see what's running on a particular box, or service-to-service ACLs, or a tool to coordinate online schema changes. Square even built their own S2S RPC protocol based on protobufs, and went on to participate in the design of gRPC. Maybe it's changed since then, I don't know. But the culture was pretty solid when I was there. And the plateng team ran multiple DC failover tests of our service while I worked there.
- dzader 3y ago
  [dead]
- shams93 3y ago
  Those are all garbage collected languages so I am betting they are reaching the size where they're going to need to transition to stronger iron like rust or c++.
  - 3y ago

arberavdullahu 3y ago

Also the Incident history(https://www.issquareup.com/history https://www.issquareup.com/history) is wild in my opinion. During August there were 31 incidents, some of them lasted for hours