6 ms·

Square says it has resolved day-long outage

37 points by rossdavidh 3y ago

bob1029 3y ago
This seems like it went under the radar a bit. I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. I feel like you might be getting what you are paying for with those merchant fees. They seem like extortion until you get a chance to take a tour through the fully-staffed command center and get to see just how much pain & suffering that real, live humans go through 24/7/365 to ensure the payment network is stable. During my time working for Pulse (now part of Discover), I was expected to respond to a network outage within 60 seconds. Like, on the phone having a conversation with the bank's IT team. Anything going beyond 5 minutes would be escalated to a multi-party bridge call that doesn't end until everything is green again. The notion that the entire payment network could be down for more than a few hours is unimaginable at the scale of an organization like Visa, Amex or Discover.
- tpurves 3y ago
  Square is built on relatively modern (eg this century) tech stack. Modern tech stack are inherently more fragile than the real legacy stuff. The Mainframe tech that powers visanet has enormous practical limitations. But holy crap is it stable. Something about the dozens of abstraction layers that hadn't been invented yet when the original ISO payment standards and processes were first invented. The very close to metal and low surface area for breakage. And the benefit of decades of experience in these legacy stacks to shake out bugs. All of Visa's global commerce literally runs through just one of seven (mostly one of three) physical computers. Crazy stuff. But probably not sustainable many more decades. Eventually core networks will have to migrate to modern hardware, OS and SW stacks. That migration could create a lot of benefits, but I would not expect that maintaining the currently observed ~5-7 nines reliability to be one them.
  - galenmarchetti 3y ago
    I wonder how much of the stability is the extreme reluctance to do any upgrades or touch anything related to that mainframe code. In my experience with larger institutions using mainframes, its largely "code freeze, do not mess with it". I wonder if this is a consequence of the stability, or a cause of it? Modern tech stacks are always upgrading components underneath to remain modern and frequency of changes cause instability no matter how hard you try. Not really any meaningful tooling to get around that yet
- Scoundreller 3y ago
  > I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. Last year one of Canada’s biggest ISPs went down for about a day. Knocked out its entire home/biz fixed internet, mobile network, private links, everything. A bad update on Friday morning that seemed to catch them off-guard when it failed, so response was delayed as their processes were poor for a potentially (and ultimately) enterprise-breaking change. Their CTO was on vacation and unaware for a while because they were on a dead company phone and figured it was a roaming issue. Because the towers were up (but islanded), if you had a Rogers phone and tried to call 9-1-1, it would just fail instead of failing over to other networks because the phone saw the tower. Many biz couldn’t do electronic payments, and any cloud POS knocked out so sometimes cash didn’t work either. This also knocked out the Interac/Debit network nationwide because they depended on this ISP, even if you used another provider yourself. https://www.cbc.ca/amp/1.6515869 https://www.cbc.ca/amp/1.6515869 https://www.itworldcanada.com/article/interac-outage-exacerbated-by-poor-network-design-says-expert/492141?amp=1 https://www.itworldcanada.com/article/interac-outage-exacerb... Our regulator let them redact tons of info in the investigation, which is hilarious because the Canadian or US Transportation Safety Board or US Chemical Safety Board would make the investigation totally open and say “fuck you” to your corporate “secrets” (that everyone should avoid, not steal). We don’t treat telecom infra like we do other physical infra. And our regulator is toothless.
  - sekh60 3y ago
    Didn't Rogers have two outages last year? I only really remember the one you mentioned, but I seem to recall cell service being down another day as well. My spouse and I use different cell providers (Bell and Rogers) to avoid a single point of failure.
    - Scoundreller 3y ago
      2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... 2021 (but was generally just mobile/voice, not fixed line): https://www.cbc.ca/news/business/rogers-outage-1.5992954 https://www.cbc.ca/news/business/rogers-outage-1.5992954
nkotov 3y ago
I was at a business yesterday that was affected by this. Their POS restarted several times and then the rep tried to have me pay through the phone app instead. Eventually the payment started processing an hour later for me.
cypherpunks01 3y ago
The worst part for small businesses is that Square's offline payments mode (where you can accept credit cards without live authorization, at risk of later declines when they post) has been unreliable historically. With some Square outages, customers have been logged out, and then unable to enter offline mode to accept credit cards. Not enough attention has been paid to this critical feature unfortunately, it does not seem to activate properly in many situations. I hope it worked OK for businesses yesterday and cards were able to process today.
- rossdavidh 3y ago
  My understanding is that it did not.
- djmips 3y ago
  Anecdata - a local farmers market weren only taking cash payments yesterday AFAIK - it sounded like a big deal
  - Scoundreller 3y ago
    Good reason to always carry a bit of cash with you. Yeah yeah, you’re losing $5 per $100 a year, but situations like this won’t affect you. If visa or Mastercard go down, I could say ATMs being emptied out.
- altairprime 3y ago
  Latch and Luxer One are two other apps that log you out if any HTTP error code returns - but instead of locking you out of your payments service, they lock you out of your apartment building. Apparently the lazy way to code “access terminated” these days is “if http.ok is false”.
  - ebiester 3y ago
    This is a much larger problem. Fail open and you potentially let someone who has been evicted with a relatively unsophisticated method. Fail closed and you lock someone out. Now, if you are going to say you are solving the problem, it means you have to think deeply here. However, all of the existing solutions have had unpleasant tradeoffs in degraded cases.
cypherpunks01 3y ago
The status updates on this incident were very frustrating: https://www.issquareup.com/incidents/06zffytdqjdz https://www.issquareup.com/incidents/06zffytdqjdz There is a global outage, stopping hundreds of millions of dollars of credit card transactions from processing, and all the detail they share is: - "We are continuing to work on a fix for a disruption" - "At this time, we do not have a solution for the disruption, though we have all the right people working to get it resolved" - "Very sorry for the inconvenience today." - "Thank you for your patience!"
- lopkeny12ko 3y ago
  I have never understood this line of thinking. Why do you, as the customer, care about the details? It's not like the issue will be fixed any faster if they posted updates about redeploying a service, waiting for CI to pass on a commit, someone investigating a bad database query, or toggling feature flags.
  - a10c 3y ago
    I don’t necessarily care about the technical details as a customer, but I do care about things like “we are prioritising restoring our PoS payment processing first, then online payments second” After 8 hours, telling me about my patience is a slap in the face
  - bob1029 3y ago
    > Why do you, as the customer, care about the details? I might be a gigantic customer. I might be so big that I get to have a little bit of influence over certain strategic decisions you make in your product stack. When I worked in the industry, there were big customers that would frequently tour the facilities to get a sense for capabilities and compliance. Part of the tour involves a private datacenter in central US that houses the mainframes and strictly designed around those assets. I was allowed to visit the facility and developed a deep sense of respect for how much capital & energy goes into making this stuff the bedrock. If you are interested in doing 1bn+ USD business through a payment network, you are going to want hands-on assurances that things are going to support your needs. So, even the barest of detailed reporting from Square would go a long way in helping to establish this sense of "bedrock", imo. Let some bigger customers push back on technology decisions that resulted in this outcome and insist upon more resilient choices.
galenmarchetti 3y ago
"One X user responded by saying that their small business missed out on more than $2,000 worth of sales due to the outage, during what is the biggest month of the year for the company" Ouch...that's sad. And they say cause of outage not yet known...unknown, or not yet reported/shared with the public?
- drpgq 3y ago
  A friend that works there said it was DNS related.
  - anonacct37 3y ago
    I can second this.

37 points by rossdavidh 3y ago

This seems like it went under the radar a bit. I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. I feel like you might be getting what you are paying for with those merchant fees. They seem like extortion until you get a chance to take a tour through the fully-staffed command center and get to see just how much pain & suffering that real, live humans go through 24/7/365 to ensure the payment network is stable. During my time working for Pulse (now part of Discover), I was expected to respond to a network outage within 60 seconds. Like, on the phone having a conversation with the bank's IT team. Anything going beyond 5 minutes would be escalated to a multi-party bridge call that doesn't end until everything is green again. The notion that the entire payment network could be down for more than a few hours is unimaginable at the scale of an organization like Visa, Amex or Discover.

tpurves 3y ago
Square is built on relatively modern (eg this century) tech stack. Modern tech stack are inherently more fragile than the real legacy stuff. The Mainframe tech that powers visanet has enormous practical limitations. But holy crap is it stable. Something about the dozens of abstraction layers that hadn't been invented yet when the original ISO payment standards and processes were first invented. The very close to metal and low surface area for breakage. And the benefit of decades of experience in these legacy stacks to shake out bugs. All of Visa's global commerce literally runs through just one of seven (mostly one of three) physical computers. Crazy stuff. But probably not sustainable many more decades. Eventually core networks will have to migrate to modern hardware, OS and SW stacks. That migration could create a lot of benefits, but I would not expect that maintaining the currently observed ~5-7 nines reliability to be one them.
- galenmarchetti 3y ago
  I wonder how much of the stability is the extreme reluctance to do any upgrades or touch anything related to that mainframe code. In my experience with larger institutions using mainframes, its largely "code freeze, do not mess with it". I wonder if this is a consequence of the stability, or a cause of it? Modern tech stacks are always upgrading components underneath to remain modern and frequency of changes cause instability no matter how hard you try. Not really any meaningful tooling to get around that yet
Scoundreller 3y ago
> I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. Last year one of Canada’s biggest ISPs went down for about a day. Knocked out its entire home/biz fixed internet, mobile network, private links, everything. A bad update on Friday morning that seemed to catch them off-guard when it failed, so response was delayed as their processes were poor for a potentially (and ultimately) enterprise-breaking change. Their CTO was on vacation and unaware for a while because they were on a dead company phone and figured it was a roaming issue. Because the towers were up (but islanded), if you had a Rogers phone and tried to call 9-1-1, it would just fail instead of failing over to other networks because the phone saw the tower. Many biz couldn’t do electronic payments, and any cloud POS knocked out so sometimes cash didn’t work either. This also knocked out the Interac/Debit network nationwide because they depended on this ISP, even if you used another provider yourself. https://www.cbc.ca/amp/1.6515869 https://www.cbc.ca/amp/1.6515869 https://www.itworldcanada.com/article/interac-outage-exacerbated-by-poor-network-design-says-expert/492141?amp=1 https://www.itworldcanada.com/article/interac-outage-exacerb... Our regulator let them redact tons of info in the investigation, which is hilarious because the Canadian or US Transportation Safety Board or US Chemical Safety Board would make the investigation totally open and say “fuck you” to your corporate “secrets” (that everyone should avoid, not steal). We don’t treat telecom infra like we do other physical infra. And our regulator is toothless.
- sekh60 3y ago
  Didn't Rogers have two outages last year? I only really remember the one you mentioned, but I seem to recall cell service being down another day as well. My spouse and I use different cell providers (Bell and Rogers) to avoid a single point of failure.
  - Scoundreller 3y ago
    2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... 2021 (but was generally just mobile/voice, not fixed line): https://www.cbc.ca/news/business/rogers-outage-1.5992954 https://www.cbc.ca/news/business/rogers-outage-1.5992954

nkotov 3y ago

I was at a business yesterday that was affected by this. Their POS restarted several times and then the rep tried to have me pay through the phone app instead. Eventually the payment started processing an hour later for me.

cypherpunks01 3y ago

The worst part for small businesses is that Square's offline payments mode (where you can accept credit cards without live authorization, at risk of later declines when they post) has been unreliable historically. With some Square outages, customers have been logged out, and then unable to enter offline mode to accept credit cards. Not enough attention has been paid to this critical feature unfortunately, it does not seem to activate properly in many situations. I hope it worked OK for businesses yesterday and cards were able to process today.

rossdavidh 3y ago
My understanding is that it did not.
djmips 3y ago
Anecdata - a local farmers market weren only taking cash payments yesterday AFAIK - it sounded like a big deal
- Scoundreller 3y ago
  Good reason to always carry a bit of cash with you. Yeah yeah, you’re losing $5 per $100 a year, but situations like this won’t affect you. If visa or Mastercard go down, I could say ATMs being emptied out.
altairprime 3y ago
Latch and Luxer One are two other apps that log you out if any HTTP error code returns - but instead of locking you out of your payments service, they lock you out of your apartment building. Apparently the lazy way to code “access terminated” these days is “if http.ok is false”.
- ebiester 3y ago
  This is a much larger problem. Fail open and you potentially let someone who has been evicted with a relatively unsophisticated method. Fail closed and you lock someone out. Now, if you are going to say you are solving the problem, it means you have to think deeply here. However, all of the existing solutions have had unpleasant tradeoffs in degraded cases.

cypherpunks01 3y ago

The status updates on this incident were very frustrating: https://www.issquareup.com/incidents/06zffytdqjdz https://www.issquareup.com/incidents/06zffytdqjdz There is a global outage, stopping hundreds of millions of dollars of credit card transactions from processing, and all the detail they share is: - "We are continuing to work on a fix for a disruption" - "At this time, we do not have a solution for the disruption, though we have all the right people working to get it resolved" - "Very sorry for the inconvenience today." - "Thank you for your patience!"

lopkeny12ko 3y ago
I have never understood this line of thinking. Why do you, as the customer, care about the details? It's not like the issue will be fixed any faster if they posted updates about redeploying a service, waiting for CI to pass on a commit, someone investigating a bad database query, or toggling feature flags.
- a10c 3y ago
  I don’t necessarily care about the technical details as a customer, but I do care about things like “we are prioritising restoring our PoS payment processing first, then online payments second” After 8 hours, telling me about my patience is a slap in the face
- bob1029 3y ago
  > Why do you, as the customer, care about the details? I might be a gigantic customer. I might be so big that I get to have a little bit of influence over certain strategic decisions you make in your product stack. When I worked in the industry, there were big customers that would frequently tour the facilities to get a sense for capabilities and compliance. Part of the tour involves a private datacenter in central US that houses the mainframes and strictly designed around those assets. I was allowed to visit the facility and developed a deep sense of respect for how much capital & energy goes into making this stuff the bedrock. If you are interested in doing 1bn+ USD business through a payment network, you are going to want hands-on assurances that things are going to support your needs. So, even the barest of detailed reporting from Square would go a long way in helping to establish this sense of "bedrock", imo. Let some bigger customers push back on technology decisions that resulted in this outcome and insist upon more resilient choices.

galenmarchetti 3y ago

"One X user responded by saying that their small business missed out on more than $2,000 worth of sales due to the outage, during what is the biggest month of the year for the company" Ouch...that's sad. And they say cause of outage not yet known...unknown, or not yet reported/shared with the public?

drpgq 3y ago
A friend that works there said it was DNS related.
- anonacct37 3y ago
  I can second this.