3 ms·

This seems like it went under the radar a bit. I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. I feel like

This seems like it went under the radar a bit. I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. I feel like you might be getting what you are paying for with those merchant fees. They seem like extortion until you get a chance to take a tour through the fully-staffed command center and get to see just how much pain & suffering that real, live humans go through 24/7/365 to ensure the payment network is stable. During my time working for Pulse (now part of Discover), I was expected to respond to a network outage within 60 seconds. Like, on the phone having a conversation with the bank's IT team. Anything going beyond 5 minutes would be escalated to a multi-party bridge call that doesn't end until everything is green again. The notion that the entire payment network could be down for more than a few hours is unimaginable at the scale of an organization like Visa, Amex or Discover.

tpurves 3y ago
Square is built on relatively modern (eg this century) tech stack. Modern tech stack are inherently more fragile than the real legacy stuff. The Mainframe tech that powers visanet has enormous practical limitations. But holy crap is it stable. Something about the dozens of abstraction layers that hadn't been invented yet when the original ISO payment standards and processes were first invented. The very close to metal and low surface area for breakage. And the benefit of decades of experience in these legacy stacks to shake out bugs. All of Visa's global commerce literally runs through just one of seven (mostly one of three) physical computers. Crazy stuff. But probably not sustainable many more decades. Eventually core networks will have to migrate to modern hardware, OS and SW stacks. That migration could create a lot of benefits, but I would not expect that maintaining the currently observed ~5-7 nines reliability to be one them.
- galenmarchetti 3y ago
  I wonder how much of the stability is the extreme reluctance to do any upgrades or touch anything related to that mainframe code. In my experience with larger institutions using mainframes, its largely "code freeze, do not mess with it". I wonder if this is a consequence of the stability, or a cause of it? Modern tech stacks are always upgrading components underneath to remain modern and frequency of changes cause instability no matter how hard you try. Not really any meaningful tooling to get around that yet
Scoundreller 3y ago
> I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. Last year one of Canada’s biggest ISPs went down for about a day. Knocked out its entire home/biz fixed internet, mobile network, private links, everything. A bad update on Friday morning that seemed to catch them off-guard when it failed, so response was delayed as their processes were poor for a potentially (and ultimately) enterprise-breaking change. Their CTO was on vacation and unaware for a while because they were on a dead company phone and figured it was a roaming issue. Because the towers were up (but islanded), if you had a Rogers phone and tried to call 9-1-1, it would just fail instead of failing over to other networks because the phone saw the tower. Many biz couldn’t do electronic payments, and any cloud POS knocked out so sometimes cash didn’t work either. This also knocked out the Interac/Debit network nationwide because they depended on this ISP, even if you used another provider yourself. https://www.cbc.ca/amp/1.6515869 https://www.cbc.ca/amp/1.6515869 https://www.itworldcanada.com/article/interac-outage-exacerbated-by-poor-network-design-says-expert/492141?amp=1 https://www.itworldcanada.com/article/interac-outage-exacerb... Our regulator let them redact tons of info in the investigation, which is hilarious because the Canadian or US Transportation Safety Board or US Chemical Safety Board would make the investigation totally open and say “fuck you” to your corporate “secrets” (that everyone should avoid, not steal). We don’t treat telecom infra like we do other physical infra. And our regulator is toothless.
- sekh60 3y ago
  Didn't Rogers have two outages last year? I only really remember the one you mentioned, but I seem to recall cell service being down another day as well. My spouse and I use different cell providers (Bell and Rogers) to avoid a single point of failure.
  - Scoundreller 3y ago
    2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... 2021 (but was generally just mobile/voice, not fixed line): https://www.cbc.ca/news/business/rogers-outage-1.5992954 https://www.cbc.ca/news/business/rogers-outage-1.5992954

tpurves 3y ago

Square is built on relatively modern (eg this century) tech stack. Modern tech stack are inherently more fragile than the real legacy stuff. The Mainframe tech that powers visanet has enormous practical limitations. But holy crap is it stable. Something about the dozens of abstraction layers that hadn't been invented yet when the original ISO payment standards and processes were first invented. The very close to metal and low surface area for breakage. And the benefit of decades of experience in these legacy stacks to shake out bugs. All of Visa's global commerce literally runs through just one of seven (mostly one of three) physical computers. Crazy stuff. But probably not sustainable many more decades. Eventually core networks will have to migrate to modern hardware, OS and SW stacks. That migration could create a lot of benefits, but I would not expect that maintaining the currently observed ~5-7 nines reliability to be one them.

galenmarchetti 3y ago
I wonder how much of the stability is the extreme reluctance to do any upgrades or touch anything related to that mainframe code. In my experience with larger institutions using mainframes, its largely "code freeze, do not mess with it". I wonder if this is a consequence of the stability, or a cause of it? Modern tech stacks are always upgrading components underneath to remain modern and frequency of changes cause instability no matter how hard you try. Not really any meaningful tooling to get around that yet

Scoundreller 3y ago

> I cannot comprehend the impact if someone like VISA went down like this for the same amount of time. Last year one of Canada’s biggest ISPs went down for about a day. Knocked out its entire home/biz fixed internet, mobile network, private links, everything. A bad update on Friday morning that seemed to catch them off-guard when it failed, so response was delayed as their processes were poor for a potentially (and ultimately) enterprise-breaking change. Their CTO was on vacation and unaware for a while because they were on a dead company phone and figured it was a roaming issue. Because the towers were up (but islanded), if you had a Rogers phone and tried to call 9-1-1, it would just fail instead of failing over to other networks because the phone saw the tower. Many biz couldn’t do electronic payments, and any cloud POS knocked out so sometimes cash didn’t work either. This also knocked out the Interac/Debit network nationwide because they depended on this ISP, even if you used another provider yourself. https://www.cbc.ca/amp/1.6515869 https://www.cbc.ca/amp/1.6515869 https://www.itworldcanada.com/article/interac-outage-exacerbated-by-poor-network-design-says-expert/492141?amp=1 https://www.itworldcanada.com/article/interac-outage-exacerb... Our regulator let them redact tons of info in the investigation, which is hilarious because the Canadian or US Transportation Safety Board or US Chemical Safety Board would make the investigation totally open and say “fuck you” to your corporate “secrets” (that everyone should avoid, not steal). We don’t treat telecom infra like we do other physical infra. And our regulator is toothless.

sekh60 3y ago
Didn't Rogers have two outages last year? I only really remember the one you mentioned, but I seem to recall cell service being down another day as well. My spouse and I use different cell providers (Bell and Rogers) to avoid a single point of failure.
- Scoundreller 3y ago
  2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_outage https://en.wikipedia.org/wiki/2022_Rogers_Communications_out... 2021 (but was generally just mobile/voice, not fixed line): https://www.cbc.ca/news/business/rogers-outage-1.5992954 https://www.cbc.ca/news/business/rogers-outage-1.5992954