3 ms·

The status updates on this incident were very frustrating: https://www.issquareup.com/incidents/06zffytdqjdz https://www.issquareup.com/incidents/06zffytdqjdz

The status updates on this incident were very frustrating: https://www.issquareup.com/incidents/06zffytdqjdz https://www.issquareup.com/incidents/06zffytdqjdz There is a global outage, stopping hundreds of millions of dollars of credit card transactions from processing, and all the detail they share is: - "We are continuing to work on a fix for a disruption" - "At this time, we do not have a solution for the disruption, though we have all the right people working to get it resolved" - "Very sorry for the inconvenience today." - "Thank you for your patience!"

lopkeny12ko 3y ago
I have never understood this line of thinking. Why do you, as the customer, care about the details? It's not like the issue will be fixed any faster if they posted updates about redeploying a service, waiting for CI to pass on a commit, someone investigating a bad database query, or toggling feature flags.
- a10c 3y ago
  I don’t necessarily care about the technical details as a customer, but I do care about things like “we are prioritising restoring our PoS payment processing first, then online payments second” After 8 hours, telling me about my patience is a slap in the face
- bob1029 3y ago
  > Why do you, as the customer, care about the details? I might be a gigantic customer. I might be so big that I get to have a little bit of influence over certain strategic decisions you make in your product stack. When I worked in the industry, there were big customers that would frequently tour the facilities to get a sense for capabilities and compliance. Part of the tour involves a private datacenter in central US that houses the mainframes and strictly designed around those assets. I was allowed to visit the facility and developed a deep sense of respect for how much capital & energy goes into making this stuff the bedrock. If you are interested in doing 1bn+ USD business through a payment network, you are going to want hands-on assurances that things are going to support your needs. So, even the barest of detailed reporting from Square would go a long way in helping to establish this sense of "bedrock", imo. Let some bigger customers push back on technology decisions that resulted in this outcome and insist upon more resilient choices.
Scoundreller 3y ago
I thought they scrubbed their hourly non-updates, but they’s just buried a level, e.g.: https://ca.issquareup.com/incidents/wyb09mlsvvpq https://ca.issquareup.com/incidents/wyb09mlsvvpq

lopkeny12ko 3y ago

I have never understood this line of thinking. Why do you, as the customer, care about the details? It's not like the issue will be fixed any faster if they posted updates about redeploying a service, waiting for CI to pass on a commit, someone investigating a bad database query, or toggling feature flags.

a10c 3y ago
I don’t necessarily care about the technical details as a customer, but I do care about things like “we are prioritising restoring our PoS payment processing first, then online payments second” After 8 hours, telling me about my patience is a slap in the face
bob1029 3y ago
> Why do you, as the customer, care about the details? I might be a gigantic customer. I might be so big that I get to have a little bit of influence over certain strategic decisions you make in your product stack. When I worked in the industry, there were big customers that would frequently tour the facilities to get a sense for capabilities and compliance. Part of the tour involves a private datacenter in central US that houses the mainframes and strictly designed around those assets. I was allowed to visit the facility and developed a deep sense of respect for how much capital & energy goes into making this stuff the bedrock. If you are interested in doing 1bn+ USD business through a payment network, you are going to want hands-on assurances that things are going to support your needs. So, even the barest of detailed reporting from Square would go a long way in helping to establish this sense of "bedrock", imo. Let some bigger customers push back on technology decisions that resulted in this outcome and insist upon more resilient choices.

Scoundreller 3y ago

I thought they scrubbed their hourly non-updates, but they’s just buried a level, e.g.: https://ca.issquareup.com/incidents/wyb09mlsvvpq https://ca.issquareup.com/incidents/wyb09mlsvvpq