7 ms·

I think the most shocking thing about this outage was that the country's national payment system, Interac, was down most of the day. This is an interbank netwo

I think the most shocking thing about this outage was that the country's national payment system, Interac, was down most of the day. This is an interbank network which is essentially a cooperative operated by the country's major financial institutions. Who would have thought Interac would've only had one ISP? Thankfully, they've stated they're adding at least one additional service provider. I drove from Winnipeg to Calgary that day (a 12 hour drive) and had to stop in several small towns with TDs to pull cash to fill up. It was quite bizarre checking Twitter all day to see if it was back yet and seeing it was still down.

notatoad 4y ago
the story i heard, which i'm going to irresponsibly repeat without knowing the truth of, is that Interac did have a backup ISP but it was one of the smaller ISPs who buys access to rogers fibre, so their backup went down because rogers was down.
throwawaysleep 4y ago
Canadians are very risk averse people except when it comes to preparedness. They avoid change if at all possible.
- FpUser 4y ago
  Not only that. They often are pathologically incapable of giving straight answers (I am talking about business dealings). It was / still is a major pain in the ass for me as I am very straight guy and have no desire to decipher what their evasive answers really mean. There are some rare exceptions though. Dealing with XXX Motors Canada almost shocked me in comparison because when their guy said they would do something it was like set in stone. They went out of their way to meet my needs. Dealing with Americans was a pleasure for me as well. They say A and it means A and it will be done. Disclaimer - I am Canadian but was born in USSR, came here some 30 years ago.
  - VancouverMan 4y ago
    Who/what do you consider to be a "Canadian" in 2022? In Toronto and the surrounding cities, for example, about half of the population are foreign-born. A significant proportion of the population of many other Canadian urban areas are foreign-born. About 20% of the overall Canadian population are foreign-born. Many of these people are from cultures that are quite different to anything resembling "traditional" Canadian culture. Acquiring Canadian citizenship later in life doesn't necessarily change a person's values, attitudes, and so on. A lot of Canadian-born individuals have one or both parents who are foreign-born, which also can have an impact on one's values, attitudes, and behaviour. Even among those with multi-generational ties to Canada, there were already significant cultural/values/attitude/behavioural differences among the various groupings. Ultimately, in any given interaction with somebody in Canada today, there's a good chance you're dealing with somebody whose ties to "traditional" Canadian culture are limited, or even non-existent. Maybe some kind of a relatively cohesive "Canadian" culture or identity existed at one point, several decades ago, but I don't think that's the case any longer, especially in the urban areas.
    - FpUser 4y ago
      >"Who/what do you consider to be a "Canadian" in 2022" I consider myself one even though I am foreign born. But maybe since I do not pour maple syrup on my eggs'n bacon I am a fake. >"About 20% of the overall Canadian population are foreign-born." For reasons unknown most of company owners/reps I've done business with were WASPs. Count me "lucky". Also what you said does not explain why is it so different in the US.
  - throwawaysleep 4y ago
    I am frequently that person giving evasive answers. Why? Avoid blame and I do not speak up when my manager leaves me in a bad position. My priority in many meetings is merely to not get nailed down on something that I can be whacked with later. Better to avoid all accountability, as Canada doesn't really reward doing a good job.
    - pschuegr 4y ago
14 4y ago
This definitely showed me that a cashless society is a bad idea. Thankfully I had some cash on hand otherwise would have been stuck for food and gas. It also caused a lot of issues at work as all my coworkers need to drive to do their job and some had no cash on hand. Of course that same day the chip on my visa stopped working and a new one was ordered in the mail. But if not for the cash on hand it would have been a bad day.
hinkley 4y ago
After a very frustrating experience with fraud prevention shutting off my CC in Paris, I called and very pointedly made sure they weren't going to fuck me again on my next trip. Day 2 in London, biggest Visa network outage in recent memory happened. What was most surprising was how long it took some store clerks to flip their point of sale systems into queuing mode so that people could keep buying things. There was a period where we just went from shop to shop trying to buy the same thing until we found one that didn't decline the card.
aledalgrande 4y ago
A much bigger entity, but I know the European Bank has 2 datacenters (Amsterdam and Rome), connected by 2 separate backbones going through completely separate geographic areas and they switch from one datacenter to another every 6 months to test out their DR. In fact, they have been doing that for decades.
- vlovich123 4y ago
  Why switch 100% instead of continuously running both with transparent failover in case of failure? It seems like you risk a major outage every 6 months with the current scheme unless I’m missing something.
  - jbverschoor 4y ago
    Because of the same reasons we do continuous integration.
  - aledalgrande 4y ago
    It was told me years ago, so I'm not sure of the details. What they are doing is probably switching the traffic to the main DC, not switching off completely one.
hluska 4y ago
I live in the east end of Regina and had an interesting experience with someone driving east and looking for a Royal Bank. The only Royal I could think of is downtown, both our phones were on the Rogers network and I didn’t even know where to find a phone book. It was amazing to think that one provider could go out and that would render me completely useless in my own city. I’ve outsourced memory to my phone for so long that I only know places I frequent. He ended up using a TD. I failed prairie hospitality.

notatoad 4y ago

the story i heard, which i'm going to irresponsibly repeat without knowing the truth of, is that Interac did have a backup ISP but it was one of the smaller ISPs who buys access to rogers fibre, so their backup went down because rogers was down.

throwawaysleep 4y ago

Canadians are very risk averse people except when it comes to preparedness. They avoid change if at all possible.

FpUser 4y ago
Not only that. They often are pathologically incapable of giving straight answers (I am talking about business dealings). It was / still is a major pain in the ass for me as I am very straight guy and have no desire to decipher what their evasive answers really mean. There are some rare exceptions though. Dealing with XXX Motors Canada almost shocked me in comparison because when their guy said they would do something it was like set in stone. They went out of their way to meet my needs. Dealing with Americans was a pleasure for me as well. They say A and it means A and it will be done. Disclaimer - I am Canadian but was born in USSR, came here some 30 years ago.
- VancouverMan 4y ago
  Who/what do you consider to be a "Canadian" in 2022? In Toronto and the surrounding cities, for example, about half of the population are foreign-born. A significant proportion of the population of many other Canadian urban areas are foreign-born. About 20% of the overall Canadian population are foreign-born. Many of these people are from cultures that are quite different to anything resembling "traditional" Canadian culture. Acquiring Canadian citizenship later in life doesn't necessarily change a person's values, attitudes, and so on. A lot of Canadian-born individuals have one or both parents who are foreign-born, which also can have an impact on one's values, attitudes, and behaviour. Even among those with multi-generational ties to Canada, there were already significant cultural/values/attitude/behavioural differences among the various groupings. Ultimately, in any given interaction with somebody in Canada today, there's a good chance you're dealing with somebody whose ties to "traditional" Canadian culture are limited, or even non-existent. Maybe some kind of a relatively cohesive "Canadian" culture or identity existed at one point, several decades ago, but I don't think that's the case any longer, especially in the urban areas.
  - FpUser 4y ago
    >"Who/what do you consider to be a "Canadian" in 2022" I consider myself one even though I am foreign born. But maybe since I do not pour maple syrup on my eggs'n bacon I am a fake. >"About 20% of the overall Canadian population are foreign-born." For reasons unknown most of company owners/reps I've done business with were WASPs. Count me "lucky". Also what you said does not explain why is it so different in the US.
- throwawaysleep 4y ago
  I am frequently that person giving evasive answers. Why? Avoid blame and I do not speak up when my manager leaves me in a bad position. My priority in many meetings is merely to not get nailed down on something that I can be whacked with later. Better to avoid all accountability, as Canada doesn't really reward doing a good job.
  - pschuegr 4y ago

14 4y ago

This definitely showed me that a cashless society is a bad idea. Thankfully I had some cash on hand otherwise would have been stuck for food and gas. It also caused a lot of issues at work as all my coworkers need to drive to do their job and some had no cash on hand. Of course that same day the chip on my visa stopped working and a new one was ordered in the mail. But if not for the cash on hand it would have been a bad day.

hinkley 4y ago

After a very frustrating experience with fraud prevention shutting off my CC in Paris, I called and very pointedly made sure they weren't going to fuck me again on my next trip. Day 2 in London, biggest Visa network outage in recent memory happened. What was most surprising was how long it took some store clerks to flip their point of sale systems into queuing mode so that people could keep buying things. There was a period where we just went from shop to shop trying to buy the same thing until we found one that didn't decline the card.

aledalgrande 4y ago

A much bigger entity, but I know the European Bank has 2 datacenters (Amsterdam and Rome), connected by 2 separate backbones going through completely separate geographic areas and they switch from one datacenter to another every 6 months to test out their DR. In fact, they have been doing that for decades.

vlovich123 4y ago
Why switch 100% instead of continuously running both with transparent failover in case of failure? It seems like you risk a major outage every 6 months with the current scheme unless I’m missing something.
- jbverschoor 4y ago
  Because of the same reasons we do continuous integration.
- aledalgrande 4y ago
  It was told me years ago, so I'm not sure of the details. What they are doing is probably switching the traffic to the main DC, not switching off completely one.

hluska 4y ago

I live in the east end of Regina and had an interesting experience with someone driving east and looking for a Royal Bank. The only Royal I could think of is downtown, both our phones were on the Rogers network and I didn’t even know where to find a phone book. It was amazing to think that one provider could go out and that would render me completely useless in my own city. I’ve outsourced memory to my phone for so long that I only know places I frequent. He ended up using a TD. I failed prairie hospitality.