10 ms·

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't affor

by mabbo 8y ago

Incredible. They had all their mission-critical infrastructure in a single data center. How many billions of dollars do they make per year? And they can't afford even a tiny bit of redundancy? If I were a customer, I'd use this as a sign that this company is not technically competent enough to manage my money.

ModFightsCron 8y ago
Incredible. You made up a really stupid and (for a bank) illegal thing, then said it's stupid! If I were a company, I'd use this as a sign you are not competent enough to receive our money. No dipshit. They have active->sync->async or star replication, live every other bank. Guess what - when you actually fails over it's not always perfectly smooth. The OP made up that "all of wells fargo" went offline, when in reality tiny bits and pieces die. Have you ever worked or did you wander off the high school tech subreddit? When you actually fail over, sometimes not everything fails over smoothly and you have to do database media recovery and replay redo logs.
barkingcat 8y ago
I don't think you need this as a sign it's not technically competent. There are other much larger markers.
mywittyname 8y ago
Lots of old companies are like this. They don't invest in critical infrastructure because it would bring them below their 20% margins.
- nostrademons 8y ago
  Wells is weird though. When I started banking with them in 2009, they were ahead of the curve in online infrastructure - it was much easier to access them online than any of the banks I was used to back in Massachusetts. They also had a reputation for being both honest and conservative - they were the only AAA-rated bank in 2007, they were #1 in green rankings, and Warren Buffett had invested in them because of their sound balance sheet. And then starting around 2010 but rapidly accelerating around 2014, everything about them went to shit. The best explanation I can think of is that John Stumpf is a slash & burn sociopath, juicing the numbers so he can get his 473x-the-median-worker paycheck while ruining the company. He wouldn't be alone in the financial world, but it's a shame that a 150+ year old institution can so rapidly go down the toilet.
  - Zelphyr 8y ago
    In 2009/2010 my wife accidentally made a charge to PayPal that was linked with our Wells Fargo checking account. Since we weren’t planning on that charge happening from that account, there was an NSF fee and Wells rejected the charger. Understandable, right? Except then PayPal made a few more attempts for god knows why and each time Wells Fargo kicked an NSF fee our way. Now, PayPal shouldn’t have repeatedly attempted a rejected charge. But, Wells Fargo shouldn’t have allowed those attempts. They just couldn’t help themselves to that $35 NSF fee though. We fought it to no avail. With all the NSF fees and interest (and fees they added to fees while we fought it), what started as a $300 transaction ultimately cost us over $1200. Wells Fargo is now and was in 2009/2010 a criminal enterprise.
  - Rapzid 8y ago
    Wells was one of the earlier companies to have the CTO report directly to the CEO. All the crazy sales numbers and bogus account shenanigans were going on back in 2003/2004 when I worked there. I ratted out more than one professional banker to branch managers and up over that crap. A fun one was the home equity lines people would open without customer knowledge and link up to overdraft protection. The customer would never owe, nor know, anything until one day an overdraft hit their equity line, and then they got notified of late payments.. I don't miss working for a Bank.
  - mulmen 8y ago
    I have banked with WF since about 2007 for my student and auto loans. Their online portal has always been worse than whatever my local credit union had. The WF business is clearly set up to confuse and exploit consumers. My credit union websites have always helped me do what I want and need with my money. This includes the tiny local credit union in Idaho.
xdlachx 8y ago
true, but you also give too much credit to the average person. Im employed by WF and even Im a little bewildered by the fact there doesn't seem to be any redundancy implemented somewhere.
Zelphyr 8y ago
Wells Fargo is an awful company. Hopefully this outage will be the final straw for many people.
benj111 8y ago
What kind of redundancy are we talking about here? You can't really roll back say 10 minutes of transactions, so are you maintaining 2 parallel systems? How do you keep them perfectly in sync? This isn't my area of expertise by a long shot, but it occurs to me this is probably hard, especially when your codebase started in the 60s, and has been accreting ever since.
- gniv 8y ago
  Yes it’s hard but there are mature solutions. See Google Spanner for example.
  - ex3ndr 8y ago
    Spanner is slow as hell
- esonoi 8y ago
  You have a primary and a backup with a synchronous commit protocol. When a commit request is made on the primary, the primary writes to its transaction log and the backup’s log. If the backup does not acknowledge, the commit fails. The backup doesn’t need to be in the same exact state as the primary (it’s not meant to service requests), it just needs to have a persistent log of what changes were applied so that it can roll forward when needed. Most relational DBs do something like this for their DR product offering. Oracle has Active Data Guard. DB2 has HADR.
dakial1 8y ago
Maybe this had something to do with it... https://searchdatacenter.techtarget.com/news/4500243727/Wells-Fargo-continues-massive-data-center-consolidation https://searchdatacenter.techtarget.com/news/4500243727/Well...
- aboutruby 8y ago
  Badly formatted paste of the content of the article: https://8cf91ddf-e6b2-465a-9f10-1f631650f4ed.htmlpasta.com https://8cf91ddf-e6b2-465a-9f10-1f631650f4ed.htmlpasta.com (Even if you can just put anything in the email form)
deleted 8y ago
[deleted]
danpalmer 8y ago
The comment says that they have failovers in place, the just didn’t work correctly.
- perl4ever 8y ago
  It's not lost, I know exactly where it is...at the bottom of the ocean!
otakucode 8y ago
Everyone assumes that the banking industry and the financial industry in general somehow have a magic touch when it comes to technology. All those digital balance sheets, they're pristine. All those networks, they're unpenetrated. All those databases, they're filled with the purest divine truth. And if anyone ever figures out that isn't the way it is, and that the numbers are not representative of anything of substance? If nations refuse to honor the claimed 'transfers' done through these rickety electronic systems? It would make for an interesting few days.
surge 8y ago
They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.
- vonmoltke 8y ago
  Bloomberg does it once a year. That said, I have yet to encounter a company more obsessed with business continuity. I don't doubt that their failover systems and testing of them are well beyond the typical.
  - surge 8y ago
    I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought or merged with have challenges you won't see many places that have benefited from the 30 years of lessons that were taught at these companies.
    - kamikaz1k 8y ago
      Well that's not true. Chaos engineering and AWS weren't real things when they started building the company. And the system they have now doesn't resemble much of it was once. Truth of the matter is they invested more in their infrastructure, but that's because their business plan required them to grow on the back of technological advances. Banks, it's seems, do not. Or maybe they do, and the some of these start up banks will usurp them.
      - ianai 8y ago
        Standard good practice should be to have a redundancy in place and test it at a regular interval. It should be part of periodic maintenance - fail to the backup so updates/grades can be applied to production and fail back to production once done. But I’m guessing wellsfargo just doesn’t have a reason to care.
gdulli 8y ago
> If I were a customer Are you sure you're not one?
dekhn 8y ago
you jumped immediately to an unfounded speculation which is known to be wrong (WF maintains multiple data centers). Also, failover is hard. Few companies outside of a few larger Internetz companies can really do it well.
paul7986 8y ago
What all their scandals didn’t convince you of that? After all their logo is that of a stage coach aka the wild Wild West(robbers, thieves, etc). They do not hide who they really are.
- jstarfish 8y ago
  The stagecoach refers to their origins as an armored car/delivery service.
  - paul7986 8y ago
    uh that may and well be true, but their behavior in recent years says stagecoach.. wild, wild west.. robbers and thus their logo is now a punchline/comedy routine.
WrtCdEvrydy 8y ago
If you're waiting for this to be the straw to move away from Wells Fargo after the last set of scandals, you may have some issues...
- CydeWeys 8y ago
  Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.
  - edoceo 8y ago
    Refi that bad-boy to your local credit union
    - abawany 8y ago
      I moved loans to First Tech Federal Credit Union, which used to (and still probably do) state in the loan contract that they will service the loan for its life. Very happy with them so far.
      - funkymike 8y ago
        That's good language to look for. I didn't check closely enough and my mortgage documents stated that the loan issuer (my bank) would service the loan, but without any guarantee of how long. They sold the loan shortly after issuing the mortgage.
    - hinkley 8y ago
      It doesn't work that way. Mortgages get resold between banks all the time. You can walk into your local bank and get a house loan, and five months from now you'll get a business envelope from Wells Fargo informing you that you should send your checks to them now. I would expect if WF decided your first loan met their purchasing criteria, your refinance would get the same treatment.
    - CydeWeys 8y ago
samstave 8y ago
List of WF scandals since 2015 https://finance.yahoo.com/news/every-wells-fargo-consumer-scandal-since-2015-timeline-194946222.html https://finance.yahoo.com/news/every-wells-fargo-consumer-sc...
elliekelly 8y ago
Every major financial regulator has business continuity and disaster recovery requirements but the standards are woefully outdated. A plan that gets the bank back to full functionality within 24 hours would be acceptable to most regulators and even considered speedy to some. Tangentially related, I highly recommend the movie "Out of the Clear Blue Sky." Cantor Fitzgerald was a bond trading firm at the top of one of the twin towers and lost every employee who was in the office on 9/11. Incredibly, despite losing the majority of their employees and despite losing almost all of their trading infrastructure, they managed to resume operations in time for the bond market's reopening 48 hours later.
dpcan 8y ago
Okay, so say you switch banks. How do you have ANY idea if your new bank is better or worse? Are you going to ask customer support? They barely know how to access your account let alone the technical layout of their data centers.
- zepearl 8y ago
  I know that both Switzerland and Singapore have at least "soft" guidelines (no direct retaliation if you don't adhere to them, but frown upon if you don't) about the HW (including buildings & workplaces) & SW and personnel setup required to remediate a potential catastrophic failure of the data centers and/or key-employee's office building. Example of high-level guidelines (Singapore): http://www.mas.gov.sg/~/media/resource/legislation_guidelines/securities_futures/sub_legislation/BCMGuidelines.pdf http://www.mas.gov.sg/~/media/resource/legislation_guideline... I think that in Switzerland all major banks test their disaster-readiness (by switching everything to their secondary datacenters & working locations) of all critical applications/software-layers and employees at least once every 3 years - reaction/recovery times depend on the criticality of the service provided by the person/application.
- cybercontrol 8y ago
  2017 Bank of America had a similar outage,
zhte415 8y ago
I was shocked to learn the Fortune-500 bank I was working for had only a single data centre. Should they ever go down, much of the world would be disrupted. I think this is not unique.
lIl1l 8y ago
And they can't afford even a tiny bit of redundancy? Big companies tend to defer risk. Managers and project leads want to start new projects rather than upgrade existing infrastructure. Combine these forces and sometimes you get a catastrophe.

by mabbo 8y ago

ModFightsCron 8y ago

Incredible. You made up a really stupid and (for a bank) illegal thing, then said it's stupid! If I were a company, I'd use this as a sign you are not competent enough to receive our money. No dipshit. They have active->sync->async or star replication, live every other bank. Guess what - when you actually fails over it's not always perfectly smooth. The OP made up that "all of wells fargo" went offline, when in reality tiny bits and pieces die. Have you ever worked or did you wander off the high school tech subreddit? When you actually fail over, sometimes not everything fails over smoothly and you have to do database media recovery and replay redo logs.

barkingcat 8y ago

I don't think you need this as a sign it's not technically competent. There are other much larger markers.

mywittyname 8y ago

Lots of old companies are like this. They don't invest in critical infrastructure because it would bring them below their 20% margins.

nostrademons 8y ago
Wells is weird though. When I started banking with them in 2009, they were ahead of the curve in online infrastructure - it was much easier to access them online than any of the banks I was used to back in Massachusetts. They also had a reputation for being both honest and conservative - they were the only AAA-rated bank in 2007, they were #1 in green rankings, and Warren Buffett had invested in them because of their sound balance sheet. And then starting around 2010 but rapidly accelerating around 2014, everything about them went to shit. The best explanation I can think of is that John Stumpf is a slash & burn sociopath, juicing the numbers so he can get his 473x-the-median-worker paycheck while ruining the company. He wouldn't be alone in the financial world, but it's a shame that a 150+ year old institution can so rapidly go down the toilet.
- Zelphyr 8y ago
  In 2009/2010 my wife accidentally made a charge to PayPal that was linked with our Wells Fargo checking account. Since we weren’t planning on that charge happening from that account, there was an NSF fee and Wells rejected the charger. Understandable, right? Except then PayPal made a few more attempts for god knows why and each time Wells Fargo kicked an NSF fee our way. Now, PayPal shouldn’t have repeatedly attempted a rejected charge. But, Wells Fargo shouldn’t have allowed those attempts. They just couldn’t help themselves to that $35 NSF fee though. We fought it to no avail. With all the NSF fees and interest (and fees they added to fees while we fought it), what started as a $300 transaction ultimately cost us over $1200. Wells Fargo is now and was in 2009/2010 a criminal enterprise.
- Rapzid 8y ago
  Wells was one of the earlier companies to have the CTO report directly to the CEO. All the crazy sales numbers and bogus account shenanigans were going on back in 2003/2004 when I worked there. I ratted out more than one professional banker to branch managers and up over that crap. A fun one was the home equity lines people would open without customer knowledge and link up to overdraft protection. The customer would never owe, nor know, anything until one day an overdraft hit their equity line, and then they got notified of late payments.. I don't miss working for a Bank.
- mulmen 8y ago
  I have banked with WF since about 2007 for my student and auto loans. Their online portal has always been worse than whatever my local credit union had. The WF business is clearly set up to confuse and exploit consumers. My credit union websites have always helped me do what I want and need with my money. This includes the tiny local credit union in Idaho.

xdlachx 8y ago

true, but you also give too much credit to the average person. Im employed by WF and even Im a little bewildered by the fact there doesn't seem to be any redundancy implemented somewhere.

Zelphyr 8y ago

Wells Fargo is an awful company. Hopefully this outage will be the final straw for many people.

benj111 8y ago

What kind of redundancy are we talking about here? You can't really roll back say 10 minutes of transactions, so are you maintaining 2 parallel systems? How do you keep them perfectly in sync? This isn't my area of expertise by a long shot, but it occurs to me this is probably hard, especially when your codebase started in the 60s, and has been accreting ever since.

gniv 8y ago
Yes it’s hard but there are mature solutions. See Google Spanner for example.
- ex3ndr 8y ago
  Spanner is slow as hell
esonoi 8y ago
You have a primary and a backup with a synchronous commit protocol. When a commit request is made on the primary, the primary writes to its transaction log and the backup’s log. If the backup does not acknowledge, the commit fails. The backup doesn’t need to be in the same exact state as the primary (it’s not meant to service requests), it just needs to have a persistent log of what changes were applied so that it can roll forward when needed. Most relational DBs do something like this for their DR product offering. Oracle has Active Data Guard. DB2 has HADR.

dakial1 8y ago

Maybe this had something to do with it... https://searchdatacenter.techtarget.com/news/4500243727/Wells-Fargo-continues-massive-data-center-consolidation https://searchdatacenter.techtarget.com/news/4500243727/Well...

aboutruby 8y ago
Badly formatted paste of the content of the article: https://8cf91ddf-e6b2-465a-9f10-1f631650f4ed.htmlpasta.com https://8cf91ddf-e6b2-465a-9f10-1f631650f4ed.htmlpasta.com (Even if you can just put anything in the email form)

deleted 8y ago

[deleted]

danpalmer 8y ago

The comment says that they have failovers in place, the just didn’t work correctly.

perl4ever 8y ago
It's not lost, I know exactly where it is...at the bottom of the ocean!

otakucode 8y ago

Everyone assumes that the banking industry and the financial industry in general somehow have a magic touch when it comes to technology. All those digital balance sheets, they're pristine. All those networks, they're unpenetrated. All those databases, they're filled with the purest divine truth. And if anyone ever figures out that isn't the way it is, and that the numbers are not representative of anything of substance? If nations refuse to honor the claimed 'transfers' done through these rickety electronic systems? It would make for an interesting few days.

surge 8y ago

They had redundancy, the redundancy failed. I mean truthfully when do you get to test your redundancy against a true disaster. It was a mess. WF is 20 companies rolled into one so the fact the disparate systems from 10 different banks works at all is kind of a miracle.

vonmoltke 8y ago
Bloomberg does it once a year. That said, I have yet to encounter a company more obsessed with business continuity. I don't doubt that their failover systems and testing of them are well beyond the typical.
- surge 8y ago
  I think Bloomberg can stand to be down for 8 hours to simulate a disaster. Banks with legacy systems and people constantly dependent on them to conduct business can't risk an actual incident happening because they were testing what would happen if an incident happened. Netflix designed their stuff from the ground up to fail over. Large monolith corporations who've inherited systems from other companies they've bought or merged with have challenges you won't see many places that have benefited from the 30 years of lessons that were taught at these companies.
  - kamikaz1k 8y ago
    Well that's not true. Chaos engineering and AWS weren't real things when they started building the company. And the system they have now doesn't resemble much of it was once. Truth of the matter is they invested more in their infrastructure, but that's because their business plan required them to grow on the back of technological advances. Banks, it's seems, do not. Or maybe they do, and the some of these start up banks will usurp them.
    - ianai 8y ago
      Standard good practice should be to have a redundancy in place and test it at a regular interval. It should be part of periodic maintenance - fail to the backup so updates/grades can be applied to production and fail back to production once done. But I’m guessing wellsfargo just doesn’t have a reason to care.

gdulli 8y ago

> If I were a customer Are you sure you're not one?

dekhn 8y ago

you jumped immediately to an unfounded speculation which is known to be wrong (WF maintains multiple data centers). Also, failover is hard. Few companies outside of a few larger Internetz companies can really do it well.

paul7986 8y ago

What all their scandals didn’t convince you of that? After all their logo is that of a stage coach aka the wild Wild West(robbers, thieves, etc). They do not hide who they really are.

jstarfish 8y ago
The stagecoach refers to their origins as an armored car/delivery service.
- paul7986 8y ago
  uh that may and well be true, but their behavior in recent years says stagecoach.. wild, wild west.. robbers and thus their logo is now a punchline/comedy routine.

WrtCdEvrydy 8y ago

If you're waiting for this to be the straw to move away from Wells Fargo after the last set of scandals, you may have some issues...

CydeWeys 8y ago
Unfortunately they're the holders of my mortgage (it was sold twice and they ended up with it). So I don't even have a choice of being their customer for the next ~25 years.
- edoceo 8y ago
  Refi that bad-boy to your local credit union
  - abawany 8y ago
    I moved loans to First Tech Federal Credit Union, which used to (and still probably do) state in the loan contract that they will service the loan for its life. Very happy with them so far.
    - funkymike 8y ago
      That's good language to look for. I didn't check closely enough and my mortgage documents stated that the loan issuer (my bank) would service the loan, but without any guarantee of how long. They sold the loan shortly after issuing the mortgage.
  - hinkley 8y ago
    It doesn't work that way. Mortgages get resold between banks all the time. You can walk into your local bank and get a house loan, and five months from now you'll get a business envelope from Wells Fargo informing you that you should send your checks to them now. I would expect if WF decided your first loan met their purchasing criteria, your refinance would get the same treatment.
  - CydeWeys 8y ago

samstave 8y ago

List of WF scandals since 2015 https://finance.yahoo.com/news/every-wells-fargo-consumer-scandal-since-2015-timeline-194946222.html https://finance.yahoo.com/news/every-wells-fargo-consumer-sc...

elliekelly 8y ago

Every major financial regulator has business continuity and disaster recovery requirements but the standards are woefully outdated. A plan that gets the bank back to full functionality within 24 hours would be acceptable to most regulators and even considered speedy to some. Tangentially related, I highly recommend the movie "Out of the Clear Blue Sky." Cantor Fitzgerald was a bond trading firm at the top of one of the twin towers and lost every employee who was in the office on 9/11. Incredibly, despite losing the majority of their employees and despite losing almost all of their trading infrastructure, they managed to resume operations in time for the bond market's reopening 48 hours later.

dpcan 8y ago

Okay, so say you switch banks. How do you have ANY idea if your new bank is better or worse? Are you going to ask customer support? They barely know how to access your account let alone the technical layout of their data centers.

zepearl 8y ago
I know that both Switzerland and Singapore have at least "soft" guidelines (no direct retaliation if you don't adhere to them, but frown upon if you don't) about the HW (including buildings & workplaces) & SW and personnel setup required to remediate a potential catastrophic failure of the data centers and/or key-employee's office building. Example of high-level guidelines (Singapore): http://www.mas.gov.sg/~/media/resource/legislation_guidelines/securities_futures/sub_legislation/BCMGuidelines.pdf http://www.mas.gov.sg/~/media/resource/legislation_guideline... I think that in Switzerland all major banks test their disaster-readiness (by switching everything to their secondary datacenters & working locations) of all critical applications/software-layers and employees at least once every 3 years - reaction/recovery times depend on the criticality of the service provided by the person/application.
cybercontrol 8y ago
2017 Bank of America had a similar outage,

zhte415 8y ago

I was shocked to learn the Fortune-500 bank I was working for had only a single data centre. Should they ever go down, much of the world would be disrupted. I think this is not unique.

lIl1l 8y ago

And they can't afford even a tiny bit of redundancy? Big companies tend to defer risk. Managers and project leads want to start new projects rather than upgrade existing infrastructure. Combine these forces and sometimes you get a catastrophe.