14 ms·
Tell HN: Wells Fargo completely offline
Wells Fargo is completely offline. How does this happen? Been this way since 6am PST!
- bridanp 8y agoRegional bank checking in. We've been told to reroute certain workloads through other means until 6am central Friday. Regardless of that, the story coming out of this over the next few days is really going to help Business Continuity departments at other institutions plan for this scenario. My teams are running mock scenarios this afternoon based on the auto power shutdown causing a data center to be unable to fail over. Might end up being something else entirely, but it's still a good scenario to investigate.
- a3n 8y agoI get paid by direct deposit into my WF acct between midnight tonight and 6am tomorrow. I suppose I can eventually rely on my employer keeping records and replaying everything.
- bridanp 8y agoIf ACH areas are involved, hopefully they already had those queued a day early per normal routines and you'll (hopefully) be fine. There are exceptions allowing companies to provide ACH files late. That's the exception and not the rule.
- byron_fast 8y agoI work at a place that just had a near 24 hour outage due to a frozen water pipe. Water!
- ams6110 8y agoCooling water? Essential in a data center.
- clairity 8y agofor those who haven't taken the plunge, now is a good time to move to a credit union, regional bank, or online bank: - for instance, in cali: https://www.golden1.com/ https://www.golden1.com/ - high interest accounts: https://www.bankrate.com/banking/checking/best-checking-account-rates/ https://www.bankrate.com/banking/checking/best-checking-acco... - good rate online bank: https://empower.me/ https://empower.me/ get better service, lower/no fees, and good convenience without all the economy-damaging selfishness and hubris.
- legohead 8y agoI imagine a credit union would be susceptible to the same type of thing (single location/point of failure) due to their smaller sizes.
- tatersolid 8y agoCredit unions and small banks outsource almost all of their core systems to companies like Fiserv. Literally thousands of banks and CUs all ride their infrastructure
- themodelplumber 8y agoI ran a website for a local CU for about 10 years. It was pretty fun doing things for their website that made their Fiserv web tools look like they were from 1970. There was no interaction with user or account data on my end, just CMS-building, letting them go crazy with coupon promotions or pet shelter PSA's or warnings about ATMs being down, etc. They'd pitch me on a new project, I'd send them a timeline, and we'd add new features or tools to the website. I even hired a local college student to create some super-basic but useful financial tools for the site. I loved how small-town-feel the whole thing was. At the beginning of the relationship, they gave me a list of broad requirements like "SAS-70", I found a DC to match, I sent them a contract, and I couldn't believe I actually had a banking institution as a client, 4 years out of school and a brand-new business owner. Eventually they merged with another CU and went away, but apart from the very occasional "server down" notifications while I was on vacation, those are some really fun memories.
- rixrax 8y agoAnd I’m not surprised they don’t have a services status page either that many modern companies do offer[0][1][2]. [0] https://api.twitterstat.us https://api.twitterstat.us [1] https://help.netflix.com/en/is-netflix-down https://help.netflix.com/en/is-netflix-down [2] https://developers.facebook.com/status/dashboard/ https://developers.facebook.com/status/dashboard/
- 0xfeeddeadbeef 8y agoLooks like Mr. Robot is at it again.
- api 8y agoToo big to fail over?
- rawrmaan 8y agoUnderrated comment
- meesterdude 8y agothey have a history of outages - though not as long.
- MuffinFlavored 8y agoWhat is more likely going down in the middle of the day? Their mainframe/COBOL stack? Their Java stack?
- berbec 8y agoTheir entire data center, it seems
- webdestroya 8y agoFrom their API team: Please be advised that the Wells Fargo Gateway is currently unavailable. We're experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We're working to restore services as soon as possible. We apologize for the inconvenience as we continue to work on a resolution. We will update you no later than 3:00 pm, Eastern Time.
- rococode 8y agoAmazing that one of the largest banks in the US is somehow still (apparently) reliant on a single location.
- nemothekid 8y agoApparently there was a fire in a Datacenter - theres a thread on /r/sysadmin by an insider. https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo_is_down_declining_transactions_and_no/ https://np.reddit.com/r/sysadmin/comments/ao4g2y/wells_fargo...
- jwcooper 8y agoLocal fire Department commented on their facebook page [1] that dust from construction triggered the fire suppression system and not a fire. [1] https://www.facebook.com/LakeJoFD https://www.facebook.com/LakeJoFD
- bitxbitxbitcoin 8y agoThe next worst thing for electronics after fire - water! Or are there non liquid fire suppression systems such as pulling all the oxygen out of a room (if there aren't any people in it)?
- ptr 8y agoYes water isn't typically used in datacenters, halon gas can be used for example. But the noise of the nozzles releasing the gas can actually damage disk drives.
- xdlachx 8y agoyeah, the data center affected apparently had a halon gas suppression system.
- regecks 8y agoThere's a nozzle for that :) - http://www.fire-protection.com.au/news/hush-nozzle http://www.fire-protection.com.au/news/hush-nozzle From what I understand, using water/dry pipe isn't unheard of. Some prefer it over Halon - https://blog.equinix.com/blog/2014/03/26/we-must-protect-this-house-against-disaster/ https://blog.equinix.com/blog/2014/03/26/we-must-protect-thi... .
- Talyen42 8y ago3rd largest bank in the world offline for an entire day because a smoke alarm went off great job guys
- wolfpwner 8y agoMore like 3rd largest bank in US
- qaq 8y agopretty interesting attack vector
- drdeadringer 8y agoI feel like I've seen this in 'Mr Robot' or similar.
- rubbingalcohol 8y agoYou can still access Wells Fargo Online via the direct link: https://connect.secure.wellsfargo.com/auth/login/present?origin=cob&LOB=CONS https://connect.secure.wellsfargo.com/auth/login/present?ori...
- sxates 8y agoConfirmed - was able to log in. Some account info is still there, though my mortgage is unavailable. Maybe they'll just lose it?
- howard941 8y agoYour deposits, sure, your trust, of course. The mortgage loan? Fat chance.
- propogandist 8y agoUnless evil corp was using a blockchain to have an immutable ledger, there's a chance (although extremely small)
- tastroder 8y agoIf evil corp stores said buzzword stack exclusively on that large rack that's currently on fire because of compliance nothing is gained. Not every problem is a nail.
- mulmen 8y agoI don’t follow, are you saying this outage was caused by compliance or that the blockchain can’t help here because of compliance?
- tastroder 8y agoNeither, I was trying to suggest that offering Blockchain and related technologies as a solution to every problem in existence is kind of an annoying trend. I don't really see the outage discussed here as a technical problem. Quite literally any failover mechanism would have been preferrable here.
- miscreanity 8y agoOne data center brings the whole bank down? Right... This is suspiciously quickly after military exercises in LA right next to a Wells Fargo. It looked like something was being carried out of the building. https://www.rt.com/usa/450746-helicopters-blasts-los-angeles-drill/ https://www.rt.com/usa/450746-helicopters-blasts-los-angeles...
- aphextron 8y agoRT link? Check. Completely unfounded conspiracy theories? Check. /r/the_donald is that way buddy ->
- SteveNuts 8y agoWow, even the Shoreview MN fire department is in on this conspiracy! This goes all the way to the top! https://www.facebook.com/LakeJoFD/ https://www.facebook.com/LakeJoFD/
- samstave 8y agoThis will be an interesting root cause if it manages to get leaked....
- fidla 8y ago4chan's pol forum is all over it. Apparently it's a "happening" worthy of hundreds of comments
- krapp 8y agoReally? /pol/? this is what the merry band of racist edgelords have fixated on? Seems like it would be more /g/'s thing.
- lenticular 8y agoAnything that they can spin into some insane fascist conspiracy theory.
- imsodrunklol 8y agoNormal Thursday for them.
- joejerryronnie 8y agoAOC_Sec bringing down the global financial industry.
- deevolution 8y agoBullish on bitcoin b/c of its highly resilient characteristics. Might not process as many transactions per second and it might pollute more than the average bank, but sure as hell dont need to worry about a natural disaster or freak incident single handedly taking out the entire network.
- amaccuish 8y agoThere's always one...
- mikeash 8y agoI’d wager that WF is still processing a lot more transactions than Bitcoin is even during this outage.
- jameskilton 8y agoWells Fargo customers must be terrified that their money is going to be gone when the service finally comes back. This downtime is obviously to prevent a bank rush while the leadership leaves the country with millions in customer funds. Oh wait...
- deevolution 8y agoQueue infinite stream of downvotes. News flash, you dont need an exchange to store or transact bitcoin!!! And people using exchanges should understand the risks associated with using a centralized service.
- perl4ever 8y agoWhat do you mean by "need"? If you don't have an exchange you can trust, doesn't that make things worse, because you have to determine whether you can trust your counterparties and you have to secure your assets with your own resources? Matt Levine wrote the other day: "...people think they don’t trust the traditional financial system, but in fact its most basic functionality—not forgetting where it put your money—is so reliable that people just keep assuming without evidence that the crypto system must work the same way. Surely a big crypto exchange must have, like, systems, and lawyers, and more than one employee, so you can trust it, right? Wrong! If I ever buy Bitcoins I will keep them at Gemini, the Winklevoss twins’ exchange, just because I am absolutely sure there are two of them."
- sys_64738 8y agoLots of new Wells Fargo job vacancies will be published tomorrow I fear.
- mikestew 8y agoYeah, but it won't be to fill the roles vacated by those that failed to fund data center redundancy despite repeated requests and proposals.
- rhacker 8y agoIn before fight club soap.
- godelmachine 8y agoWell, for a second I thought they have closed business operations completely, keeping in mind how embroiled they were in graft cases in 2008 recession. Pardon my immense ignorance!
- nodesocket 8y agoI thought at one time Stripe used Wells Fargo as their bank. Any reported problems from Stripe?
- driverdan 8y agoI signed up for Stripe in 2011 or 2012. About six months after I signed up an account rep from Wells Fargo called me asking about my merchant account. Stripe never made it quite clear what was going on but apparently in the early days they opened accounts at Wells Fargo on your behalf without telling you.
- dawnerd 8y agoWasn’t Wells Fargo caught for making accounts for people without permission just to boost their numbers?
- patio11 8y agoI also signed up for Stripe in 2011. The Terms of Service mentioned Wells, beginning in the first paragraph. You might want to look in your files and/or the Wayback Machine. https://web.archive.org/web/20111017081436/https://stripe.com/tos https://web.archive.org/web/20111017081436/https://stripe.co...
- peter_d_sherman 8y agoSpeaking as a Law afficionado, that is one great Terms of Service document! Kudos to whichever legal team wrote it! I'm not saying that it completely anticipates all potential customer-company interactions that could occur (no legal document could), but it seems to go very far, and seems to show a good understanding of the many potential issues that can arise. My future company might use this document (and others) for inspiration on how to craft its own Terms of Service documents. Thanks for the link!
- headgasket 8y agoWells Fargo..ne
- habosa 8y agoFor those who have worked in bank IT, I have some questions. So these days there's very little physical money. Most of my "wealth" is just entries in various digital ledgers. My bank says I have $XXX and my brokerage says I have $YYY and my retirement account says I have $ZZZ. Let's go with the bank account case. Is it possible that a catastrophic accident or attack could wipe my balance down to $0 with no way to recover? What if a data center was nuked? What if two data centers were nuked? How much redundancy is in the system? Are there third-party agencies that track private bank ledgers? How hard is it to take them out too? Ever since I read "The End of Alchemy" (a great book, btw) this thought has haunted me.
- 40acres 8y agoGenerally I think here is where the FDIC and SPIC would come in.
- bowmessage 8y agoDo they also have backups of account balances across all insured accounts then?
- gdubs 8y agoI’d love someone who knows the details to explain how that claim process would work, because while the FDIC would insure in the event a bank goes under, in that scenario there’s usually records. What would happen if there were no records? Surely there’d be lots of people making significant deposits in between snapshots. If you’re “paperless” I’m not even sure how you’d reconstruct your balance. Would certainly be time consuming, and in this case time would certainly be money.
- Aloha 8y agoYou presume the live system is the only record - its not, there are backups of those, then historical records to reconstruct what might have been there. Bear in mind the live records are only one part of what a bank keeps.
- 8y ago
- rolph 8y agoOK, Hanlon's razor applies here: https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor recall Stuxnet, its possible this was an attack, somesort of malicious mod to firmware, but_ Hanlon's razor. from the reddit: "throwawayfordays75 1399 points 8 hours ago*2 Throwaway since I have first hand knowledge. Fire suppression went off in one of their main Data Centres from some utility work this morning. No power to any of the network or compute equipment and some failovers did not work as expected. " At this point im wondering what "utility work" was happening.
- hinkley 8y agoThat user posted an update: "everything minus core network gear was manually being unplugged from any PDUs to help the control the initial power-on." Can't we... Why don't we have rack hardware that can handle this situation? I thought some HDD RAID solutions had circuitry to keep them from browning themselves out while spinning up the disks. I guess I'm surprised this isn't a solved problem at the rack level now. Or have we been so focused on never cold booting a rack of servers that we haven't spent any effort on foolproofing of cold booting a rack of servers? [Edit: answering my own question] apparently these exist and are called Managed PDUs. Can we deduce WF doesn't have them?
- rolph 8y agoit would seem to be a possibility. im also thinking 10 finger power up sequence, as in: insert power cord 1 into powersocket 1, wait till audible beep, insert power cord 2 into power socket 2, wait till audible beep etc. :-D
- deleted 8y ago[deleted]
- mustardo 8y agoSome servers "stage" power on to disks. as spinning up disks is often the largest source of inrush current for e.g. a 10 disk server might only apply power to bays 2 at a time with a delay between each, this doesn't solve the inrush current problem of switching on a rack full or servers together (where managed PDUs come in)
- eweise 8y agoTry Varo
- chad_strategic 8y agoWells Fargo is nothing more than a federally regulated crime organization, supported by the federal reserve and the American tax payer, but I digress. This thread has primarily focused on redundancy and software architecture. That could be the case, but there is no better way to fight proxy war via hacking of banks. It’s a domino effect... pull your money out of the bank, the bank calls loans, then there is credit crunch, investors lose confidence in the stock, value lost etc... the enemy has secretly inflicted civilian problems across the economy. Distrupting the banks and the flow of money can lead to a revolution... or take your mind off of America’s enemy or diverting energy else where.
- jedberg 8y agoFor everyone complaining about "how could they not have failover" let me ask you this: Would you take a job with Wells Fargo to fix their infrastructure? And if you did take such a job, how long do you think it would take you to get the budget and approvals from all the auditors necessary to fix everything? How much would they have to pay you to take that job, knowing how frustrating it would be to get anything done? And now you see why banks have such terrible IT. (One of my mentors actually works at BofA, and says he only does it because he gets to work 6 hours a day, gets a VP title and a ton of money, but nothing ever gets done)
- benoror 8y agoNice mentor you got there
- deleted 8y ago[deleted]
- tastroder 8y agoIt's not like people blame the staff working on wiring the network hardware here. For any company in that position I'd assume they realize their dependence on that part of the infrastructure for their core business. I don't really see the relevance of how long compliance processes take, that's part of the industry and WF didn't discover the internet last week. Salaries in finance reflect that frustration, who cares. If the answer to that challenge was one of "too hard", "it works for now" or "too expensive" that's still a strategic failure that impacts their core business now.
- smashingx 8y agoWho cares anyway, that bank sucks. Whoever banks with them should be so stupid to do it.
- bigbassroller 8y agoI got an email today from WF titled “New year. New look. Continued commitment.” Maybe they are flipping the switch on their “New year. New look...”; which sounds like a new redesign and they are having downtime while switching over.
- williamstein 8y agoWow, functionality is still significantly degraded over 24 hours later. I just tried to log in to check on an account balance, and the site was extremely slow. It showed my balance, but failed every time to load the transaction history with "Error in external system" or something like that. The sign in page still shows: "Alert: Some customers may be experiencing issues accessing online and mobile banking. We apologize for any inconvenience."
- sugarygrind 8y ago#bitcoin -- just had to say this.