12 ms·
How Equifax Was Breached in 2017
- vb-8448 3y agoWow, basically everything that could go wrong ... gone wrong. I wonder how they realized about their NIDS expired certificate.
- demizer 3y agoIf any company deserves the corporate death penalty, it's Equifax after this fiasco. That company should no longer exist.
- deleted 3y ago[deleted]
- dang 3y agoOk, but please don't post generic comments, and especially not generic-indignant comments, to HN. They lead to significantly less interesting discussion.
- smegsicle 3y ago[flagged]
- hyperdunc 3y agoYes, the punishment should be immense. But we all know there's no real justice to be had here. In places like China, there's personal accountability at the highest level of an org for major screw ups - sometimes even capital punishment. If we put such options on the table here, perhaps corporations would be a little less callous with people's private data, and a little less eager to collect it.
- cookiengineer 3y agoTLDR is: Equifax had no working firewall / intrusion detection for almost a year, because they did not update their snakeoil MITM certificate and forgot about it. Remind me again, how did Equifax get SOC 1&2, and ISO27001 certified? Oh yeah, they probably have a checklist for that, so they must be secure. /s
- bishopsmother 3y agoDoes the use of snakeoil in the TLDR not run contrary to the narrative in the blog? When the 'snakeoil MITM' certificate was updated - they became aware of, as a direct result of MITM, a problem that had not previously been known to them?
- Roark66 3y agoPersonally I think the root cause of this was bad documentation practices. If the old system was properly documented they would've scanned the right folder. Likewise with the certificate, if there was documentation to indicate when that cert expires (or monitoring to alert few weeks in advance) they would have a functioning ids and these web shells would be found immediately. Unfortunately, out of half a dozen fortune 500 companies I worked for perhaps 2 had doc practices good enough to prevent this.
- deleted 3y ago[deleted]
- yuliyp 3y agoThat feels like the wrong conclusion. Assuming documentation will be followed properly is not a reasonable security strategy. Validation and monitoring is needed. That their NIDS gracefully degraded to a "don't monitor the payloads" when it was expected that it would be monitoring those and nobody noticed is a problem. A scan of a system which misses a web server running it without erroring is a problem.
- hn_throwaway_99 3y agoCouldn't agree with this more. While I think it's important to have good documentation, it is nearly always a very bad idea to rely on that documentation being 100% correct. Businesses simply have way too many moving parts to assume the state of the world is always up-to-date in the documentation. You also highlight a very good point. Things like security software should "break loudly", i.e. beyond just sending alerts (which can be ignored), there should be some explicitly "painful" steps that occur if the security system is in a broken state for long.
- EGreg 3y agoRelated: https://qbix.com/blog/2023/06/12/no-way-to-prevent-this-says-only-industry-where-this-regularly-happens-2/ https://qbix.com/blog/2023/06/12/no-way-to-prevent-this-says...
- justinclift 3y agoDidn't Equifax receive practically no penalty for it though? So, what would be the motivation to avoid future things like this happening again?
- smegsicle 3y agomake sure to become too big to fail before you fail
- RachelF 3y agoA tiny penalty. The CIO got a $3M bonus, too. Odd thing is that she had a music degree and little experience in IT, but was an old friend of the board members.
- hn_throwaway_99 3y agoNot enough downvotes for this. I'm assuming this is all BS considering you got all the details wrong. It was the CEO who got a $3 million bonus in 2016, not the CIO. Susan Mauldin, who earned a music degree in college, was the Equifax CISO, not their CIO. The reason I'm so salty about your response is when the breach happened, there were tons of news reports denigrating the CISO because she had a music degree. There may be a ton of reasons she wasn't good at her job (though it's hard to say as CISO is often a "sacrificial lamb" job anyway), and I'm certainly not defending Equifax, but I take major issue with the implication that a music degree makes someone unqualified for a tech job. First, as she was CISO, she was presumably done with college many, many years ago. Lots of people have college degrees that aren't necessarily directed to the career they end up in. More importantly, though, I've found that there is a direct correlation between highly trained musicians and great software engineers. I don't know if it's a "same part of the brain" thing or whatever, but I'm actually astounded at the sheer number of "best of the best" software engineers I've worked with that are classically trained musicians. It's to the point that when hiring I give "extra points" if you will to musicians because, it my experience at least, the correlation is so strong. So, frankly, you can take your "she had a music degree" shade and shove it.
- 3y ago
- schemescape 3y ago> Malicious actors had been exfiltrating data for several months and had already collected personal information from 163 million customers. I don't think "customers" is the right term, considering I never wanted them collecting data about me.
- RachelF 3y agoYes, this is what most people don't understand with data breaches: it's not the company's data, it's data on others. That's why they don't really care about protecting it.
- dwd 3y agoThat is not correct for a data brokerage as the data is the business. Lose your monopoly on that data and you have no business. If it is information collected as part of doing business, then yes; they don't care. A good reason to question any Gov attempt to implement centralisation of data like identity or medical records.
- forkerenok 3y ago> Lose your monopoly on that data and you have no business. But do these breaches affect their monopoly? My thinking is: 1. B2B customers won't go on darknet to source illegal data dumps. 2. This data, even if it doesn't quickly become effectively stale, would be considered stale by businesses very quickly if it's not connected to the continuous data ingestion pipeline.
- dwd 3y ago1) Customers, probably not. Competitors I would not be so sure they wouldn't have look. 2) This is not specific to the data that underlines consumer credit scoring; a broker could be selling products derived from data on historical house prices or car sales for example. A competitor might use it to compare and validate their own dataset or simply have a look. Third party investigators, journalists, etc though could have a field day fact-checking it.
- k1rcher 3y agoWow, the fact that they remained undetected for so long and used wget for data exfiltration.. Hopefully security posture has increased since
- x86a 3y agousing wget here does not make it more embarrassing as its user agent was almost certainly randomized to look like normal web traffic. Normal traffic downloading lots of 10MB files... well, yeah that's not great.
- wordpad25 3y agoWasn't Equifax Chief of Security a Music major? That was hilarious to read about...
- YeBanKo 3y agoI have met quiet a few people with no-tech degrees that made it in computer related IT fields. This alone is not an issue, but she hadn’t seem to have much experience in security at all.
- te_chris 3y agoMusic major CTO here. Jog on. God forbid our executives be trained in creativity.
- sk0g 3y agoMost CXOs have extensive experience in their relevant fields. Your retort seems needlessly defensive.
- xwolfi 3y agoAs long as you're also trained in IT...
- orwin 3y agoYou mean college-trained? Because GP is a CTO, so I guess he has experience. I don't respect the C-suite that much, but I've never seen CTO without solid SWE knowledge.
- H8crilA 3y agoThere are different types of creativity. Working in security actually requires a lot of a specific type of imagination/creativity that pretty much isn't used anywhere else.
- saagarjha 3y agoA lot of people in security don’t have a degree at all, let alone in computer science. Judging people’s qualifications is much more complicated than just looking at their degree.
- est 3y agoah, the good old struts2 exploit.
- dwd 3y agoNot mentioned here was that the group that exploited the vulnerability handed over to PLA linked individuals who then conducted the exfiltration. https://www.justice.gov/opa/pr/chinese-military-personnel-charged-computer-fraud-economic-espionage-and-wire-fraud-hacking https://www.justice.gov/opa/pr/chinese-military-personnel-ch... As far as I am aware the data has never been seen on the open market, so there's a whole other National Security story around whether the information was used to compromise individuals with credit issues for commercial and military espionage purposes. It would seem that this was known very early on and possibly factored into the settlement.
- b112 3y agoAlso mis-mentioned, is that I heard nothing was "missed" but security upgrades were not possible due to the age of the stack. Pre-0 days are one thing. But leaving systems unpatched for months, because your stack is too old, is a common, but inexcusable theme. This is why it is vital to use libraries, frameworks, with a stable, unchanging LTS branch. Failure to do so, means a security update that needs to be applied instantly, cannot be done, without extensive app changes. New shiny is fine. But it must never, ever override basic security concerns. Security comes first. Not last. Always.
- dwd 3y agoBeen a few years since I read it, but worth a look due to the detail it goes into. https://www.hsgac.senate.gov/wp-content/uploads/imo/media/doc/FINAL%20Equifax%20Report.pdf https://www.hsgac.senate.gov/wp-content/uploads/imo/media/do...
- hnthrowaway0315 3y ago"They routed traffic through approximately 34 servers located in nearly 20 countries to obfuscate their true location, used encrypted communication channels within Equifax’s network to blend in with normal network activity, and deleted compressed files and wiped log files on a daily basis in an effort to eliminate records of their activity." I wonder how they managed to figure that out. Did they have to look into each of the servers? How did they get the names?
- hn_throwaway_99 3y agoI really appreciate detailed breach reports like this. This was the money quote for me: > The attackers continued their search and eventually discovered a mounted NFS share on the web server. This file share contained notes and configuration files used by Equifax engineers, in which they found many database credentials. Seriously, WTF? I get paranoid all the time worrying about my application security - it often feels like there is always some potential issue around the corner you don't know about. But then I read about how lots of these kinds of breaches occur (storing prod DB credentials in plaintext on an NFS share, reusing passwords and not using 2FA, leaving your server password as "solarwinds123", etc.) and I think maybe I'm not so bad after all.
- keyle 3y agoI'm totally with you on this one; but remember, if you have 25 developers in groups of 5, in only takes 1 muppet in any of the 5 groups to have low standards, and voila. I've seen it, in pretty much every large business I've worked in. This goes back to the saying: "you should never hire someone less good than yourself". Sadly when the people hiring literally come from sales or airline customer service, your company is boned. It's only a matter of time.
- hn_throwaway_99 3y agoI agree with all that, with the one small caveat that more than anything else I think what is most important about security is a strong security culture at a company. All the checklists and compliance frameworks in the world are doomed in the face of a poor security culture. On the flip side, a strong and constantly reinforced security culture can help protect against the occasional muppet. One example: years ago I started work at a tech company (a fintech no less), and shortly after starting I asked the head of customer service how I could get an account to access an internal admin portal (I was an engineer and needed to understand some of ops processes). "Oh, you just log in with my account, and the password is <CompanyName><Year> - all the reps just use that shared account" I got an immediate sinking, sinking feeling of despair.
- 3y ago
- Forgotthepass8 3y agoBold of them to not spoof the wget UserAgent
- Falkon1313 3y agoThis is a nice list of could've, should've, would've's. But you'd have to dig deeper to get to the core. Why did these things happen (or not happen)? Insufficient training? Insufficient processes? Were changes being reviewed and accepted by people who didn't really understand the changes, for expediency? Were there alerts but they were lost in the noise of thousands of bogus alerts people had learned to ignore? Was the lack of segmentation a known issue but allowed because it made some things easier? Were the credentials stored on NFS because they simply hadn't setup a more appropriate system yet and that was considered low-priority? Were business priorities getting in the way of technical priorities such that known issues were backlogged? It's fairly easy to make a bullet list of things that should (or shouldn't) be done. It's a bit more difficult to figure out why, in a specific organization, those things aren't (or are) being done. Even if/when people might know that they should/shouldn't. The surface level mistakes are interesting. The deeper organizational causes of those mistakes would be interesting. Solving those things at a higher systemic/organizational level can reduce the whack-a-mole nature of individual mistakes.
- blisterpeanuts 3y agoI’m taking a cybersecurity course right now and this article is timely and informative. I’m a programmer with a lot of Java and database experience, but not really knowledgeable about security practices. Maybe security certification should be more of a requirement in hiring software engineers; I don’t recall it ever being mentioned in job listings. Anyway, it got me wondering, how did devs get away with storing database credentials in a file on an NFS share? That’s sheer recklessness. As a regular procedure, an audit should include scanning all files for passwords, for example; run find-grep-dired or similar on every mount, every disk, every cloud instance etc. And, obviously, require regular password changes. It should be assumed that the entire system is vulnerable, and hardening should be done regularly and rigorously. A company as big as Equifax (or Target) should have a dedicated team whose job it is to constantly probe and audit. Since, after all, the black hats are constantly probing, too.
- hnthrowaway0315 3y agoA lot of companies are like that. In my previous company people sharing username and password through MS team and I'm sure someone stores them in team folders too.
- blisterpeanuts 3y agoIn the early 90s, at a large financial company I worked for, the system user name/password for a Sybase db was sa/sa. It was so convenient. Of course this was the primordial days but still.
- hnthrowaway0315 3y agoSybase! Ah, my father has a book about that. Yeah I get back in the days many are ignorant about security.
- tekla 3y ago> an audit should include scanning all files for passwords Please continue taking the security course. Scanning all files for passwords is madness. How do you differentiate "thisissupersecret" and "123fqfqlfni34235r4" and "git@somegitrepo.com" as passwords? You can't, they're all valid passwords for a majority of services. At some point, you need to trust developers to do the right thing, which is impossible.
- simonswords82 3y agoNothing irritates me more than two for profit companies (Equifax and Experian) who have a license to print money by collecting my data without my explicit permission. Even with the introduction of GDPR and all the new consumer protection this brought about, I cannot ask them to delete all of my data. They should not exist, or if they must exist they should be not for profit. It's a total scam.
- JCM9 3y agoHaving a plaintext file with “notes” containing the login credentials for a database containing actual customer data with PII is borderline criminal negligence. What a friggin disgrace.
- nunez 3y ago> The attackers continued their search and eventually discovered a mounted NFS share on the web server. This file share contained notes and configuration files used by Equifax engineers, in which they found many database credentials. Crazy that the user that ACIS was running as had enough permissions to access NFS mounts to begin with. It’s also crazy that the attackers even found ACIS. This was an insanely dedicated attack.
- e79 3y agoFor anyone curious about how the exploit worked: https://www.aon.com/cyber-solutions/aon_cyber_labs/an-analysis-of-cve-2017-5638/ https://www.aon.com/cyber-solutions/aon_cyber_labs/an-analys...
- throwaw1yyy 3y agoI’m planning to sue in small claims court, did anyone also do this and have any tips?