8 ms·

> The attackers transferred this data out of the Equifax environment, unbeknownst to Equifax. Equifax did not see the data exfiltration because the device used

> The attackers transferred this data out of the Equifax environment, unbeknownst to Equifax. Equifax did not see the data exfiltration because the device used to monitor ACIS network traffic had been inactive for 19 months due to an expired security certificate. > Equifax had allowed over 300 security certificates to expire, including 79 certificates for monitoring business critical domains. (on page 2 of the Executive Summary) I've been following the Equifax breach story but this is the first I'm hearing about the expired certificates. That is shockingly bad. I'm a little disappointed in the final "conclusion" of the report, though. The end of the executive summary basically chalks the breach up to two things: "Equifax's IT management structure was complicated" and "Equifax uses legacy software that is hard to secure". These are valid points, but these are also issues that nearly every single major corporation in the world faces, and yet many of them still manage to prevent (or at least mitigate) major breaches. These aren't good enough reasons to explain why Equifax failed so spectacularly compared to every other bureaucratic company with legacy software. Also, I know this report isn't meant to be a remediation strategy roadmap, but it's also pretty disappointing that the recommendations section is basically just 3 pages of fluffy, vague, "X and Y should work together to increase cybersecurity" bullshit. Such a high profile incident would have been a great time for the federal government to really show some leadership (or at least strong guidance) in this realm, but they really didn't. I mean hell, at least link your recommendations to the NIST Cybersecurity Framework...

codazoda 8y ago
Can someone explain why an expired certificate, on a monitoring device, would cause the device to completely fail rather than just spit out warnings about the certificate?
- david_shaw 8y ago
  My guess is that they were using private certificates to read encrypted data, without which they couldn't inspect their traffic.
  - jacquesm 8y ago
    The exfiltration attempt would have likely been triggered on volume alone. Possibly it was simply a service that could not longer connect due to a lapsed cert trying to raise the alarm.
- txcwpalpha 8y ago
  Spitballing here based on my experience in the field but without any specific knowledge about Equifax's situation: Their monitoring system might have been using TLS to communicate the events to their aggregation tool, and when the cert expired, you don't really want log data with potentially confidential/critical security information traversing insecure channels, so they may have had it configured to not send any data if the cert wasn't valid. As for warnings about the cert, it's possible they (stupidly) configured it to not send warnings, or maybe it was sending warnings but nobody was paying attention. I've seen situations before where such warnings were set to go to XYZ person's mailbox, but XYZ person leaves the company and nobody remembered to update the destination address for the alerts.
- viraptor 8y ago
  (speculation) It could be that the expired certs were on the ingestion side. The monitoring agents tried to report something to the central place, but failed due to expired cert on the TLS connection. No other monitoring picked up that new data is not incoming.
- ianmcgowan 8y ago
  Uniformed speculation, but I imagine the monitoring device sends warnings and such to a central SIEM/log server (like splunk) for analysis and correlation across multiple devices. That channel is likely web server calls or json posts, over https.
- evolvedlight 8y ago
  Page 34: The default setting for this device allowed web traffic to continue through to the ACIS system, even when the SSL certificate was expired. When this occurs, traffic flowing to and from the internet is not analyzed by the intrusion detection or prevention systems because these security tools cannot analyze encrypted traffic.
- sorlon 8y ago
  Maybe the cert on the web server was updated, but the private key for the updated cert was never copied to the SSL Visibility Appliance?
david_shaw 8y ago
This is the first time I'd heard anything about expired certs, too. It's absolutely ridiculous that any kind of a security team -- not to mention one that's supposedly safeguarding such an immense amount of data -- could let monitoring certificates expire at all. It's almost inconceivable that they could let seventy nine certificates that are required for security monitoring expire for months on-end. I've worked in security for my whole career, and I'm firmly of the optimistic belief that someday, far fewer security-minded people will be needed to keep organizations secure. This is an excellent counterpoint to my argument, though: even the people who supposedly specialize in keeping things secure can apparently be absolutely clueless.
- txcwpalpha 8y ago
  > It's almost inconceivable that they could let seventy nine certificates that are required for security monitoring expire for months on-end. Even worse, the "seventy nine" figure was just the certs that were associated with "critical" systems. The total number of expired certs was at least 324. > At the time of the breach, however, Equifax had allowed at least 324 of its SSL certificates to expire. Seventy-nine of the expired certificates were for devices monitoring highly business critical domains. (from pg 70 of the report)
- rconti 8y ago
  It's entirely possible they were monitoring cert validity, and that monitoring itself failed due to the expired certs :) We don't really know what the certs were used for (eg, whether it was needed to decrypt traffic), or just as part of the reporting... And yes, the team is responsible for keeping them valid, but... this isn't the first time that cert expiration "broke" something and left us worse off than with no SSL at all.
rcoveson 8y ago
Detecting data exfiltration has always seemed like an impossible problem to me. The implication is that "infiltration" has already occurred, and now the attacker just has to read/send the sensitive data somewhere. But if they have enough control to tell some system to dump its contents somewhere, how could they not have enough control to obfuscate the data itself? It seems like these systems could only work by either using some heuristic like data volume to decide when something is being "exfiltrated", which isn't nearly as useful, or by whitelisting allowed communications, which seems absurd. What am I missing?
- txcwpalpha 8y ago
  Ideally, the systems that prevent "infiltration" are separate from the systems that detect (and even prevent) "exfiltration". Just because you breached the firewall, that doesn't mean you can delete all the logs of network traffic passing through it. It's of course much more difficult than that in practice, but that's the general idea.
- viraptor 8y ago
  > But if they have enough control to tell some system to dump its contents somewhere This doesn't have to mean that they have full, unrestricted access. Access to data may mean that they could get raw records via some internal API. That API may still have logging, access control, DoS protection, etc. Just like you can detect and prevent an external person from scraping your website in most cases, you can detect some internal service requesting way more than an average hourly number of records. Or with a better audit - specific verifiable entity (signed employee requests, external request id markers, etc.) requesting more than they should. That assumes you have well designed system with multiple tiers and assume no trust. It's not possible if every service connects to the same database with yolo-root-access.
  - nickodell 8y ago
    I get the impression that they did connect with yolo root access. >Although the ACIS application required access to only three databases within the Equifax environment to perform its business function, the ACIS application was not segmented off from other, unrelated databases. As a result, the attackers used the application credentials to gain access to 48 unrelated databases outside of the ACIS environment.
im_cynical 8y ago
> I'm a little disappointed in the final "conclusion" of the report, though. The end of the executive summary basically chalks the breach up to two things: "Equifax's IT management structure was complicated" and "Equifax uses legacy software that is hard to secure". These are valid points, but these are also issues that nearly every single major corporation in the world faces, and yet many of them still manage to prevent (or at least mitigate) major breaches. These aren't good enough reasons to explain why Equifax failed so spectacularly compared to every other bureaucratic company with legacy software. Sure lots of companies can manage legacy software, arguably though Equifax's target on their head is substantially larger than most companies. They are the holy grail of personal data. Nothing should be legacy with them
- sorlon 8y ago
  Legacy systems are a red herring. It's just a way to shift blame to predecessors.
- rco8786 8y ago
  Everyone has legacy software. NASA has legacy software. But legacy isn’t an excuse to leave it unsecured or unmanaged.
  - dylan604 8y ago
    Maybe that's how Veeger was created. The Voyager software was clearly not kept up to date with the vendor's critical updates. When the satellite was found, the alien hackers laughed out how out of date it was, and how an old 0-day could still be used. The hardest part was to decide to make it into a bot in an botnet used for intergalactic DDoS attacks.
danShumway 8y ago
If the House's conclusion is that software of this complexity is impossible to secure, then it seems reasonable that we should treat any data stored in it as insecure. Maybe companies as complex as Equifax shouldn't have access to the data they have access to. I'm pretty tired of companies telling me that it's fine for them to hoover up extremely sensitive information like my social security number and then turning around after a breach and saying, "well, there was nothing we could do." It can't be both. If it's impossible to secure companies, then maybe Merriott shouldn't be asking for anybody's real name when they sign up for a hotel. Maybe we should stop using credit agencies for identity verification and start investing government resources into a separate 2-factor system. Maybe you should have a legally protected right to lie to businesses that ask for your personal information. Equifax leaked personal information for 50% of the US population. If you were voting, and there was a 50% chance that your ballet and voting history was going to be leaked publicly after the election, you would expect either: A) Someone is so incompetent that they're going to jail, or B) The system we're using is so fundamentally broken that we need to rethink the core paradigms of how it's built. To me, a report like this sounds like the House is saying that where corporate security is concerned, B is the answer.
- mpd 8y ago
  > as complex as Equifax I can only imagine that this point becomes lawsuits that span decades.
- rhizome 8y ago
  Someone is so incompetent that they're going to jail, or On this point, I can see no other explanation except that government and business are colluding against the people whose data is being leaked in order to normalize a lack of privacy. Token fines are handed out on the order of seconds worth of annual profit, sometimes minutes, but nobody's going to jail, certainly nobody in the boardroom or executive bathroom. Because fuck you that's why. Not only this, but we certainly aren't going to be able to sue as individuals over it, but why not? Why does personal data only have a value when someone sells it, why not at rest as well? These leaks contribute to financial criteria that are used to determine interest rates paid on loans, e.g., so a number value is calculable. I'm not holding my breath. Target should have had to close stores to close the shortfall from leak penalties, and Equifax and Marriott should have their corporate charters dissolved.
ryanmarsh 8y ago
"Equifax's IT management structure was complicated" This is like a loss of primary containment at a nuclear facility and the writeup saying "nuclear plants are hard". Color me flabbergasted.
- joering2 8y ago
  Devil's Advocate here But what were you expecting? That the people heavily sponsored by large corps like Equifax will write a nasty report against people that are basically paying their salaries? Have you ever seen 911 report? After all the investigations, time and money spent, the report basically stated that.. there was a terrorist attack and buildings fell. Long time ago I gave up on buying popcorn when high caliber political committees reports come out. Bottom line is, "this is America and this is business," and Equifax is in business of making billions of dollars, even if someone forgets to renew expired certificate, the caravan goes on.
sytelus 8y ago
I am just surprised why Equifax isn’t bankrupted. Even assuming nominal damage aggregated over next 10 years for class action lawsuit should bankrupted them. I feel corporate responsibility isn’t going to come by until these security breaches causes companies to go out of business.

codazoda 8y ago

Can someone explain why an expired certificate, on a monitoring device, would cause the device to completely fail rather than just spit out warnings about the certificate?

david_shaw 8y ago
My guess is that they were using private certificates to read encrypted data, without which they couldn't inspect their traffic.
- jacquesm 8y ago
  The exfiltration attempt would have likely been triggered on volume alone. Possibly it was simply a service that could not longer connect due to a lapsed cert trying to raise the alarm.
txcwpalpha 8y ago
Spitballing here based on my experience in the field but without any specific knowledge about Equifax's situation: Their monitoring system might have been using TLS to communicate the events to their aggregation tool, and when the cert expired, you don't really want log data with potentially confidential/critical security information traversing insecure channels, so they may have had it configured to not send any data if the cert wasn't valid. As for warnings about the cert, it's possible they (stupidly) configured it to not send warnings, or maybe it was sending warnings but nobody was paying attention. I've seen situations before where such warnings were set to go to XYZ person's mailbox, but XYZ person leaves the company and nobody remembered to update the destination address for the alerts.
viraptor 8y ago
(speculation) It could be that the expired certs were on the ingestion side. The monitoring agents tried to report something to the central place, but failed due to expired cert on the TLS connection. No other monitoring picked up that new data is not incoming.
ianmcgowan 8y ago
Uniformed speculation, but I imagine the monitoring device sends warnings and such to a central SIEM/log server (like splunk) for analysis and correlation across multiple devices. That channel is likely web server calls or json posts, over https.
evolvedlight 8y ago
Page 34: The default setting for this device allowed web traffic to continue through to the ACIS system, even when the SSL certificate was expired. When this occurs, traffic flowing to and from the internet is not analyzed by the intrusion detection or prevention systems because these security tools cannot analyze encrypted traffic.
sorlon 8y ago
Maybe the cert on the web server was updated, but the private key for the updated cert was never copied to the SSL Visibility Appliance?

david_shaw 8y ago

This is the first time I'd heard anything about expired certs, too. It's absolutely ridiculous that any kind of a security team -- not to mention one that's supposedly safeguarding such an immense amount of data -- could let monitoring certificates expire at all. It's almost inconceivable that they could let seventy nine certificates that are required for security monitoring expire for months on-end. I've worked in security for my whole career, and I'm firmly of the optimistic belief that someday, far fewer security-minded people will be needed to keep organizations secure. This is an excellent counterpoint to my argument, though: even the people who supposedly specialize in keeping things secure can apparently be absolutely clueless.

txcwpalpha 8y ago
> It's almost inconceivable that they could let seventy nine certificates that are required for security monitoring expire for months on-end. Even worse, the "seventy nine" figure was just the certs that were associated with "critical" systems. The total number of expired certs was at least 324. > At the time of the breach, however, Equifax had allowed at least 324 of its SSL certificates to expire. Seventy-nine of the expired certificates were for devices monitoring highly business critical domains. (from pg 70 of the report)
rconti 8y ago
It's entirely possible they were monitoring cert validity, and that monitoring itself failed due to the expired certs :) We don't really know what the certs were used for (eg, whether it was needed to decrypt traffic), or just as part of the reporting... And yes, the team is responsible for keeping them valid, but... this isn't the first time that cert expiration "broke" something and left us worse off than with no SSL at all.

rcoveson 8y ago

Detecting data exfiltration has always seemed like an impossible problem to me. The implication is that "infiltration" has already occurred, and now the attacker just has to read/send the sensitive data somewhere. But if they have enough control to tell some system to dump its contents somewhere, how could they not have enough control to obfuscate the data itself? It seems like these systems could only work by either using some heuristic like data volume to decide when something is being "exfiltrated", which isn't nearly as useful, or by whitelisting allowed communications, which seems absurd. What am I missing?

txcwpalpha 8y ago
Ideally, the systems that prevent "infiltration" are separate from the systems that detect (and even prevent) "exfiltration". Just because you breached the firewall, that doesn't mean you can delete all the logs of network traffic passing through it. It's of course much more difficult than that in practice, but that's the general idea.
viraptor 8y ago
> But if they have enough control to tell some system to dump its contents somewhere This doesn't have to mean that they have full, unrestricted access. Access to data may mean that they could get raw records via some internal API. That API may still have logging, access control, DoS protection, etc. Just like you can detect and prevent an external person from scraping your website in most cases, you can detect some internal service requesting way more than an average hourly number of records. Or with a better audit - specific verifiable entity (signed employee requests, external request id markers, etc.) requesting more than they should. That assumes you have well designed system with multiple tiers and assume no trust. It's not possible if every service connects to the same database with yolo-root-access.
- nickodell 8y ago
  I get the impression that they did connect with yolo root access. >Although the ACIS application required access to only three databases within the Equifax environment to perform its business function, the ACIS application was not segmented off from other, unrelated databases. As a result, the attackers used the application credentials to gain access to 48 unrelated databases outside of the ACIS environment.

im_cynical 8y ago

> I'm a little disappointed in the final "conclusion" of the report, though. The end of the executive summary basically chalks the breach up to two things: "Equifax's IT management structure was complicated" and "Equifax uses legacy software that is hard to secure". These are valid points, but these are also issues that nearly every single major corporation in the world faces, and yet many of them still manage to prevent (or at least mitigate) major breaches. These aren't good enough reasons to explain why Equifax failed so spectacularly compared to every other bureaucratic company with legacy software. Sure lots of companies can manage legacy software, arguably though Equifax's target on their head is substantially larger than most companies. They are the holy grail of personal data. Nothing should be legacy with them

sorlon 8y ago
Legacy systems are a red herring. It's just a way to shift blame to predecessors.
rco8786 8y ago
Everyone has legacy software. NASA has legacy software. But legacy isn’t an excuse to leave it unsecured or unmanaged.
- dylan604 8y ago
  Maybe that's how Veeger was created. The Voyager software was clearly not kept up to date with the vendor's critical updates. When the satellite was found, the alien hackers laughed out how out of date it was, and how an old 0-day could still be used. The hardest part was to decide to make it into a bot in an botnet used for intergalactic DDoS attacks.

danShumway 8y ago

If the House's conclusion is that software of this complexity is impossible to secure, then it seems reasonable that we should treat any data stored in it as insecure. Maybe companies as complex as Equifax shouldn't have access to the data they have access to. I'm pretty tired of companies telling me that it's fine for them to hoover up extremely sensitive information like my social security number and then turning around after a breach and saying, "well, there was nothing we could do." It can't be both. If it's impossible to secure companies, then maybe Merriott shouldn't be asking for anybody's real name when they sign up for a hotel. Maybe we should stop using credit agencies for identity verification and start investing government resources into a separate 2-factor system. Maybe you should have a legally protected right to lie to businesses that ask for your personal information. Equifax leaked personal information for 50% of the US population. If you were voting, and there was a 50% chance that your ballet and voting history was going to be leaked publicly after the election, you would expect either: A) Someone is so incompetent that they're going to jail, or B) The system we're using is so fundamentally broken that we need to rethink the core paradigms of how it's built. To me, a report like this sounds like the House is saying that where corporate security is concerned, B is the answer.

mpd 8y ago
> as complex as Equifax I can only imagine that this point becomes lawsuits that span decades.
rhizome 8y ago
Someone is so incompetent that they're going to jail, or On this point, I can see no other explanation except that government and business are colluding against the people whose data is being leaked in order to normalize a lack of privacy. Token fines are handed out on the order of seconds worth of annual profit, sometimes minutes, but nobody's going to jail, certainly nobody in the boardroom or executive bathroom. Because fuck you that's why. Not only this, but we certainly aren't going to be able to sue as individuals over it, but why not? Why does personal data only have a value when someone sells it, why not at rest as well? These leaks contribute to financial criteria that are used to determine interest rates paid on loans, e.g., so a number value is calculable. I'm not holding my breath. Target should have had to close stores to close the shortfall from leak penalties, and Equifax and Marriott should have their corporate charters dissolved.

ryanmarsh 8y ago

"Equifax's IT management structure was complicated" This is like a loss of primary containment at a nuclear facility and the writeup saying "nuclear plants are hard". Color me flabbergasted.

joering2 8y ago
Devil's Advocate here But what were you expecting? That the people heavily sponsored by large corps like Equifax will write a nasty report against people that are basically paying their salaries? Have you ever seen 911 report? After all the investigations, time and money spent, the report basically stated that.. there was a terrorist attack and buildings fell. Long time ago I gave up on buying popcorn when high caliber political committees reports come out. Bottom line is, "this is America and this is business," and Equifax is in business of making billions of dollars, even if someone forgets to renew expired certificate, the caravan goes on.

sytelus 8y ago

I am just surprised why Equifax isn’t bankrupted. Even assuming nominal damage aggregated over next 10 years for class action lawsuit should bankrupted them. I feel corporate responsibility isn’t going to come by until these security breaches causes companies to go out of business.