14 ms·

Equifax Releases Details on Cybersecurity Incident, Announces Personnel Changes

180 points by diggernet 9y ago

noncoml 9y ago
- In addition, credit card numbers for approximately 209,000 U.S. consumers, and certain dispute documents with personal identifying information for approximately 182,000 U.S. consumers, were accessed. This looks really bad. I don't know what's the point of letting them operate anymore. They utterly failed on the single thing they were supposed to do.
- QAPereo 9y ago
  I assume they're incredibly well connected, and lobby hard.
- harryh 9y ago
  The business they are in is supplying information to creditors about the credit-worthiness of potential customers. AFAIK they have succeeded at this. I don't want to downplay their security failure. It's obviously really bad, but saying that security was the "single thing they were supposed to do" is false.
  - noncoml 9y ago
    You are right, it was a hyperbole, but I still don’t see why they should be allowed to continue operating. Would you let a bank continue to operate after they have lost all their money? Never mind, I guess too big to fail applies here as well.
deleted 9y ago
[deleted]
yebyen 9y ago
Wow, MarketWatch is right. They are trying to erase Susan Mauldin (the former CISO), she is not even mentioned in this article. Does anyone with Google juice have a way to recover the interviews that are referenced in [1]? I watched them before they came down on Sept 10, and they were eye opening. I can say with certainty that the transcripts are not complete, because I remember "resistance to cloud is futile" and other such gems which are nowhere to be found in the partial transcripts that you can still find on the linked archive.is pages. [1]: https://hollywoodlanews.com/equifax-chief-security-officer/ https://hollywoodlanews.com/equifax-chief-security-officer/
- hn_throwaway_99 9y ago
  This article, http://www.marketwatch.com/story/equifax-ceo-hired-a-music-major-as-the-companys-chief-security-officer-2017-09-15 http://www.marketwatch.com/story/equifax-ceo-hired-a-music-m... , is brutally ridiculous and unfair. The major point seems to be "Hah, they hired a stupid music major to be their Chief Security Officer!" Some of the best software developers I've known were music majors. I know nothing about Susan Mauldin, including any of her other qualifications or lack thereof, but implying someone is only qualified to do what their college major was tells me this author is a complete idiot.
  - yebyen 9y ago
    That is not my purpose. I have no idea what her qualifications are, I saw the LinkedIn page and it contained almost no information. That is hardly relevant. I personally believe that she was backed into a corner, based on the interviews I watched from some time before the breach (that I can't show you, and I won't link to the transcripts because I know they are incomplete and by far less impactful.) She sounded to me, exactly like a person who was given a budget that was effectively no budget, and then put into this role because people with a larger stake were sure that she would comply when they said "this is your budget, and not a penny more." I know exactly what that is like, because I have been made CIO and put in that position before. (I'm sure she was better paid than I was...) There is absolutely 100% a coverup going on. I am so bummed that I did not save the interviews when I first watched them. You would agree too, if I could show you. Edit: Here is the transcript, but with the disclaimer that it is 100% not the full transcript. Although it is in her own words. Make your own judgement about her qualifications please. https://archive.is/6M8mg https://archive.is/6M8mg
    - deleted 9y ago
      [deleted]
    - yebyen 9y ago
      "I always say that given enough time, we can secure anything and find a way to say yes to it and business-driven CISOs are of that mind" It's really a pretty balanced interview. This is what she said. There was a second interview, but I'm a lot more incensed that this information is being suppressed than at any one thing she says. I want to be furious when she says things like "If a CISO can come up with a list of controls that he/she is comfortable with, then by and large the evidence proves that those controls are working effectively and are going to satisfy the elements of any framework that you use." but all this interview really can show is that she knows some jargon, and her mindset. And I think it's really (truly, not in a figurative sense but literally) criminal that this information is suppressed. This is the case study of an interview that reveals insights from the mind of a CISO before disaster strikes... I literally don't even care that she was a Music major, how can anyone justify taking this down? It's not just potentially criminal, it's also unconscionable. This should be preserved for posterity, I want to tar and feather the company, but I want to hear more from Susan Mauldin about what went wrong at Equifax. I don't just want this interview back online, I want there to be a follow-up to this interview! And if it takes a pardon from Trump to make that happen, let's start the conversation.
simpfai 9y ago
Failure to apply a patch for a two month old bug led to this entire nightmare scenario. What are some best practices to ensure that ones dependencies are always up to date? -asking as a relatively inexperienced dev
- pishpash 9y ago
  Have a process to know what all your shit is and be paranoid about using any shit. Being paranoid helps.
  - seandougall 9y ago
    Although I'm sure Equifax is not going to be very forthcoming about this aspect of it, not having plaintext passwords visible after logging in as admin/admin also helps.
- godzillabrennus 9y ago
  Applying patches when they come out...
  - deleted 9y ago
    [deleted]
    - la6470 9y ago
      They should provide free credit monitoring for LIFE for affected individuals and not just one year...the individuals have been affected for life because now their SSN numbers are going to be out in the wild FOR EVER!!!
- tommynicholas 9y ago
  Equifax makes their partners have a fully implemented and tested patch management program and audits annually (or via a third party) that you stick to it, making this situation even more hilarious.
- ccrush 9y ago
azurezyq 9y ago
If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being protected by a single layer of security. Which means, any one from the inside can access pretty much the info as the hacker, which is horrible. For example, are the items in DB encrypted? Are database backups encrypted? Are different items encrypted using different keys? I don't think EFX did it right.
- sillysaurus3 9y ago
  If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being protected by a single layer of security. Which means, any one from the inside can access pretty much the info as the hacker, which is horrible. This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals. It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau.
  - azurezyq 9y ago
    I think it is covered by compliance, e.g., https://www.google.com/search?q=pci+dss+encryption+at+rest&oq=pci+dss+encryption&aqs=chrome.1.69i57j0l5.8302j0j4&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=pci+dss+encryption+at+rest&o... Batch processing on encrypted data is absolutbly ok. For example, you can just pull row-level keys and run your audited and signed binary. Also the pipeline should be put away from the frontend. I'm not an expert but I think it should already be documented in the compliance doc.
    - 9y ago
21 9y ago
What about storing sensitive data in something like a HSM, which rate-limits access rate, so you could only lets say access 10000 records per day. Yes, developing against such a system might be annoying (think about updating a new piece of data for all records). But it feels to me that we need a way to rate-limit access to sensitive data, to prevent wholesale dump in a short time. But you still need other systems in place, to prevent a hacker lurking around for months until it gets all the data.
- dannyw 9y ago
  Or, you know, monitoring systems. "Hey... Splunk tells me there's been a 34x increase in record queries. Page page"
guelo 9y ago
The CVE and the patch came out on March 7. Exploits were already being detected in the wild at the time by perimeter security vendors. See http://blog.talosintelligence.com/2017/03/apache-0-day-exploited.html http://blog.talosintelligence.com/2017/03/apache-0-day-explo... Equifax "believes" that the hackers got in on May 13. They had some kind of intrusion detection system that finally detected the intrusion on July 29. 5 months after the "Critical" CVE alert went out. During that time security vendors were adding firewall rules to stop the attack. But apparently Equifax didn't have any other security in front of the Struts server. That just seems like unconscionable incompetence and malpractice for such a high value target.
- dannyw 9y ago
  This should be negligence. Perhaps not criminal negligence, but should be sue-able c
- Flammy 9y ago
  > They had some kind of intrusion detection system that finally detected the intrusion on July 29. If they had some sort of intrusion detection system, I gotta think they would have picked up on this in days, not more than 2 months later. I feel like this is one of those situations where a dev is saying something like "where is all of the CPU on this database going?" or "Why is the network connection so slow?" and digs into it and finds some strange behavior...
EAPressley1968 9y ago
Companies need to stop making CIO & CISO positions diversity quota hires. The CIO during the Target breach was originally a department store buyer.
la6470 9y ago
They should provide free credit monitoring for LIFE for affected individuals and not just one year...the individuals have been affected for life because now their SSN numbers are going to be out in the wild FOR EVER!!!!
niij 9y ago
They were able to use a subdomain for their announcement, but not for their Breach service which asks for your SSN.
blacksqr 9y ago
Gotta admire the artful way they gave the appearance of disclosure while avoiding answering the most damning question: why did it take so long for them to patch Struts? "The particular vulnerability in Apache Struts was identified and disclosed by U.S. CERT in early March 2017." "Equifax's Security organization was aware of this vulnerability at that time, and took efforts to identify and to patch any vulnerable systems in the company's IT infrastructure." ?????????!!!!!!?????? "While Equifax fully understands the intense focus on patching efforts, the company's review of the facts is still ongoing."
- freeone3000 9y ago
  It's struts, and the CVE fix was pushed to a newer version than the one they were using. That's an intense app rebuild, akin to moving mongo versions.
  - krirken 9y ago
    > That's an intense app rebuild That is not even close to an excuse. A remote code execution vulnerability has the potential to destroy your whole company.
    - dannyw 9y ago
      I hope Equifax will be learning from this, but can you tell your CEO that your core business must be shut down for 3 weeks as you upgrade and rebuild the system?
      - krirken 9y ago
        Yes, the risk is much higher than the cost. From the article: > The company's internal review of the incident continued. Upon discovering a vulnerability in the Apache Struts web application framework as the initial attack vector, Equifax patched the affected web application before bringing it back online. That bullet point lies between the "July 30th" and "August 2nd" bullet points. Based on that timeline, the vulnerability took days to patch.
dingo_bat 9y ago
Do they plan to change their Chief Security Officer who is a MA in Music Composition?
rmrfrmrf 9y ago
Actually looking at the CVE made my stomach drop. That is a horrible horrible bug. Getting access to the shell while under the web app user's environment potentially means that all secrets were available either in the app server administration, the user environment, or in a readable file on the system. Effing yikes. Also knowing how JNDI usually gets configured on app servers (sometimes with credentials and all) would have made recon ridiculously easy. I think some have alluded to it, but what some people in the comments don't seem to understand is that RCE is an "all bets are off" type of situation. DEFCON 1 to be sure. Prevention is really the only good answer.
qaq 9y ago
For everyone patting themselves on the back for how much better they are at securing their data realize that for way over 80% of incidents the attack vector is email and social engineering. Look at Red team exercises of competent teams and try to honestly answer the question would have they succeed with that tactic at your company. So yes having much better practices vs what we see here is very important but will not really help much if you are being targeted by a competent adversary.
heisenbit 9y ago
They are really in deep trouble and need to manage the story carefully. If people on a broad basis start freezing their credit history (one of the most effective means to protect against abuse) their and their competitors ability to sell data to other parties will suffer (see also https://wolfstreet.com/2017/09/15/equifax-sacks-2-executives-gets-devious-to-stop-you-from-getting-a-credit-freeze/ https://wolfstreet.com/2017/09/15/equifax-sacks-2-executives... )
leesalminen 9y ago
I just had a scamming debt collector call me about a debt I already paid off to another collection company. They had all the right information. I suspect it has to do with the hack. Watch out for First Equity Alliance.
hyperbole 9y ago
Having a remote execution exploit shouldn't mean keys to the kingdom. I find it hard to believe that this company whose whole business is electronic didn't adapt it's technology stack to remedy this type of attack limiting the scope of a data leak. I wouldn't be surprised if struts has another exploit of similar magnitude, what then? They might as well be running their business on a cluster of tomcat servers sitting atop sqlite. Hopefully they don't recover from this - they should not have the data they posses if they cannot mitigate risk.

180 points by diggernet 9y ago

- In addition, credit card numbers for approximately 209,000 U.S. consumers, and certain dispute documents with personal identifying information for approximately 182,000 U.S. consumers, were accessed. This looks really bad. I don't know what's the point of letting them operate anymore. They utterly failed on the single thing they were supposed to do.

QAPereo 9y ago
I assume they're incredibly well connected, and lobby hard.
harryh 9y ago
The business they are in is supplying information to creditors about the credit-worthiness of potential customers. AFAIK they have succeeded at this. I don't want to downplay their security failure. It's obviously really bad, but saying that security was the "single thing they were supposed to do" is false.
- noncoml 9y ago
  You are right, it was a hyperbole, but I still don’t see why they should be allowed to continue operating. Would you let a bank continue to operate after they have lost all their money? Never mind, I guess too big to fail applies here as well.

deleted 9y ago

[deleted]

yebyen 9y ago

Wow, MarketWatch is right. They are trying to erase Susan Mauldin (the former CISO), she is not even mentioned in this article. Does anyone with Google juice have a way to recover the interviews that are referenced in [1]? I watched them before they came down on Sept 10, and they were eye opening. I can say with certainty that the transcripts are not complete, because I remember "resistance to cloud is futile" and other such gems which are nowhere to be found in the partial transcripts that you can still find on the linked archive.is pages. [1]: https://hollywoodlanews.com/equifax-chief-security-officer/ https://hollywoodlanews.com/equifax-chief-security-officer/

hn_throwaway_99 9y ago
This article, http://www.marketwatch.com/story/equifax-ceo-hired-a-music-major-as-the-companys-chief-security-officer-2017-09-15 http://www.marketwatch.com/story/equifax-ceo-hired-a-music-m... , is brutally ridiculous and unfair. The major point seems to be "Hah, they hired a stupid music major to be their Chief Security Officer!" Some of the best software developers I've known were music majors. I know nothing about Susan Mauldin, including any of her other qualifications or lack thereof, but implying someone is only qualified to do what their college major was tells me this author is a complete idiot.
- yebyen 9y ago
  That is not my purpose. I have no idea what her qualifications are, I saw the LinkedIn page and it contained almost no information. That is hardly relevant. I personally believe that she was backed into a corner, based on the interviews I watched from some time before the breach (that I can't show you, and I won't link to the transcripts because I know they are incomplete and by far less impactful.) She sounded to me, exactly like a person who was given a budget that was effectively no budget, and then put into this role because people with a larger stake were sure that she would comply when they said "this is your budget, and not a penny more." I know exactly what that is like, because I have been made CIO and put in that position before. (I'm sure she was better paid than I was...) There is absolutely 100% a coverup going on. I am so bummed that I did not save the interviews when I first watched them. You would agree too, if I could show you. Edit: Here is the transcript, but with the disclaimer that it is 100% not the full transcript. Although it is in her own words. Make your own judgement about her qualifications please. https://archive.is/6M8mg https://archive.is/6M8mg
  - deleted 9y ago
    [deleted]
  - yebyen 9y ago
    "I always say that given enough time, we can secure anything and find a way to say yes to it and business-driven CISOs are of that mind" It's really a pretty balanced interview. This is what she said. There was a second interview, but I'm a lot more incensed that this information is being suppressed than at any one thing she says. I want to be furious when she says things like "If a CISO can come up with a list of controls that he/she is comfortable with, then by and large the evidence proves that those controls are working effectively and are going to satisfy the elements of any framework that you use." but all this interview really can show is that she knows some jargon, and her mindset. And I think it's really (truly, not in a figurative sense but literally) criminal that this information is suppressed. This is the case study of an interview that reveals insights from the mind of a CISO before disaster strikes... I literally don't even care that she was a Music major, how can anyone justify taking this down? It's not just potentially criminal, it's also unconscionable. This should be preserved for posterity, I want to tar and feather the company, but I want to hear more from Susan Mauldin about what went wrong at Equifax. I don't just want this interview back online, I want there to be a follow-up to this interview! And if it takes a pardon from Trump to make that happen, let's start the conversation.

simpfai 9y ago

Failure to apply a patch for a two month old bug led to this entire nightmare scenario. What are some best practices to ensure that ones dependencies are always up to date? -asking as a relatively inexperienced dev

pishpash 9y ago
Have a process to know what all your shit is and be paranoid about using any shit. Being paranoid helps.
- seandougall 9y ago
  Although I'm sure Equifax is not going to be very forthcoming about this aspect of it, not having plaintext passwords visible after logging in as admin/admin also helps.
godzillabrennus 9y ago
Applying patches when they come out...
- deleted 9y ago
  [deleted]
  - la6470 9y ago
    They should provide free credit monitoring for LIFE for affected individuals and not just one year...the individuals have been affected for life because now their SSN numbers are going to be out in the wild FOR EVER!!!
tommynicholas 9y ago
Equifax makes their partners have a fully implemented and tested patch management program and audits annually (or via a third party) that you stick to it, making this situation even more hilarious.
ccrush 9y ago

azurezyq 9y ago

If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being protected by a single layer of security. Which means, any one from the inside can access pretty much the info as the hacker, which is horrible. For example, are the items in DB encrypted? Are database backups encrypted? Are different items encrypted using different keys? I don't think EFX did it right.

sillysaurus3 9y ago
If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being protected by a single layer of security. Which means, any one from the inside can access pretty much the info as the hacker, which is horrible. This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals. It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau.
- azurezyq 9y ago
  I think it is covered by compliance, e.g., https://www.google.com/search?q=pci+dss+encryption+at+rest&oq=pci+dss+encryption&aqs=chrome.1.69i57j0l5.8302j0j4&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=pci+dss+encryption+at+rest&o... Batch processing on encrypted data is absolutbly ok. For example, you can just pull row-level keys and run your audited and signed binary. Also the pipeline should be put away from the frontend. I'm not an expert but I think it should already be documented in the compliance doc.
  - 9y ago

21 9y ago

What about storing sensitive data in something like a HSM, which rate-limits access rate, so you could only lets say access 10000 records per day. Yes, developing against such a system might be annoying (think about updating a new piece of data for all records). But it feels to me that we need a way to rate-limit access to sensitive data, to prevent wholesale dump in a short time. But you still need other systems in place, to prevent a hacker lurking around for months until it gets all the data.

dannyw 9y ago
Or, you know, monitoring systems. "Hey... Splunk tells me there's been a 34x increase in record queries. Page page"

guelo 9y ago

The CVE and the patch came out on March 7. Exploits were already being detected in the wild at the time by perimeter security vendors. See http://blog.talosintelligence.com/2017/03/apache-0-day-exploited.html http://blog.talosintelligence.com/2017/03/apache-0-day-explo... Equifax "believes" that the hackers got in on May 13. They had some kind of intrusion detection system that finally detected the intrusion on July 29. 5 months after the "Critical" CVE alert went out. During that time security vendors were adding firewall rules to stop the attack. But apparently Equifax didn't have any other security in front of the Struts server. That just seems like unconscionable incompetence and malpractice for such a high value target.

dannyw 9y ago
This should be negligence. Perhaps not criminal negligence, but should be sue-able c
Flammy 9y ago
> They had some kind of intrusion detection system that finally detected the intrusion on July 29. If they had some sort of intrusion detection system, I gotta think they would have picked up on this in days, not more than 2 months later. I feel like this is one of those situations where a dev is saying something like "where is all of the CPU on this database going?" or "Why is the network connection so slow?" and digs into it and finds some strange behavior...

EAPressley1968 9y ago

Companies need to stop making CIO & CISO positions diversity quota hires. The CIO during the Target breach was originally a department store buyer.

la6470 9y ago

They should provide free credit monitoring for LIFE for affected individuals and not just one year...the individuals have been affected for life because now their SSN numbers are going to be out in the wild FOR EVER!!!!

niij 9y ago

They were able to use a subdomain for their announcement, but not for their Breach service which asks for your SSN.

blacksqr 9y ago

Gotta admire the artful way they gave the appearance of disclosure while avoiding answering the most damning question: why did it take so long for them to patch Struts? "The particular vulnerability in Apache Struts was identified and disclosed by U.S. CERT in early March 2017." "Equifax's Security organization was aware of this vulnerability at that time, and took efforts to identify and to patch any vulnerable systems in the company's IT infrastructure." ?????????!!!!!!?????? "While Equifax fully understands the intense focus on patching efforts, the company's review of the facts is still ongoing."

freeone3000 9y ago
It's struts, and the CVE fix was pushed to a newer version than the one they were using. That's an intense app rebuild, akin to moving mongo versions.
- krirken 9y ago
  > That's an intense app rebuild That is not even close to an excuse. A remote code execution vulnerability has the potential to destroy your whole company.
  - dannyw 9y ago
    I hope Equifax will be learning from this, but can you tell your CEO that your core business must be shut down for 3 weeks as you upgrade and rebuild the system?
    - krirken 9y ago
      Yes, the risk is much higher than the cost. From the article: > The company's internal review of the incident continued. Upon discovering a vulnerability in the Apache Struts web application framework as the initial attack vector, Equifax patched the affected web application before bringing it back online. That bullet point lies between the "July 30th" and "August 2nd" bullet points. Based on that timeline, the vulnerability took days to patch.

dingo_bat 9y ago

Do they plan to change their Chief Security Officer who is a MA in Music Composition?

rmrfrmrf 9y ago

Actually looking at the CVE made my stomach drop. That is a horrible horrible bug. Getting access to the shell while under the web app user's environment potentially means that all secrets were available either in the app server administration, the user environment, or in a readable file on the system. Effing yikes. Also knowing how JNDI usually gets configured on app servers (sometimes with credentials and all) would have made recon ridiculously easy. I think some have alluded to it, but what some people in the comments don't seem to understand is that RCE is an "all bets are off" type of situation. DEFCON 1 to be sure. Prevention is really the only good answer.

qaq 9y ago

For everyone patting themselves on the back for how much better they are at securing their data realize that for way over 80% of incidents the attack vector is email and social engineering. Look at Red team exercises of competent teams and try to honestly answer the question would have they succeed with that tactic at your company. So yes having much better practices vs what we see here is very important but will not really help much if you are being targeted by a competent adversary.

heisenbit 9y ago

They are really in deep trouble and need to manage the story carefully. If people on a broad basis start freezing their credit history (one of the most effective means to protect against abuse) their and their competitors ability to sell data to other parties will suffer (see also https://wolfstreet.com/2017/09/15/equifax-sacks-2-executives-gets-devious-to-stop-you-from-getting-a-credit-freeze/ https://wolfstreet.com/2017/09/15/equifax-sacks-2-executives... )

leesalminen 9y ago

I just had a scamming debt collector call me about a debt I already paid off to another collection company. They had all the right information. I suspect it has to do with the hack. Watch out for First Equity Alliance.

hyperbole 9y ago

Having a remote execution exploit shouldn't mean keys to the kingdom. I find it hard to believe that this company whose whole business is electronic didn't adapt it's technology stack to remedy this type of attack limiting the scope of a data leak. I wouldn't be surprised if struts has another exploit of similar magnitude, what then? They might as well be running their business on a cluster of tomcat servers sitting atop sqlite. Hopefully they don't recover from this - they should not have the data they posses if they cannot mitigate risk.