12 ms·

If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being

sillysaurus3 9y ago
If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being protected by a single layer of security. Which means, any one from the inside can access pretty much the info as the hacker, which is horrible. This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals. It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau.
- azurezyq 9y ago
  I think it is covered by compliance, e.g., https://www.google.com/search?q=pci+dss+encryption+at+rest&oq=pci+dss+encryption&aqs=chrome.1.69i57j0l5.8302j0j4&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=pci+dss+encryption+at+rest&o... Batch processing on encrypted data is absolutbly ok. For example, you can just pull row-level keys and run your audited and signed binary. Also the pipeline should be put away from the frontend. I'm not an expert but I think it should already be documented in the compliance doc.
  - sillysaurus3 9y ago
    If you can batch process the dataset, an attacker can do the same thing. They take whatever creds the batch script is using and use them directly. If the program only runs from a certain box, then they run their modified script on that box. "Well don't let that happen!" Nope, not that easy. This threat model concedes that the hackers have RCE. They're on your network. That's why they can do this.
    - azurezyq 9y ago
      "All binaries are audited and signed.", which makes it trackable. And the runtime environment is also isolated and uses different set of access permissions. I'm not saying it is impossible. Just layers and layers. For internal stuffs, Google's beyondcorp is a good start.
      - sillysaurus3 9y ago
        There is no such thing as a runtime environment that (a) attackers gained RCE into, and (b) the box with the sensitive data can verify the incoming connection is from an audited, signed binary. Simply put: if you have RCE into an app, then that breaks all the guarantees of that audited, signed binary. You can make it sing and dance and ask for a thousand rows instead of ten; you can do whatever you want to it. The modifications happen in memory, not on disk, so even if they don't persist you can still modify the app. Monitoring seems like the only solution.
    - kelnos 9y ago
      Why would you ever allow public web access to a machine that can do that level of batch processing? That sort of thing should be accessible only through internal-only tools protected by a VPN.
      - sillysaurus3 9y ago
        You'd be amazed at how easy it is to exploit internal tools once you're inside someone's network. Internal tools were the juiciest targets: a perfect storm of "created by an unqualified team that no longer works here" and "nobody realized it's still been running for a year." It can work, but evidence seems to imply this strategy makes the situation worse. I'm not sure what to do with this information other than to relay the experience. It's one of those tricky counterintuitive facts.
- lowbloodsugar 9y ago
  >It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How do you read linux firewall documentation with the first example being old school public->DMZ->private? So your web server got hacked? That means what? Your database is across a single port through the firewall, and ACLs mean that any traffic arriving at the part from the DMZ CIDR can only do the limited set of queries that provide single-person-at-time. SSH from hacked box to database? Got a zero-day for the firewall (because 22 isn't open from there), and a zero-day for sshd (because its keys only). I know this and I'm not one of our hundreds of security people. (And as I'm not the security guy, if any actual security guys want to school me here, please do so!) Sure, I agree that business peeps will say "Oh that's too expensive I don't care", but lets not pretend that we don't know how to do this. Be clear, if you are the IT security guy, and you agree to bow down to "business" people (you're a business person) who tell you "No, that's too complicated", then just be clear that you are getting fired when the shit hits the fan.
  - sillysaurus3 9y ago
    Yep. You can do all that. At the end of it, a box somewhere needs to be able to access the data. That box has access. You handwave it as single-person-at-a-time queries. But on any given day, they have to run tens or hundreds of thousands of those queries. Even if people aren't accessing their credit info, there is plenty of automation that does. Meaning if you pop that single person query box, you can scale up the queries by a factor of ten without raising any flags directly. It's much better, certainly. Companies should do that. But it's not obviously a panacea.
    - lowbloodsugar 9y ago
      Conceivably, the attackers spent the 75 days downloading the 143,000,000 records, 2,000,000 records per day, and managed to trigger no scaling alarms as a result. Or perhaps there were no alarms. I wonder if we'll ever know.
      - sova 9y ago
- mi100hael 9y ago
  Things one could do that would increase security without hindering business goals: - PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. This is an architectural decision! If you cannot put out a hotfix within months of a vulnerability being made public, YOUR APPLICATION & DEPLOYMENT DESIGNS ARE BROKEN. - WAF rules for common attacks and known CVEs - Least privilege enforced architecturally via stored procedures or microservices or something else so that a vulnerability in one application (the online dispute portal in this case) can't lead to a dump of everyone's PII. - Data should be encrypted and encryption keys should be stored apart from both the application and the data. Upon startup, the application should make an authenticated request to read the encryption keys into memory. This significantly raises the bar to dumping sensitive data. - Monitoring for connections transferring anomalous amounts of data That's just off the top of my head. All of these can be designed into a system and still maintain 100% uptime.
  - wavegeek 9y ago
    > PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. I worked in system admin for many years. If anyone was actually foolish enough to do this, they would be dealing with multiple major outages every year. > for every complex problem there is a solution that is simple, obvious, and wrong.
    - georgebarnett 9y ago
      "There might be an outage" is the stupidest reason I have ever heard to advocate not applying security patches. There's plenty of companies that are able to regularly patch and upgrade software, incident free. If you team isn't able to do this, they need to fix their process instead of putting their head in the sand.
      - wavegeek 9y ago
        > stupid I am not defending the totality of EFX's process by any means. I am just pointing out that "install every high severity patch right away" is not as easy as it looks. Others have pointed out this patch was not available for all current versions, for example. So, to install the patch you need to upgrade. Oh, that breaks <dependency>. So we need to upgrade <other thing>. But that regresses a feature we use .... The major issue with EFX was the fragility of their overall architecture. A single weakness should not be enough to lay the whole DBMS open.
- btmorex 9y ago
  Put machines with sensitive data on a private network. Have one machine/one set of machines connected to both the private network and public network that proxy requests to the data machines. Proxy by parsing original request and then rewriting (i.e. don't just pass along the request). Rate limit and implement notifications on data machines so that if the proxies are compromised, the attacker can't easily do a full data dump. Don't allow remote administration only physical (this rules out this architecture for most companies, but honestly, not a company like equifax with incredibly sensitive info and where network locality is not a big issue).
  - btown 9y ago
    To the remote administration point, if you have this architecture and still open up the private data-network to remote SSH connections, you're at least limiting the attack space to vulnerabilities in SSH and the network stack/firewall on your ingress server, but not vulnerabilities in the application server. Much better than nothing.
  - rmrfrmrf 9y ago
    If you look at the CVE, it's literally plain shell commands as strings in the Content-* headers. These commands would appear to be local with whatever user the app is running as.
    - kelnos 9y ago
      And that's the ridiculous thing: this is actually a fairly easy thing to protect against. Why the hell did the webapp servers have direct access to the database? If there was a service in the middle mediating requests, that would have been another line of defense. Sure, maybe _that_ service might have holes, but it'd be another hurdle the attacker would have to clear.
      - rmrfrmrf 9y ago
        As long as JNDI is a best practice in Java land, I'd say that the majority of Java app servers are directly connected to databases.
- IncRnd 9y ago
  "How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB." Simple. Don't store a key with the data. A credit bureau only incurs liability by knowing the PII for hundreds of millions of people readable from one location - that they provide to everyone who asks! How does a company create a better architecture? Separation of duties and responsibilities, defense in depth, authentication and authorization, cryptography, updating software, etc. Hire a security team to help with vulnerability analysis, design reviews, due diligence, and threat modeling. This gets solved every day at many companies. "You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau." Yes, but the elephant in the room is that a credit bureau possesses that information in usable form. This can be taken care of by key management, where the individual owns the key and only one key is able to exist per individual at a time. Then, only those credit grantors who have been authorized by the individual are able to update or view the data. In cryptography the operating assumption should always be that everything is known about your data except your key material, which is kept private.
  - sillysaurus3 9y ago
    Yes, but the elephant in the room is that a credit bureau possesses that information in usable form. This can be taken care of by key management, where the individual owns the key and only one key is able to exist per individual at a time. Then, only those credit grantors who have been authorized by the individual are able to update or view the data. This is what I mean by not meeting business goals. It is clearly a bad idea to give Grandma an irreplaceable key. But that's what this proposal is.
    - IncRnd 9y ago
      Wrong. It is your interpretation that this is an irreplaceable key. You are trying to say that a design is wrong, because you believe that your implementation would be wrong. Uniqueness is a different property from replaceability. And, replaceability is not necessarily the correct property to design for, key rotation and recovery from loss are the properties. What you want is the ability to rotate keys and recover from the loss of keys.
- mulmen 9y ago
  Here's a secure architecture for you: Don't ever store multiple pieces of personally identifiable information in the same place. The consequence of this is that Equifax's business model is no longer viable. I care about that as much as they care about me and my data.
- js2 9y ago
  In multiple threads now I've seen you express this sentiment. I just don't buy it. Vanguard hasn't had a leak like this. USAA hasn't had a leak like this. ETrade, Fidelity, etc, no leaks like this of which I'm aware (correct me if I'm wrong). These are big plodding institutions too. No leaks like this from Google nor Facebook nor Amazon. Maybe you're arguing it's just a matter of time, but I disagreee. Equifax had a trivially easy architecture to break. Minimally, the data should be on a separate set of servers protected by an API with rate limiting, and that API should be implemented on top of a different framework than the front-end servers to minimize the chance that a single flaw would open up the entire set of data. This isn't rocket science. Equifax is a multi-billion dollar company. They could and should have done better. Please stop defending them? Disclosure: I work for Yahoo (er, Verizon) since late 2013. I don't know the details of Yahoo's data leaks, but it was not as trivial as getting RCE on Yahoo's public facing servers.
  - sillysaurus3 9y ago
    When two people are clearly intelligent but disagree, the reason is almost always a different set of experiences. My experience differs from yours; if I could Vulcan my brain to yours, you'd see why I feel this way. You might still disagree, but it would be far less clear cut. Now, I've seen nearly a hundred codebases and they all looked pretty much like Equifax. I don't even need to see Equifax to know the kind of disaster their codebase facilitated. Equifax's codebase is your code, and my code, and Bob's code. The only way to make this situation better is to talk openly about it. And the best way to do that is to actively defend the unpopular side in order to call attention to uncomfortable truths. It's a calculated strategy. Just because you worked at a place with good security hygiene, it makes no difference to the world at large. Everybody else looks like Equifax. We need to (a) acknowledge that and (b) actively seek out solutions. That's what's happening here. Ask yourself: "Why do I disagree when I've seen one or two enterprise codebases?" Those big, plodding institutions you mention are similarly vulnerable. Everyone is. Why aren't data breaches happening almost daily? Because it's illegal. It's no coincidence that companies with public bug bounties are among the hardest targets to hit. When you make it not-illegal to attack you, surprise: you find the vulns. A good question is, "is it illegal for a white hat to attack my business?" If the answer is yes, seek to change that. It's a low effort thing that will increase your security posture. But it's no substitute for $40k pentests.
- qyv 9y ago
  > This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals. No, it's not how almost every company works, and yes it IS horrible. Do you know why people stop short of giving details of what should have been done? Because that is a full time job for a entire team of people in any place that needs real security, and part of that job is to be constantly rooting out flaws in both design and implementation and fixing them. And guess what, if your business involves storing and retrieving sensitive personal information on millions of people, then security needs to be one of your business goals. If it is not, then that is negligent. > It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? It is also easy to handwave away the real work that real security professionals do. Security is not a just some module you can strap into your systems, it is an ongoing process that requires vigilance and competence. Yes it is hard, but being hard is not an excuse > Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau. So you are implying that data encryption is not useful for protecting sensitive data because you have to store the keys somewhere. If that were the case then data encryption in general would be useless, we know that is not the case. This is a design or implementation problem, not an encryption problem.
  - x0x0 9y ago
    It's frustrating how people seem not to think it's wildly, ludicrously incompetent for nothing in your architecture to complain about your front end servers querying 140m records. The fact they don't have even the most basic data access controls in place make me assume they're still fully owned. You're playing IT if you can't deploy a CVE fix in 60 days. Finally, they only did this because of greed. They encourage online disputes because it saves them money and gives them greater rights vis-a-vis paper disputes delivered to them via snail mail. They could simply have taken the service down if if they couldn't deploy a security patch in two months. Amateur hour everywhere. ps -- the data controls I mentioned above were implemented in a company of ~15 eng dealing with similar PII. If you owned front end servers you could query individual records, but you would have to own more services to be able to "\copy (select * from data) to /legit.file; gzip /legit.file; scp /legit.file.gz $MY_SERVER". And if you tried pagers would have gone off all over the company.
- cmiles74 9y ago
  I disagree with the fatalist tone. Sure, it can be hard for big, slow moving businesses to take a time out from writing million dollar checks to C-suite executives to spend money on a competent operations and security team. But it can happen and in this case it should have happened. I don't think "better architecture" is handwavy. If you have old legacy apps that can't be updated and maintained, don't provide them with a path to the public internet. Or front then with some infrastructure and adequate alarms to detect something going wrong. I think it's reasonable to respond to a CVE with an update to your app or infrastructure within one week. Most of the time rapid testing will be fine as your patching a library or component, like Struts or Tomcat. These companies are hardly providing 100% uptime, in the case where an update like this needs a rollback, it won't be a big deal. If you have a good team and you spend the money on them.
- gloverkcn 9y ago
  This is a solved problem. Companies that store personal information at that level should be required to implement PCI-DSS level security. This includes going through the auditing process. Here's a quick overview. https://www.pcisecuritystandards.org/documents/PCI%20SSC%20Quick%20Reference%20Guide.pdf https://www.pcisecuritystandards.org/documents/PCI%20SSC%20Q... You can work to not implement the security standards, and then try to fake your way through the audit. At that point you are not ignorant of proper security, you are actively endangering the users, and your right to hold PII (Personally Identifiable Information) should be revoked.
- avip 9y ago
  I don't buy your observation that all big-corps are kind of equally sloppy security-wise. Here are 2 examples: - Equifax login form does not use STS header. - equifax.com/business exposes full server version (Server:Apache/2.4.27 (Win64) OpenSSL/1.1.0f mod_log_rotate/1.0.0) I'm not a security person but those 2 seems way out of industry standards for security-aware web services.
patcheudor 9y ago
"the system is architecturally wrongly designed" Like: https://www.fastcompany.com/40468811/heres-why-equifax-yanked-its-apps-from-apple-and-google-last-week https://www.fastcompany.com/40468811/heres-why-equifax-yanke...
- zaroth 9y ago
  I was sure the vulnerability was going to be an API key hard coded in the app which allowed retrieving any credit report over the API. That the app was "merely" sending some calls unencrypted was a total let down.
  - patcheudor 9y ago
    API creds maybe could have been excusable as a mistake which got overlooked or even permissions which changed. In the context of a credit monitoring app, the use of HTTP is really, really, really bad and can't really be excused. It would be interesting to know if there are more issues, but a lot of white-hat researchers stay away from such things when there's no official bug bounty program because of the computer fraud and abuse act. Interacting in the API outside its intended use by the app could be considered computer intrusion and it might be advisable to stay away from legal grey areas right now when it comes to Equifax research.
outoftacos 9y ago
Yeah they tried to throw Struts under the bus but actually just confessed to criminal negligence. I don't think they are competent enough to realize how damning this press release is. This company is finished.
- Exuma 9y ago
  I hope what you say is true. I am so livid after reading this.
  - sillysaurus3 9y ago
    If you kill their company, people will stop reporting breaches. It doesn't matter that it's illegal to keep it quiet.
parzivalm 9y ago
If they were doing it right the DB would be encrypted and the PCI fields would be tokenized and then if you get dumps of the DB you don't have anything you can use.
femto113 9y ago
My guess is the problems here were more banal: the account running apache had too much privilege (maybe even root?), and the vulnerability was left unpatched for a long time (weeks at least). If you can execute arbitrary commands on a web server it's essentially impossible for any architecture to keep your hands off any (or all) data that server needs access to for its normal functions.
lowbloodsugar 9y ago
Was it just a single CVE? That was the entry point, but was their network just wide open after that. Or did the have to procure more CVEs to get by ssh, firewalls, etc? I think its about time we had the equivalent of the NTSB to review and publish, publicly, what happened and what the failures were. Nobody died but certainly many people will be affected by this for years to come.

sillysaurus3 9y ago

If a hacker can use just one CVE to break into your system and do a database dump (or equivalent), the system is architecturally wrongly designed and only being protected by a single layer of security. Which means, any one from the inside can access pretty much the info as the hacker, which is horrible. This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals. It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau.

azurezyq 9y ago
I think it is covered by compliance, e.g., https://www.google.com/search?q=pci+dss+encryption+at+rest&oq=pci+dss+encryption&aqs=chrome.1.69i57j0l5.8302j0j4&sourceid=chrome&ie=UTF-8 https://www.google.com/search?q=pci+dss+encryption+at+rest&o... Batch processing on encrypted data is absolutbly ok. For example, you can just pull row-level keys and run your audited and signed binary. Also the pipeline should be put away from the frontend. I'm not an expert but I think it should already be documented in the compliance doc.
- sillysaurus3 9y ago
  If you can batch process the dataset, an attacker can do the same thing. They take whatever creds the batch script is using and use them directly. If the program only runs from a certain box, then they run their modified script on that box. "Well don't let that happen!" Nope, not that easy. This threat model concedes that the hackers have RCE. They're on your network. That's why they can do this.
  - azurezyq 9y ago
    "All binaries are audited and signed.", which makes it trackable. And the runtime environment is also isolated and uses different set of access permissions. I'm not saying it is impossible. Just layers and layers. For internal stuffs, Google's beyondcorp is a good start.
    - sillysaurus3 9y ago
      There is no such thing as a runtime environment that (a) attackers gained RCE into, and (b) the box with the sensitive data can verify the incoming connection is from an audited, signed binary. Simply put: if you have RCE into an app, then that breaks all the guarantees of that audited, signed binary. You can make it sing and dance and ask for a thousand rows instead of ten; you can do whatever you want to it. The modifications happen in memory, not on disk, so even if they don't persist you can still modify the app. Monitoring seems like the only solution.
  - kelnos 9y ago
    Why would you ever allow public web access to a machine that can do that level of batch processing? That sort of thing should be accessible only through internal-only tools protected by a VPN.
    - sillysaurus3 9y ago
      You'd be amazed at how easy it is to exploit internal tools once you're inside someone's network. Internal tools were the juiciest targets: a perfect storm of "created by an unqualified team that no longer works here" and "nobody realized it's still been running for a year." It can work, but evidence seems to imply this strategy makes the situation worse. I'm not sure what to do with this information other than to relay the experience. It's one of those tricky counterintuitive facts.
lowbloodsugar 9y ago
>It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How do you read linux firewall documentation with the first example being old school public->DMZ->private? So your web server got hacked? That means what? Your database is across a single port through the firewall, and ACLs mean that any traffic arriving at the part from the DMZ CIDR can only do the limited set of queries that provide single-person-at-time. SSH from hacked box to database? Got a zero-day for the firewall (because 22 isn't open from there), and a zero-day for sshd (because its keys only). I know this and I'm not one of our hundreds of security people. (And as I'm not the security guy, if any actual security guys want to school me here, please do so!) Sure, I agree that business peeps will say "Oh that's too expensive I don't care", but lets not pretend that we don't know how to do this. Be clear, if you are the IT security guy, and you agree to bow down to "business" people (you're a business person) who tell you "No, that's too complicated", then just be clear that you are getting fired when the shit hits the fan.
- sillysaurus3 9y ago
  Yep. You can do all that. At the end of it, a box somewhere needs to be able to access the data. That box has access. You handwave it as single-person-at-a-time queries. But on any given day, they have to run tens or hundreds of thousands of those queries. Even if people aren't accessing their credit info, there is plenty of automation that does. Meaning if you pop that single person query box, you can scale up the queries by a factor of ten without raising any flags directly. It's much better, certainly. Companies should do that. But it's not obviously a panacea.
  - lowbloodsugar 9y ago
    Conceivably, the attackers spent the 75 days downloading the 143,000,000 records, 2,000,000 records per day, and managed to trigger no scaling alarms as a result. Or perhaps there were no alarms. I wonder if we'll ever know.
    - sova 9y ago
mi100hael 9y ago
Things one could do that would increase security without hindering business goals: - PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. This is an architectural decision! If you cannot put out a hotfix within months of a vulnerability being made public, YOUR APPLICATION & DEPLOYMENT DESIGNS ARE BROKEN. - WAF rules for common attacks and known CVEs - Least privilege enforced architecturally via stored procedures or microservices or something else so that a vulnerability in one application (the online dispute portal in this case) can't lead to a dump of everyone's PII. - Data should be encrypted and encryption keys should be stored apart from both the application and the data. Upon startup, the application should make an authenticated request to read the encryption keys into memory. This significantly raises the bar to dumping sensitive data. - Monitoring for connections transferring anomalous amounts of data That's just off the top of my head. All of these can be designed into a system and still maintain 100% uptime.
- wavegeek 9y ago
  > PATCHING OF KNOWN CRITICAL SECURITY VULNERABILITIES IMMEDIATELY. I worked in system admin for many years. If anyone was actually foolish enough to do this, they would be dealing with multiple major outages every year. > for every complex problem there is a solution that is simple, obvious, and wrong.
  - georgebarnett 9y ago
    "There might be an outage" is the stupidest reason I have ever heard to advocate not applying security patches. There's plenty of companies that are able to regularly patch and upgrade software, incident free. If you team isn't able to do this, they need to fix their process instead of putting their head in the sand.
    - wavegeek 9y ago
      > stupid I am not defending the totality of EFX's process by any means. I am just pointing out that "install every high severity patch right away" is not as easy as it looks. Others have pointed out this patch was not available for all current versions, for example. So, to install the patch you need to upgrade. Oh, that breaks <dependency>. So we need to upgrade <other thing>. But that regresses a feature we use .... The major issue with EFX was the fragility of their overall architecture. A single weakness should not be enough to lay the whole DBMS open.
btmorex 9y ago
Put machines with sensitive data on a private network. Have one machine/one set of machines connected to both the private network and public network that proxy requests to the data machines. Proxy by parsing original request and then rewriting (i.e. don't just pass along the request). Rate limit and implement notifications on data machines so that if the proxies are compromised, the attacker can't easily do a full data dump. Don't allow remote administration only physical (this rules out this architecture for most companies, but honestly, not a company like equifax with incredibly sensitive info and where network locality is not a big issue).
- btown 9y ago
  To the remote administration point, if you have this architecture and still open up the private data-network to remote SSH connections, you're at least limiting the attack space to vulnerabilities in SSH and the network stack/firewall on your ingress server, but not vulnerabilities in the application server. Much better than nothing.
- rmrfrmrf 9y ago
  If you look at the CVE, it's literally plain shell commands as strings in the Content-* headers. These commands would appear to be local with whatever user the app is running as.
  - kelnos 9y ago
    And that's the ridiculous thing: this is actually a fairly easy thing to protect against. Why the hell did the webapp servers have direct access to the database? If there was a service in the middle mediating requests, that would have been another line of defense. Sure, maybe _that_ service might have holes, but it'd be another hurdle the attacker would have to clear.
    - rmrfrmrf 9y ago
      As long as JNDI is a best practice in Java land, I'd say that the majority of Java app servers are directly connected to databases.
IncRnd 9y ago
"How does your architecture work in practice? Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB." Simple. Don't store a key with the data. A credit bureau only incurs liability by knowing the PII for hundreds of millions of people readable from one location - that they provide to everyone who asks! How does a company create a better architecture? Separation of duties and responsibilities, defense in depth, authentication and authorization, cryptography, updating software, etc. Hire a security team to help with vulnerability analysis, design reviews, due diligence, and threat modeling. This gets solved every day at many companies. "You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau." Yes, but the elephant in the room is that a credit bureau possesses that information in usable form. This can be taken care of by key management, where the individual owns the key and only one key is able to exist per individual at a time. Then, only those credit grantors who have been authorized by the individual are able to update or view the data. In cryptography the operating assumption should always be that everything is known about your data except your key material, which is kept private.
- sillysaurus3 9y ago
  Yes, but the elephant in the room is that a credit bureau possesses that information in usable form. This can be taken care of by key management, where the individual owns the key and only one key is able to exist per individual at a time. Then, only those credit grantors who have been authorized by the individual are able to update or view the data. This is what I mean by not meeting business goals. It is clearly a bad idea to give Grandma an irreplaceable key. But that's what this proposal is.
  - IncRnd 9y ago
    Wrong. It is your interpretation that this is an irreplaceable key. You are trying to say that a design is wrong, because you believe that your implementation would be wrong. Uniqueness is a different property from replaceability. And, replaceability is not necessarily the correct property to design for, key rotation and recovery from loss are the properties. What you want is the ability to rotate keys and recover from the loss of keys.
mulmen 9y ago
Here's a secure architecture for you: Don't ever store multiple pieces of personally identifiable information in the same place. The consequence of this is that Equifax's business model is no longer viable. I care about that as much as they care about me and my data.
js2 9y ago
In multiple threads now I've seen you express this sentiment. I just don't buy it. Vanguard hasn't had a leak like this. USAA hasn't had a leak like this. ETrade, Fidelity, etc, no leaks like this of which I'm aware (correct me if I'm wrong). These are big plodding institutions too. No leaks like this from Google nor Facebook nor Amazon. Maybe you're arguing it's just a matter of time, but I disagreee. Equifax had a trivially easy architecture to break. Minimally, the data should be on a separate set of servers protected by an API with rate limiting, and that API should be implemented on top of a different framework than the front-end servers to minimize the chance that a single flaw would open up the entire set of data. This isn't rocket science. Equifax is a multi-billion dollar company. They could and should have done better. Please stop defending them? Disclosure: I work for Yahoo (er, Verizon) since late 2013. I don't know the details of Yahoo's data leaks, but it was not as trivial as getting RCE on Yahoo's public facing servers.
- sillysaurus3 9y ago
  When two people are clearly intelligent but disagree, the reason is almost always a different set of experiences. My experience differs from yours; if I could Vulcan my brain to yours, you'd see why I feel this way. You might still disagree, but it would be far less clear cut. Now, I've seen nearly a hundred codebases and they all looked pretty much like Equifax. I don't even need to see Equifax to know the kind of disaster their codebase facilitated. Equifax's codebase is your code, and my code, and Bob's code. The only way to make this situation better is to talk openly about it. And the best way to do that is to actively defend the unpopular side in order to call attention to uncomfortable truths. It's a calculated strategy. Just because you worked at a place with good security hygiene, it makes no difference to the world at large. Everybody else looks like Equifax. We need to (a) acknowledge that and (b) actively seek out solutions. That's what's happening here. Ask yourself: "Why do I disagree when I've seen one or two enterprise codebases?" Those big, plodding institutions you mention are similarly vulnerable. Everyone is. Why aren't data breaches happening almost daily? Because it's illegal. It's no coincidence that companies with public bug bounties are among the hardest targets to hit. When you make it not-illegal to attack you, surprise: you find the vulns. A good question is, "is it illegal for a white hat to attack my business?" If the answer is yes, seek to change that. It's a low effort thing that will increase your security posture. But it's no substitute for $40k pentests.
qyv 9y ago
> This is how it works at almost every company. Call it horrible if you want, but everyone stops short of describing in detail the exact architectural replacement. And when someone does, it inevitably has flaws that make it impossible to meet business goals. No, it's not how almost every company works, and yes it IS horrible. Do you know why people stop short of giving details of what should have been done? Because that is a full time job for a entire team of people in any place that needs real security, and part of that job is to be constantly rooting out flaws in both design and implementation and fixing them. And guess what, if your business involves storing and retrieving sensitive personal information on millions of people, then security needs to be one of your business goals. If it is not, then that is negligent. > It's too easy to handwave this away as "well, get a better architecture." Everyone knows that; the hard part is, what architecture? How does your architecture work in practice? It is also easy to handwave away the real work that real security professionals do. Security is not a just some module you can strap into your systems, it is an ongoing process that requires vigilance and competence. Yes it is hard, but being hard is not an excuse > Encrypting items won't help when the keys need to be stored in the same place to access the data. The attackers will just go after that box instead of the DB. You can guard it more carefully, but the point is that some box somewhere needs access to a substantial amount of the data at any given time. It's the nature of a credit bureau. So you are implying that data encryption is not useful for protecting sensitive data because you have to store the keys somewhere. If that were the case then data encryption in general would be useless, we know that is not the case. This is a design or implementation problem, not an encryption problem.
- x0x0 9y ago
  It's frustrating how people seem not to think it's wildly, ludicrously incompetent for nothing in your architecture to complain about your front end servers querying 140m records. The fact they don't have even the most basic data access controls in place make me assume they're still fully owned. You're playing IT if you can't deploy a CVE fix in 60 days. Finally, they only did this because of greed. They encourage online disputes because it saves them money and gives them greater rights vis-a-vis paper disputes delivered to them via snail mail. They could simply have taken the service down if if they couldn't deploy a security patch in two months. Amateur hour everywhere. ps -- the data controls I mentioned above were implemented in a company of ~15 eng dealing with similar PII. If you owned front end servers you could query individual records, but you would have to own more services to be able to "\copy (select * from data) to /legit.file; gzip /legit.file; scp /legit.file.gz $MY_SERVER". And if you tried pagers would have gone off all over the company.
cmiles74 9y ago
I disagree with the fatalist tone. Sure, it can be hard for big, slow moving businesses to take a time out from writing million dollar checks to C-suite executives to spend money on a competent operations and security team. But it can happen and in this case it should have happened. I don't think "better architecture" is handwavy. If you have old legacy apps that can't be updated and maintained, don't provide them with a path to the public internet. Or front then with some infrastructure and adequate alarms to detect something going wrong. I think it's reasonable to respond to a CVE with an update to your app or infrastructure within one week. Most of the time rapid testing will be fine as your patching a library or component, like Struts or Tomcat. These companies are hardly providing 100% uptime, in the case where an update like this needs a rollback, it won't be a big deal. If you have a good team and you spend the money on them.
gloverkcn 9y ago
This is a solved problem. Companies that store personal information at that level should be required to implement PCI-DSS level security. This includes going through the auditing process. Here's a quick overview. https://www.pcisecuritystandards.org/documents/PCI%20SSC%20Quick%20Reference%20Guide.pdf https://www.pcisecuritystandards.org/documents/PCI%20SSC%20Q... You can work to not implement the security standards, and then try to fake your way through the audit. At that point you are not ignorant of proper security, you are actively endangering the users, and your right to hold PII (Personally Identifiable Information) should be revoked.
avip 9y ago
I don't buy your observation that all big-corps are kind of equally sloppy security-wise. Here are 2 examples: - Equifax login form does not use STS header. - equifax.com/business exposes full server version (Server:Apache/2.4.27 (Win64) OpenSSL/1.1.0f mod_log_rotate/1.0.0) I'm not a security person but those 2 seems way out of industry standards for security-aware web services.

patcheudor 9y ago

"the system is architecturally wrongly designed" Like: https://www.fastcompany.com/40468811/heres-why-equifax-yanked-its-apps-from-apple-and-google-last-week https://www.fastcompany.com/40468811/heres-why-equifax-yanke...

zaroth 9y ago
I was sure the vulnerability was going to be an API key hard coded in the app which allowed retrieving any credit report over the API. That the app was "merely" sending some calls unencrypted was a total let down.
- patcheudor 9y ago
  API creds maybe could have been excusable as a mistake which got overlooked or even permissions which changed. In the context of a credit monitoring app, the use of HTTP is really, really, really bad and can't really be excused. It would be interesting to know if there are more issues, but a lot of white-hat researchers stay away from such things when there's no official bug bounty program because of the computer fraud and abuse act. Interacting in the API outside its intended use by the app could be considered computer intrusion and it might be advisable to stay away from legal grey areas right now when it comes to Equifax research.

outoftacos 9y ago

Yeah they tried to throw Struts under the bus but actually just confessed to criminal negligence. I don't think they are competent enough to realize how damning this press release is. This company is finished.

Exuma 9y ago
I hope what you say is true. I am so livid after reading this.
- sillysaurus3 9y ago
  If you kill their company, people will stop reporting breaches. It doesn't matter that it's illegal to keep it quiet.

parzivalm 9y ago

If they were doing it right the DB would be encrypted and the PCI fields would be tokenized and then if you get dumps of the DB you don't have anything you can use.

femto113 9y ago

My guess is the problems here were more banal: the account running apache had too much privilege (maybe even root?), and the vulnerability was left unpatched for a long time (weeks at least). If you can execute arbitrary commands on a web server it's essentially impossible for any architecture to keep your hands off any (or all) data that server needs access to for its normal functions.

lowbloodsugar 9y ago

Was it just a single CVE? That was the entry point, but was their network just wide open after that. Or did the have to procure more CVEs to get by ssh, firewalls, etc? I think its about time we had the equivalent of the NTSB to review and publish, publicly, what happened and what the failures were. Nobody died but certainly many people will be affected by this for years to come.