21 ms·

Apache Struts Statement on Equifax Security Breach

158 points by Randgalt 9y ago

smaili 9y ago
Our general advice to businesses and individuals utilizing Apache Struts as well as any other open or closed source supporting library in their software products and services is as follows: 1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions. 2. Establish a process to quickly roll out a security fix release of your software product once supporting frameworks or libraries needs to be updated for security reasons. Best is to think in terms of hours or a few days, not weeks or months. Most breaches we become aware of are caused by failure to update software components that are known to be vulnerable for months or even years. 3. Any complex software contains flaws. Don't build your security policy on the assumption that supporting software products are flawless, especially in terms of security vulnerabilities. 4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer such as the Apache Struts framework. A breach into the presentation layer should never empower access to significant or even all back-end information resources. 5. Establish monitoring for unusual access patterns to your public Web resources. Nowadays there are a lot of open source and commercial products available to detect such patterns and give alerts. We recommend such monitoring as good operations practice for business critical Web-based services.
- blktiger 9y ago
  #1 here can be automated. My current employer sells one such solution, though there are of course others. Here is a blog post about the vulnerability from back when it was discovered earlier this year. http://blog.sonatype.com/2017-struts2-vulnerability-exploit-speed-matters http://blog.sonatype.com/2017-struts2-vulnerability-exploit-...
- upstarter 9y ago
  > 1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions. This issue is solved by having your server OS download and install security updates automatically, which amounts to more or less uncommenting 1 line of config. Edit: Downvoter(s), please comment.
  - solomatov 9y ago
    I am not the downvoter (and I didn't donwvote you) but I explain why it won't help. Struts is packaged as a jar file which is distributed inside of the war file which is basically application. The struct's jar is an essential part of the application and it can't be updated separately by the OS.
  - wglb 9y ago
    With respect to This issue is solved I suggest rewording it as This issue is partially solved, since in almost any real-world large web application, there is a significant number of third-party libraries that won't be automatically updated.
- upstarter 9y ago
  > 4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer What are your best sources/tutorials explaining the perfect server architecture setup which address this point?
  - Rapzid 9y ago
    You're looking for information on N-tier applications; https://msdn.microsoft.com/en-us/library/bb384398.aspx?f=255&MSPPError=-2147217396 https://msdn.microsoft.com/en-us/library/bb384398.aspx?f=255.... You don't come across this too often these days in OSS or startups. It's a pretty big explosion in complexity. What hot N-tier enabling frameworks have you read about recently?
mevile 9y ago
The problem is that Equifax put within webserver's reach information that had no business being there. Apache Strut's vulnerability is unfortunate, but it shouldn't have been the keys to the kingdom, where the kingdom is the personal information of nearly the entire US adult population with a credit history. If I knew a service relied only on the security of a web server to protect deeply personal information such as my name and SSN I'd never sign up. We didn't have that choice with Equifax. Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.
- tptacek 9y ago
  This is a problem, yes, but I think it's worth keeping the perspective that it's a practically universal problem. That doesn't mean we should be letting Equifax off the hook. But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100. We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services. Nobody on HN will like that fact (at least, as stated bluntly like that; plenty of them do like the increase to their earnings capability that results from that tradeoff), but it's generally true.
  - justicezyx 9y ago
    You claim that you keep my data safe. You missed it. It's not a trade off, it's a failure on your part...
    - tptacek 9y ago
      It very much is a trade-off, because the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth, and most likely result in software engineers making substantially less money. I'm not saying you should like the tradeoff, or that the existence of the tradeoff doesn't have implications for how we should regulate banks and credit ratings agencies. I'm just saying that if you think there's some other CRA that has reliably protected its IT assets from threats like this, you're wrong.
gedy 9y ago
This is not Struts' "fault"... An entity like Equifax cannot hold such private info and power over our finances, then not have multiple layers of protection to prevent this. You expect bugs in user faceing software, so you protect against breaches so that you don't expose 140 MILLION damn records..
- luckydata 9y ago
  You could stop at "An entity like Equifax cannot hold such private info and power over our finances" Credit rating is an instrument of control used by banks and credit card organizations to force us into debt we don't want. This system is CRAZY and I don't understand why normally anti-establishment and "leave me alone" Americans so sheepishly agree to be involved in a system that hurts our individual liberties every day. When I came to this country the "credit score" system was the single most incomprehensible thing about american society. Still is.
  - lern_too_spel 9y ago
    How is credit-worthiness determined in your country of origin?
  - __david__ 9y ago
    I’m not a fan of credit ratings companies either, but how do they force me into debt?
    - magnetic 9y ago
      I suspect the statement is meant this way: in order to build credit (to buy a house later for example), you need to get in debt: you need to get credit cards, loans, etc... and "climb the ladder". Only then will you have a credit worthy status to get a loan for a house. When I came to the US from Europe, I had no credit history and that made it difficult to get any type of credit card, or loan, or anything. I thought it was a catch 22 since I couldn't build credit because I didn't have credit, but there is this thing called "Secured Credit Card", where the bank lends you your own money (that you deposit) to see if you can repay yourself. Had to do that for about a year, and I also managed to get a Macy's card with a $100 credit limit (yes, one hundred) that forced me to shop there to "build credit". After that the climbing of the ladder started with credit limits increasing slowly, etc... up to the point that I was able to get a loan for a house. There's a bit of strategic planning to building credit worthiness, and that requires you getting into some "managed debt". I think that was the spirit of the comment you are asking about.
yogthos 9y ago
I think this highlights the general problem of Java EE style architecture. There are many moving pieces and many permutations of how they interact together. It's practically impossible to understand it in its entirety, and thus impossible to guarantee that it's secure. You're basically plugging holes as you find them, but you're never sure that there aren't more holes you don't know about.
- hota_mazi 9y ago
  > I think this highlights the general problem of Java EE style architecture. Sounds like you didn't even bother reading the article and jumped at the opportunity to criticize something you don't like. The breach has nothing to do with Java EE and everything to do with Equifax' absurd network configuration.
  - yogthos 9y ago
    I'm not jumping to any conclusions. The exploit appears to be a deserialization bug in a the REST plugin for Struts that allowed the attackers to perform arbitrary code execution: >The bug specifically affects a popular plugin called REST, which developers use to handle web requests, like data sent to a server from a form a user has filled out. The vulnerability relates to how Struts parses that kind of data and converts it into information that can be interpreted by the Java programming language. When the vulnerability is successfully exploited, malicious code can be hidden inside of such data, and executed when Struts attempts to convert it. >That means intruders could easily inject malware into web servers, possibly without being detected, and use it to steal or delete sensitive data, or infect computers with ransomware, among other things. https://qz.com/1073221/the-hackers-who-broke-into-equifax-exploited-a-nine-year-old-security-flaw/ https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
    - hota_mazi 9y ago
      Like I said, none of that would have happened if Equifax had used very basic network safety principles. This has nothing to do with Java EE.
orange_county 9y ago
Apache brings up a good point about having layers of security. I wonder how Equifax was storing the data. Was it just plain text files?
throwaway699552 9y ago
Interesting to also note that the "workaround" listed in the first announcement by the Apache Struts team was wrong. I followed the directions and my web application was still vulnerable. They have since updated it, but without an announcement.
deleted 9y ago
[deleted]
idibidiart 9y ago
PCI anyone?
0xbear 9y ago
Not The Onion: Equifax's "chief security officer" majored in Music Composition: https://www.linkedin.com/in/susan-m-93069a/ https://www.linkedin.com/in/susan-m-93069a/. How did she even get this job?
- runeks 9y ago
  What does her competence in music theory say about her knowledge of IT security? Intelligent humans beings usually aren’t limited to a single area of interest.
  - 0xbear 9y ago
    It basically says she very likely doesn't know a damn thing about how computers work, let alone how to make large, critical systems secure.
- DavidWoof 9y ago
  I honestly couldn't care less what she majored in a quarter century ago. Do you honestly believe a CS bachelors degree from a couple of decades ago would have any relevance to information security today?
  - 0xbear 9y ago
    Certainly more relevance than a Music degree. Having been a security researcher in the past would have helped too, but that's not on the menu either. I would like to understand the thought process that resulted in hiring a Music major to run an organization in charge of protecting one of the largest troves of PII in the history of mankind.
    - watwut 9y ago
      Around 5 years into career, it does not matter what you studied or whether you finished. It matters only what you do now and how old are you. The process starts with music major getting job out of music industry and then working her way up the corporate ladder just as anybody else ever worked his way up - except with having some disadvantage in the beginning.
solomatov 9y ago
As a Java developer, this gives me the lesson not to use smart meta programming facilities, like reflection, where possible. You reduce amount of code, but at the cost of making your protocols injectable to arbitrary code often in unobvious ways.
- tannhaeuser 9y ago
  That's a good conclusion to take away. But reflection and dynamic bytecode manipulation is used in Java all over the place. Class loaders are a core Java feature, and reflection is used in almost all modern annotation-based packages for DB access, object serialization, remoting, dependency injection, etc. Heck, any JITing language requires process images allowed to execute code in dynamic memory segments, such that basic NoExecute hardware features can't be used. Combine this with Java server-side apps being run in a single process/address space, and I hope you can see that, if nothing else, from a security PoV Java is a dead end.

158 points by Randgalt 9y ago

Our general advice to businesses and individuals utilizing Apache Struts as well as any other open or closed source supporting library in their software products and services is as follows: 1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions. 2. Establish a process to quickly roll out a security fix release of your software product once supporting frameworks or libraries needs to be updated for security reasons. Best is to think in terms of hours or a few days, not weeks or months. Most breaches we become aware of are caused by failure to update software components that are known to be vulnerable for months or even years. 3. Any complex software contains flaws. Don't build your security policy on the assumption that supporting software products are flawless, especially in terms of security vulnerabilities. 4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer such as the Apache Struts framework. A breach into the presentation layer should never empower access to significant or even all back-end information resources. 5. Establish monitoring for unusual access patterns to your public Web resources. Nowadays there are a lot of open source and commercial products available to detect such patterns and give alerts. We recommend such monitoring as good operations practice for business critical Web-based services.

blktiger 9y ago
#1 here can be automated. My current employer sells one such solution, though there are of course others. Here is a blog post about the vulnerability from back when it was discovered earlier this year. http://blog.sonatype.com/2017-struts2-vulnerability-exploit-speed-matters http://blog.sonatype.com/2017-struts2-vulnerability-exploit-...
upstarter 9y ago
> 1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions. This issue is solved by having your server OS download and install security updates automatically, which amounts to more or less uncommenting 1 line of config. Edit: Downvoter(s), please comment.
- solomatov 9y ago
  I am not the downvoter (and I didn't donwvote you) but I explain why it won't help. Struts is packaged as a jar file which is distributed inside of the war file which is basically application. The struct's jar is an essential part of the application and it can't be updated separately by the OS.
- wglb 9y ago
  With respect to This issue is solved I suggest rewording it as This issue is partially solved, since in almost any real-world large web application, there is a significant number of third-party libraries that won't be automatically updated.
upstarter 9y ago
> 4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer What are your best sources/tutorials explaining the perfect server architecture setup which address this point?
- Rapzid 9y ago
  You're looking for information on N-tier applications; https://msdn.microsoft.com/en-us/library/bb384398.aspx?f=255&MSPPError=-2147217396 https://msdn.microsoft.com/en-us/library/bb384398.aspx?f=255.... You don't come across this too often these days in OSS or startups. It's a pretty big explosion in complexity. What hot N-tier enabling frameworks have you read about recently?

mevile 9y ago

The problem is that Equifax put within webserver's reach information that had no business being there. Apache Strut's vulnerability is unfortunate, but it shouldn't have been the keys to the kingdom, where the kingdom is the personal information of nearly the entire US adult population with a credit history. If I knew a service relied only on the security of a web server to protect deeply personal information such as my name and SSN I'd never sign up. We didn't have that choice with Equifax. Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.

tptacek 9y ago
This is a problem, yes, but I think it's worth keeping the perspective that it's a practically universal problem. That doesn't mean we should be letting Equifax off the hook. But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100. We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services. Nobody on HN will like that fact (at least, as stated bluntly like that; plenty of them do like the increase to their earnings capability that results from that tradeoff), but it's generally true.
- justicezyx 9y ago
  You claim that you keep my data safe. You missed it. It's not a trade off, it's a failure on your part...
  - tptacek 9y ago
    It very much is a trade-off, because the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth, and most likely result in software engineers making substantially less money. I'm not saying you should like the tradeoff, or that the existence of the tradeoff doesn't have implications for how we should regulate banks and credit ratings agencies. I'm just saying that if you think there's some other CRA that has reliably protected its IT assets from threats like this, you're wrong.

gedy 9y ago

This is not Struts' "fault"... An entity like Equifax cannot hold such private info and power over our finances, then not have multiple layers of protection to prevent this. You expect bugs in user faceing software, so you protect against breaches so that you don't expose 140 MILLION damn records..

luckydata 9y ago
You could stop at "An entity like Equifax cannot hold such private info and power over our finances" Credit rating is an instrument of control used by banks and credit card organizations to force us into debt we don't want. This system is CRAZY and I don't understand why normally anti-establishment and "leave me alone" Americans so sheepishly agree to be involved in a system that hurts our individual liberties every day. When I came to this country the "credit score" system was the single most incomprehensible thing about american society. Still is.
- lern_too_spel 9y ago
  How is credit-worthiness determined in your country of origin?
- __david__ 9y ago
  I’m not a fan of credit ratings companies either, but how do they force me into debt?
  - magnetic 9y ago
    I suspect the statement is meant this way: in order to build credit (to buy a house later for example), you need to get in debt: you need to get credit cards, loans, etc... and "climb the ladder". Only then will you have a credit worthy status to get a loan for a house. When I came to the US from Europe, I had no credit history and that made it difficult to get any type of credit card, or loan, or anything. I thought it was a catch 22 since I couldn't build credit because I didn't have credit, but there is this thing called "Secured Credit Card", where the bank lends you your own money (that you deposit) to see if you can repay yourself. Had to do that for about a year, and I also managed to get a Macy's card with a $100 credit limit (yes, one hundred) that forced me to shop there to "build credit". After that the climbing of the ladder started with credit limits increasing slowly, etc... up to the point that I was able to get a loan for a house. There's a bit of strategic planning to building credit worthiness, and that requires you getting into some "managed debt". I think that was the spirit of the comment you are asking about.

yogthos 9y ago

I think this highlights the general problem of Java EE style architecture. There are many moving pieces and many permutations of how they interact together. It's practically impossible to understand it in its entirety, and thus impossible to guarantee that it's secure. You're basically plugging holes as you find them, but you're never sure that there aren't more holes you don't know about.

hota_mazi 9y ago
> I think this highlights the general problem of Java EE style architecture. Sounds like you didn't even bother reading the article and jumped at the opportunity to criticize something you don't like. The breach has nothing to do with Java EE and everything to do with Equifax' absurd network configuration.
- yogthos 9y ago
  I'm not jumping to any conclusions. The exploit appears to be a deserialization bug in a the REST plugin for Struts that allowed the attackers to perform arbitrary code execution: >The bug specifically affects a popular plugin called REST, which developers use to handle web requests, like data sent to a server from a form a user has filled out. The vulnerability relates to how Struts parses that kind of data and converts it into information that can be interpreted by the Java programming language. When the vulnerability is successfully exploited, malicious code can be hidden inside of such data, and executed when Struts attempts to convert it. >That means intruders could easily inject malware into web servers, possibly without being detected, and use it to steal or delete sensitive data, or infect computers with ransomware, among other things. https://qz.com/1073221/the-hackers-who-broke-into-equifax-exploited-a-nine-year-old-security-flaw/ https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
  - hota_mazi 9y ago
    Like I said, none of that would have happened if Equifax had used very basic network safety principles. This has nothing to do with Java EE.

orange_county 9y ago

Apache brings up a good point about having layers of security. I wonder how Equifax was storing the data. Was it just plain text files?

throwaway699552 9y ago

Interesting to also note that the "workaround" listed in the first announcement by the Apache Struts team was wrong. I followed the directions and my web application was still vulnerable. They have since updated it, but without an announcement.

deleted 9y ago

[deleted]

idibidiart 9y ago

PCI anyone?

0xbear 9y ago

Not The Onion: Equifax's "chief security officer" majored in Music Composition: https://www.linkedin.com/in/susan-m-93069a/ https://www.linkedin.com/in/susan-m-93069a/. How did she even get this job?

runeks 9y ago
What does her competence in music theory say about her knowledge of IT security? Intelligent humans beings usually aren’t limited to a single area of interest.
- 0xbear 9y ago
  It basically says she very likely doesn't know a damn thing about how computers work, let alone how to make large, critical systems secure.
DavidWoof 9y ago
I honestly couldn't care less what she majored in a quarter century ago. Do you honestly believe a CS bachelors degree from a couple of decades ago would have any relevance to information security today?
- 0xbear 9y ago
  Certainly more relevance than a Music degree. Having been a security researcher in the past would have helped too, but that's not on the menu either. I would like to understand the thought process that resulted in hiring a Music major to run an organization in charge of protecting one of the largest troves of PII in the history of mankind.
  - watwut 9y ago
    Around 5 years into career, it does not matter what you studied or whether you finished. It matters only what you do now and how old are you. The process starts with music major getting job out of music industry and then working her way up the corporate ladder just as anybody else ever worked his way up - except with having some disadvantage in the beginning.

solomatov 9y ago

As a Java developer, this gives me the lesson not to use smart meta programming facilities, like reflection, where possible. You reduce amount of code, but at the cost of making your protocols injectable to arbitrary code often in unobvious ways.

tannhaeuser 9y ago
That's a good conclusion to take away. But reflection and dynamic bytecode manipulation is used in Java all over the place. Class loaders are a core Java feature, and reflection is used in almost all modern annotation-based packages for DB access, object serialization, remoting, dependency injection, etc. Heck, any JITing language requires process images allowed to execute code in dynamic memory segments, such that basic NoExecute hardware features can't be used. Combine this with Java server-side apps being run in a single process/address space, and I hope you can see that, if nothing else, from a security PoV Java is a dead end.