3 ms·

Really? * Passwords hashed with SHA-512 _unsalted_ * No lockout / notification after hundreds of thousands of failed logins * XSS vulnerability * Able to ch

Really? * Passwords hashed with SHA-512 _unsalted_ * No lockout / notification after hundreds of thousands of failed logins * XSS vulnerability * Able to change the configuration to upload executable scripts * SSH password authentication enabled And your top concern was that Slicehost didn't immediately shut down a machine? Would it have really slowed the hackers down for more than an hour anyway?

pquerna 17y ago
fwiw, 4/5 of your points are specific issues to Atlassian JIRA, a commercial product :) The 5th point, as explained in the post... We actually tried to disable Password based authentication -- It is the ASF's standard policy for it to be disabled! -- we just failed at testing the configuration on Brutus. On our other machines, we had a block of configuration options we appended to the END of the sshd configuration, and these options successfully turned off password authentication, requiring ssh keys only. On brutus however, we were missing a "UsePAM no", which meant pam went in and fell back to password authentication. This was a mistake, and we didn't realize until after the vulnerability when we were testing the machine, that its sshd was misconfigured.
- justinsb 17y ago
  I didn't mean to criticize. There were mistakes made - there always are. But amongst the list of mistakes, I would not put Slicehost's slow response anywhere near the top, and I'm surprised that anyone (particularly someone whose opinions I normally respect) could choose that as the lesson to highlight. Anyway, when is Apache going to get serious about a bug tracking project? :-)
- epochwolf 17y ago
  * Able to change the configuration to upload executable scripts Not familiar with JIRA but... couldn't you prevent this by either changing the ownership of the uploaded files or configure the web server not to execute any files in uploaded files directory.
ErrantX 17y ago
Clearly all big problems. But if someone phones you up and says we were hacked using one of your slices you would hope they would at least suspend access and investigate. In computer security an hour can be a long time.
- justinsb 17y ago
  I dislike the implicit presumption that people on the Internet owe it to you to shut down their server if (you say) they're attacking you. I strongly feel that you should presume that your servers are always being attacked, and it's not helpful to even think that you might be able to get attackers shut down.
  - ErrantX 17y ago
    Despite the fact that two days later it was used in another hack? Were I a server provider I would take such notifications very seriously; a trivial investigation would surely have uncovered the illegal activity and shut them down. (we are seeing more and more use of services such as slicehost and EC2 to perform attacks. I doubt that is a tag those companies want to carry!) who said anything about owing it to them? :-)
    - justinsb 17y ago
      There are hosting providers who openly advertise servers for sending spam illegally. The idea that you can achieve security by shutting down your attackers is ridiculous. Had slicehost acted more rapidly, Atlassian would have been hacked by the same hackers using a different server. Obviously Atlassian knows this also, because they didn't even bother putting an IP blocking rule into their firewalls. If Atlassian didn't even think that worthwhile, pointing the finger at Slicehost is dubious.
      - ErrantX 17y ago
        Just to be clear that's not the point I was making.
AngryParsley 17y ago
Most of the things you listed are issues with JIRA and Confluence, not the ASF's practices. Everything else on your list has already been fixed. The ASF infra group did almost everything right but they still got burnt by esoteric details. The sshd conf had PasswordAuthentication set to no, but enabling PAM auth caused regular password auth to work. Did you know about that caveat? And your top concern was that Slicehost didn't immediately shut down a machine? tptacek never said that. He said it was a little disturbing that Slicehost took two days to turn off an instance that was exploiting a previously-unknown vulnerability in JIRA. From the incident report https://blogs.apache.org/infra/entry/apache_org_04_09_2010 https://blogs.apache.org/infra/entry/apache_org_04_09_2010 : We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly. ( http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an_update_on_our_security_breach.html http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an... ) Apache's infrastructure group got their shit together quickly. The hacked Slicehost instance wasn't a threat to them anymore. Slicehost's inaction helped get Atlassian hacked, which I think is a bit disturbing. Would it have really slowed the hackers down for more than an hour anyway? Oh, you're right. Let's just leave hacked systems running. Shutting them down will only slow down hackers for an hour.
- justinsb 17y ago
  Wow - you're certainly living up to your username :-) Shutting down a server doesn't stop the hackers using one of the other thousands of servers they have compromised. In this regard, trying to respond to an attack by shutting down the server is essentially a waste of time: the only winning move is not to play. There were multiple mistakes made (and I'm not trying to cast blame), and I think we should be aware of the repeatable lessons for software development: hashes need to be salted, XSS matters, password lockout matters etc. If there's a lesson from Slicehost, it's that you shouldn't rely on someone else bailing you out.
  - tptacek 17y ago
    You cannot run a hosting provider and ignore reports that your own hardware is being used to launch criminal attacks. It's that simple.
tptacek 17y ago
Yes. Really. It bothers me as a Slicehost customer that Slicehost didn't get on top of a report that one of their VMs was compromised. Those VMs are on the same local networking hardware, and perhaps on the same server hardware, as my own VMs. I'm unclear why I'm supposed to care about Atlassian. I don't use Atlassian products.
- justinsb 17y ago
  OK, serious question to you as a security guru: Shouldn't we always assume that every host outside of our control is compromised? So why should I care how long it takes someone to shut down a compromised host? I think there are lessons we should all learn / double check, but I don't understand why you chose to highlight this issue.
  - tptacek 17y ago
    Yes, you should assume everything is compromised. No, that doesn't mean hosting providers can ignore breakins on their own hardware. Seriously, dude: breaks aren't like unfortunate Internet weather incidents. They're crimes. An actual crime took place using Rackspace computers. How do we even know what kind of forensics could have been conducted if Slicehost can't even acknowledge when their machines are compromised? Hosting providers have special responsibilities.
    - justinsb 17y ago
      OK, point taken. But you're assuming that Slicehost ignored the issue, rather than being bound by their own process. I simply expected more from you: as dfranke said, this is your opportunity to hammer home the importance of good security practice, and instead you're talking about something basically beyond our control (Slicehost's incident response policy) rather than the root cause (a large number of fairly basic security mistakes)
      - tptacek 17y ago
        The fact that you are startled that I'd even be talking about Slicehost's response tells me that not enough attention is being paid to this.

pquerna 17y ago

fwiw, 4/5 of your points are specific issues to Atlassian JIRA, a commercial product :) The 5th point, as explained in the post... We actually tried to disable Password based authentication -- It is the ASF's standard policy for it to be disabled! -- we just failed at testing the configuration on Brutus. On our other machines, we had a block of configuration options we appended to the END of the sshd configuration, and these options successfully turned off password authentication, requiring ssh keys only. On brutus however, we were missing a "UsePAM no", which meant pam went in and fell back to password authentication. This was a mistake, and we didn't realize until after the vulnerability when we were testing the machine, that its sshd was misconfigured.

justinsb 17y ago
I didn't mean to criticize. There were mistakes made - there always are. But amongst the list of mistakes, I would not put Slicehost's slow response anywhere near the top, and I'm surprised that anyone (particularly someone whose opinions I normally respect) could choose that as the lesson to highlight. Anyway, when is Apache going to get serious about a bug tracking project? :-)
epochwolf 17y ago
* Able to change the configuration to upload executable scripts Not familiar with JIRA but... couldn't you prevent this by either changing the ownership of the uploaded files or configure the web server not to execute any files in uploaded files directory.

ErrantX 17y ago

Clearly all big problems. But if someone phones you up and says we were hacked using one of your slices you would hope they would at least suspend access and investigate. In computer security an hour can be a long time.

justinsb 17y ago
I dislike the implicit presumption that people on the Internet owe it to you to shut down their server if (you say) they're attacking you. I strongly feel that you should presume that your servers are always being attacked, and it's not helpful to even think that you might be able to get attackers shut down.
- ErrantX 17y ago
  Despite the fact that two days later it was used in another hack? Were I a server provider I would take such notifications very seriously; a trivial investigation would surely have uncovered the illegal activity and shut them down. (we are seeing more and more use of services such as slicehost and EC2 to perform attacks. I doubt that is a tag those companies want to carry!) who said anything about owing it to them? :-)
  - justinsb 17y ago
    There are hosting providers who openly advertise servers for sending spam illegally. The idea that you can achieve security by shutting down your attackers is ridiculous. Had slicehost acted more rapidly, Atlassian would have been hacked by the same hackers using a different server. Obviously Atlassian knows this also, because they didn't even bother putting an IP blocking rule into their firewalls. If Atlassian didn't even think that worthwhile, pointing the finger at Slicehost is dubious.
    - ErrantX 17y ago
      Just to be clear that's not the point I was making.

AngryParsley 17y ago

Most of the things you listed are issues with JIRA and Confluence, not the ASF's practices. Everything else on your list has already been fixed. The ASF infra group did almost everything right but they still got burnt by esoteric details. The sshd conf had PasswordAuthentication set to no, but enabling PAM auth caused regular password auth to work. Did you know about that caveat? And your top concern was that Slicehost didn't immediately shut down a machine? tptacek never said that. He said it was a little disturbing that Slicehost took two days to turn off an instance that was exploiting a previously-unknown vulnerability in JIRA. From the incident report https://blogs.apache.org/infra/entry/apache_org_04_09_2010 https://blogs.apache.org/infra/entry/apache_org_04_09_2010 : We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly. ( http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an_update_on_our_security_breach.html http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an... ) Apache's infrastructure group got their shit together quickly. The hacked Slicehost instance wasn't a threat to them anymore. Slicehost's inaction helped get Atlassian hacked, which I think is a bit disturbing. Would it have really slowed the hackers down for more than an hour anyway? Oh, you're right. Let's just leave hacked systems running. Shutting them down will only slow down hackers for an hour.

justinsb 17y ago
Wow - you're certainly living up to your username :-) Shutting down a server doesn't stop the hackers using one of the other thousands of servers they have compromised. In this regard, trying to respond to an attack by shutting down the server is essentially a waste of time: the only winning move is not to play. There were multiple mistakes made (and I'm not trying to cast blame), and I think we should be aware of the repeatable lessons for software development: hashes need to be salted, XSS matters, password lockout matters etc. If there's a lesson from Slicehost, it's that you shouldn't rely on someone else bailing you out.
- tptacek 17y ago
  You cannot run a hosting provider and ignore reports that your own hardware is being used to launch criminal attacks. It's that simple.

tptacek 17y ago

Yes. Really. It bothers me as a Slicehost customer that Slicehost didn't get on top of a report that one of their VMs was compromised. Those VMs are on the same local networking hardware, and perhaps on the same server hardware, as my own VMs. I'm unclear why I'm supposed to care about Atlassian. I don't use Atlassian products.

justinsb 17y ago
OK, serious question to you as a security guru: Shouldn't we always assume that every host outside of our control is compromised? So why should I care how long it takes someone to shut down a compromised host? I think there are lessons we should all learn / double check, but I don't understand why you chose to highlight this issue.
- tptacek 17y ago
  Yes, you should assume everything is compromised. No, that doesn't mean hosting providers can ignore breakins on their own hardware. Seriously, dude: breaks aren't like unfortunate Internet weather incidents. They're crimes. An actual crime took place using Rackspace computers. How do we even know what kind of forensics could have been conducted if Slicehost can't even acknowledge when their machines are compromised? Hosting providers have special responsibilities.
  - justinsb 17y ago
    OK, point taken. But you're assuming that Slicehost ignored the issue, rather than being bound by their own process. I simply expected more from you: as dfranke said, this is your opportunity to hammer home the importance of good security practice, and instead you're talking about something basically beyond our control (Slicehost's incident response policy) rather than the root cause (a large number of fairly basic security mistakes)
    - tptacek 17y ago
      The fact that you are startled that I'd even be talking about Slicehost's response tells me that not enough attention is being paid to this.