5 ms·

We got hacked

70 points by codegen 14y ago

VaucGiaps 14y ago
Clear and open about it. Good.
priitp 14y ago
Asked my team to review their Jenkins passwords and Jenkins user rights...
- reginaldo 14y ago
  I don't know if you're doing this, but I think it's a bad idea to leave Jenkins publicly accessible. Indeed, IMHO, it's a bad idea to leave stuff that should not be accessible by the general public publicly accessible. Especially things that have access to your code. Do ask your team to review passwords and user rights, but also put this service and others like it behind a VPN. Then both the VPN server and Jenkins will have to have holes simultaneously before you get hacked.
  - Nikem 14y ago
    As the main hero of that story, I can assure you that we are working on VPN setup right now. I guess this is not in top-10 checklist for a startup.
- skrebbel 14y ago
  Should've coded the whole thing in Makumba and this would've never happened!!one
waitwhat7 14y ago
Was the original attack via jenkins? all it says some vague privilege escalation was used to upload c file. what?
- reginaldo 14y ago
  I believe the issue was related to the fact that the user running Jenkins was a full passwordless sudo user. Maybe the attacker used the groovy console too...
- ihsw 14y ago
  No the author seems to indicate that it was on their application code and an attacker was able to get OS access, and the attacker subsequently replaced the ssh service with one that instead ignores login attempts and harvests the username/password pairs. It seems to have been a coincidence that their Jenkins service was not secured.
  - indigo_daddy 14y ago
    What I got from the article is that they likely ssh brute-forced the Jenkins service user and then had passwordless sudo; however you wouldn't think that this user would have a shell assigned.
OptimusSubprime 14y ago
This whole post reads like "How Not to Use AWS" instruction guide. If you're running anything important on a single EC2 instance, you're doing it wrong. If you're logging in and manually configuring an EC2 instance, you're also doing it wrong.
- alan_cx 14y ago
  That's helpful, cheers. How about letting us lesser plebs know how to do it properly then?
  - zacharydanger 14y ago
    This is probably beyond the scope of such a thread, but configuration automation via Chef or Puppet is where you'd start.
    - eli 14y ago
      You really think that's the best place to start? I would begin with a figuring out a more secure, desired configuration before trying to automate it. (Especially given, IMHO, the very steep learning curve for Chef & Puppet)
  - bmelton 14y ago
    Puppet and Chef are great tools, but my first entry into AWS management was a little more attainable without having to learn anything new. I simply set up the instance to automatically download the current codebase from our 'production' repo on every boot, automatically install requirements, run database migrations, etc. Then freeze that AMI as the production AMI. If you migrate your databases off of the instance to RDS, don't use EBS, and manage user uploads and static assets to go straight to S3, then what you end up with a very easily load balanceable configuration. To set up load balancing, just spin up two of those instances, attach them to an elastic load balancer, attach the load balancer to your elastic IP, then you ought to be more redundant and management free. (Note, this doesn't fix your security issues, though you could very easily bake a nightly apt-get run for security updates into that image). After that, you want to look into Fabric (or language-specific alternatives if they exist) to allow for remote management of your machines for things like bouncing web services, pulling new code deploys, etc. If you want to sidestep all this management altogether, I'd strongly suggest looking into something like dotcloud - http://dotcloud.com/ http://dotcloud.com/ - which effectively does all this for you.
krobertson 14y ago
"my fellow co-founder who just happens to have a lot more experience in the field of hacking and getting hacked" I see irony in the "getting hacked" part. But seriously, if he is more experienced, then why do they have machines unpatched like that for 6 months? "it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks" It is rather bold to be posting about how you just got hacked by not applying simple patches, and to then bash the "kid" who couldn't install a proper rootkit.
- ivom2gi 14y ago
  That is indeed the case - both players in this match were not careful. We are open about our shortcomings and also think that the other player was not too careful to say the least ...
- spqr 14y ago
  In fairness to the hacker, that machine was probably the 20th one he hacked that day. I've seen it dozens of times - hackers hack boxes but they have so many that they just become careless. You'll see root shells running right off low ports through inetd because they just want convenience and don't care if someone else "steals" the box from them. I wouldn't be surprised if the hacker accidentally killed SSH through a complete lack of caring due to glutted inventory of hacked boxes. Presumably, OP was fairly dependent on this box so I don't necessarily understand their carelessness.
OGC 14y ago
So... if the attacker would not have had deactivated SSH by accident they would not have noticed?
scoot 14y ago
no customer data was exposed during the attack How did you confirm that?
- jlgaddis 14y ago
  That's a good question. Given some of the "fundamentals" of security that were obviously ignored, I'm reluctant to believe that they can make that assertion with any confidence.
ladzoppelin 14y ago
"C source named backdoor.h" Why would an attacker leave a file with that name? Seems like misdirection or a mistake. How are you going to verify they did nothing else?
deleted 14y ago
[deleted]
drivebyacct2 14y ago
First of all, your stupid copy/paste script has completely broken copy-paste. When I copy, I get nothing except the spam you're trying to inject, so congrats on managing to piss me off right off the bat. Second, why even list A, B, or C? Whoever thought running Jenkins as a passwordless sudo user shouldn't be doing sysops. Why was Jenkins even public facing? At worst, put it behind a VPN.

We got hacked

70 points by codegen 14y ago

VaucGiaps 14y ago

Clear and open about it. Good.

priitp 14y ago

Asked my team to review their Jenkins passwords and Jenkins user rights...

reginaldo 14y ago
I don't know if you're doing this, but I think it's a bad idea to leave Jenkins publicly accessible. Indeed, IMHO, it's a bad idea to leave stuff that should not be accessible by the general public publicly accessible. Especially things that have access to your code. Do ask your team to review passwords and user rights, but also put this service and others like it behind a VPN. Then both the VPN server and Jenkins will have to have holes simultaneously before you get hacked.
- Nikem 14y ago
  As the main hero of that story, I can assure you that we are working on VPN setup right now. I guess this is not in top-10 checklist for a startup.
skrebbel 14y ago
Should've coded the whole thing in Makumba and this would've never happened!!one

waitwhat7 14y ago

Was the original attack via jenkins? all it says some vague privilege escalation was used to upload c file. what?

reginaldo 14y ago
I believe the issue was related to the fact that the user running Jenkins was a full passwordless sudo user. Maybe the attacker used the groovy console too...
ihsw 14y ago
No the author seems to indicate that it was on their application code and an attacker was able to get OS access, and the attacker subsequently replaced the ssh service with one that instead ignores login attempts and harvests the username/password pairs. It seems to have been a coincidence that their Jenkins service was not secured.
- indigo_daddy 14y ago
  What I got from the article is that they likely ssh brute-forced the Jenkins service user and then had passwordless sudo; however you wouldn't think that this user would have a shell assigned.

OptimusSubprime 14y ago

This whole post reads like "How Not to Use AWS" instruction guide. If you're running anything important on a single EC2 instance, you're doing it wrong. If you're logging in and manually configuring an EC2 instance, you're also doing it wrong.

alan_cx 14y ago
That's helpful, cheers. How about letting us lesser plebs know how to do it properly then?
- zacharydanger 14y ago
  This is probably beyond the scope of such a thread, but configuration automation via Chef or Puppet is where you'd start.
  - eli 14y ago
    You really think that's the best place to start? I would begin with a figuring out a more secure, desired configuration before trying to automate it. (Especially given, IMHO, the very steep learning curve for Chef & Puppet)
- bmelton 14y ago
  Puppet and Chef are great tools, but my first entry into AWS management was a little more attainable without having to learn anything new. I simply set up the instance to automatically download the current codebase from our 'production' repo on every boot, automatically install requirements, run database migrations, etc. Then freeze that AMI as the production AMI. If you migrate your databases off of the instance to RDS, don't use EBS, and manage user uploads and static assets to go straight to S3, then what you end up with a very easily load balanceable configuration. To set up load balancing, just spin up two of those instances, attach them to an elastic load balancer, attach the load balancer to your elastic IP, then you ought to be more redundant and management free. (Note, this doesn't fix your security issues, though you could very easily bake a nightly apt-get run for security updates into that image). After that, you want to look into Fabric (or language-specific alternatives if they exist) to allow for remote management of your machines for things like bouncing web services, pulling new code deploys, etc. If you want to sidestep all this management altogether, I'd strongly suggest looking into something like dotcloud - http://dotcloud.com/ http://dotcloud.com/ - which effectively does all this for you.

krobertson 14y ago

"my fellow co-founder who just happens to have a lot more experience in the field of hacking and getting hacked" I see irony in the "getting hacked" part. But seriously, if he is more experienced, then why do they have machines unpatched like that for 6 months? "it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks" It is rather bold to be posting about how you just got hacked by not applying simple patches, and to then bash the "kid" who couldn't install a proper rootkit.

ivom2gi 14y ago
That is indeed the case - both players in this match were not careful. We are open about our shortcomings and also think that the other player was not too careful to say the least ...
spqr 14y ago
In fairness to the hacker, that machine was probably the 20th one he hacked that day. I've seen it dozens of times - hackers hack boxes but they have so many that they just become careless. You'll see root shells running right off low ports through inetd because they just want convenience and don't care if someone else "steals" the box from them. I wouldn't be surprised if the hacker accidentally killed SSH through a complete lack of caring due to glutted inventory of hacked boxes. Presumably, OP was fairly dependent on this box so I don't necessarily understand their carelessness.

OGC 14y ago

So... if the attacker would not have had deactivated SSH by accident they would not have noticed?

scoot 14y ago

no customer data was exposed during the attack How did you confirm that?

jlgaddis 14y ago
That's a good question. Given some of the "fundamentals" of security that were obviously ignored, I'm reluctant to believe that they can make that assertion with any confidence.

ladzoppelin 14y ago

"C source named backdoor.h" Why would an attacker leave a file with that name? Seems like misdirection or a mistake. How are you going to verify they did nothing else?

deleted 14y ago

[deleted]

drivebyacct2 14y ago

First of all, your stupid copy/paste script has completely broken copy-paste. When I copy, I get nothing except the spam you're trying to inject, so congrats on managing to piss me off right off the bat. Second, why even list A, B, or C? Whoever thought running Jenkins as a passwordless sudo user shouldn't be doing sysops. Why was Jenkins even public facing? At worst, put it behind a VPN.