5 ms·

DigitalOcean Root Vulnerability in the Wild

38 points by goodwink 14y ago

instakill 14y ago
I think the lesson we're all learning here is that if you don't do your own Sysadmin/devops then you're almost guaranteed to be dealing with imperfect systems.
- gpcz 14y ago
  I've got bad news for you about your perfect system...
- ohboyacomment 14y ago
  If you do your own sysadmin/devops, you are certainly guaranteed to be dealing with imperfect systems. (It's very likely worse to do your own unless you have the resources to do it right. Hosting companies have scale. Not defending DigitalOcean here, just challenging your assertion.)
Goranek 14y ago
This makes sense, i got my account blocked for spamming via SMTP, and i didn't send a single mail. It took 5 support letters to get my account unblocked, and the solution was to block SMTP ports... I didn't have anything installed !!! Clear Ubuntu image
- deleted 14y ago
  [deleted]
  - ohboyacomment 14y ago
    What is it, exactly, that you expect to be running and 'default open vulnerable' on a fresh Ubuntu install? On my fresh Ubuntu deploys at Linode, the only network service listening is SSH. (Lest we forget that an open port is not a security vulnerability until a vulnerable service is listening on it, but I'll leave that assertion alone.) EDIT: The cowardly deleted comment chided the poster for not securing his system and mentioned 'default open vulnerable' services.
    - deleted 14y ago
      [deleted]
      - ohboyacomment 14y ago
        Oh, your comment is specific to this DigitalOcean scenario now? Because when you left it, it was phrased as a generality for systems administration, where you "FTFY"'d (God, I hate that) the person you are replying to in order to mock him for not securing his base Linux install. As if there's a hell of a lot to do out of the box. You're probably a -P INPUT DROP kind of guy, aren't you? Path MTU discovery and ping replies be damned? The person you are replying to is not the original poster of this scenario, and the person you are replying to very likely did not trigger the exact same scenario in his dealings. I do not believe one is related to the other, but nor did you, because you were careful not to mention anything related to this scenario at all in your dim, mean-spirited reply. So you're assuming that his root password was flown over e-mail with no evidence to that effect. (A root password flown over e-mail is a vulnerability, but an external one, and has nothing to do with securing a Linux install, which was the gist of your comment.) Couldn't even finish my comment before you turned gray and deleted out. Sad. I wish you the best of luck in your cheap karma adventure.
        Goranek 14y ago
        Wait what!? I was talking about Digital Ocean.
zalew 14y ago
> it is likely that if you reset the root password via the web interface and don't change it afterwards that you are vulnerable shocker.
- goodwink 14y ago
  The shocker is that this is true even if you rebuild the box from the scratch image.
namidark 14y ago
You left root login enabled and kept passwords on. And you're upset you got compromised? You can also enable keys in the control panel and you won't have to deal with passwords being emailed. Those are the first few things you should be checking when setting up a new server (disable password logins and only allow keys).
- goodwink 14y ago
  I'm not upset I got compromised, it was very likely given my configuration, I acknowledge that. I did have keys enabled so the password wasn't email initially, it was only emailed as a result of a root password reset. The article is about what the compromise means about DigitalOcean not about my one box.
gregd 14y ago
Aren't we jumping the gun a bit by saying, "DigitalOcean Root Vulnerability in the Wild"? I'm not yet comfortable with that conclusion...
- readme 14y ago
  I agree. To me it sounds more like root vulnerability on OPs system. He did not change his password after using the automated password reset, knowing the password was sent in plain text.
  - goodwink 14y ago
    No I didn't change the password. Instead I rebuilt the machine from the base OS image. Would you expect that a new system with the disk wiped would retain the old root password? I certainly wouldn't. Where were they keeping it?
    - bpicolo 14y ago
      Whenever I do a rebuild on a droplet it sends me a new root password in the email.
      - goodwink 14y ago
        I have ssh keys set up so it doesn't do this. Instead what it seems to do is reuse your old root password for the new image.
    - readme 14y ago
      Sorry -- my fault. I skimmed your article and I guess I missed this part. That is a bit ridiculous that they retained your password. Still though, logging in after rebuilding and configuring your system would have been a sane move.
    - wfn 14y ago
      > No I didn't change the password. Instead I rebuilt the machine from the base OS image. Probably not strictly related (it would seem the attack in question was via a compromised (on whoever's end) ssh password), but in any case good practice: always regenerate ssh keys after installation from a base image. Some OS images may contain pregenerated keypairs (private key is usually at ~/.ssh/id_rsa ), which should always be regenerated. (This bit is most probably not actually related to the issue at hand, but IAC should not be forgotten.)
thomseddon 14y ago
As soon as I saw the text "Your root password will be emailed to you" on the bottom of the droplet create page I opened a ticket, here's how it went: "Your root password will be emailed to you" -- start -- Me Just seen this at the bottom of the "Create a droplet" page. You're kidding right? Them The root password is sent via email because it is the easiest and fastest way to get a user online and running a virtual server. We strongly recommend updating the root password after you login for the first time. We also have SSH keys support so you can add your SSH key to the server during creation in which case no email is sent and instead the SSH keys are added under the root user for more secure access. Thanks, Me Just added an SSH key and you're quite right, I retract my blunt reaction. I must say however that I still think emailing them is a fairly terrible idea and I'm surprised your not worried about being found liable for a subsequent server hack. That aside, thank you for your swift response and for pointing out I can use my SSH key. I look forward to using DO more -- end -- N.B. I actually received their response twice from different agents suggesting it was a canned response Frankly I don't even think passwords should be an option, as on AWS (not that they're perfect)
- goodwink 14y ago
  But if you reset the root password it's still emailed, regardless of ssh key.
- danielweber 14y ago
  For lots of users, they have to send _something_. And while they could build a PGP-whatzits system for the small fraction of their people who use PGP, those people are already going to be immediately changing their passwords anyway. Heuristics like "keys are better than passwords" or "don't email passwords" are good heuristics, but they shouldn't become absolute rules.
  - viraptor 14y ago
    They don't have to send the password. Allowing users to submit their password over https and storing only the hashed version is another option.
martinced 14y ago
"The successful root login followed only one unsuccessful attempt" Once a system is compromised by a root exploit what makes you think that any information that this system is giving is true? While it may be likely seen the circumstances it is by no way certain. For all we know it may have been a bruteforce attempt which, once in, got disguised as a known-root-password attack. As long as people are going to think that a compromised system is actually giving true information about what happened we'll be in big trouble. Or OP tells us that SSH login and SSH login attemps are logged automatically on another server which hasn't been compromised and then it's a different story...
- goodwink 14y ago
  The brute force attack would have been disallowed due to fail2ban being enabled.
  - martinced 14y ago
    fail2ban only blocks brute-force attacks from similar IPs right? What if a botnet is used to do the brute-force, does fail2ban lock everybody out, effectively making fail2ban a tool that can be used for Denial of Service? All I'd have to do now is to hammer your system with SSH attempts from my botnet and you can't log in anymore...
    - brokentone 14y ago
      If my IP is on your botnet than I've got bigger issues than not getting into my server. Namely you stealing my password and you getting into my server.
      - andybak 14y ago
        I think he means that fail2ban can't be effective against botnets attacks because if it didn't take IPs into account then it becomes a DDOS tool. Therefore it must take IPs into account. Therefore it's not protection against brute-force password guessing from a botnet.
kamme 14y ago
Too bad the author makes it sound like digitalocean has severe problems while his setup was clearly flawed. The really sad part is that this kind problem will unfortunately only grow as vps systems become cheaper and cheaper. People with less knowledge will set up their own stack and not think of the consequences... I wonder what hosting providers will come up with to tackle this problem.
- gregd 14y ago
  Force a root password change on first login?
- goodwink 14y ago
  My setup was clearly flawed and I acknowledged that in the message. Their system /does/ have serious problems though since someone is likely intercepting their password reset emails or else accessing their root password database which shouldn't even exist. I'd say those are real problems regardless of the poor config of the machine (which I readily accept my mea culpas for).
  - DanOWar 14y ago
    Was fail2ban even working? After a reformat, did you install fail2ban manually, or from a repo? Are you using syslog or rsyslog? Each's log format is slightly different, meaning you have to edit the filter to accomodate. The base install filter didn't even work correctly for me on a fresh CentOS re-image. Also, the latest version on the website is v0.8.7.1, but on CentOS epel it's v0.8.4.
  - kamme 14y ago
    Well, to be honest, every setup has serious problems, it just depends how far you want it to go. 100% Secure doesn't exist and while it may look like bad service from your point of view, truth is I've seen other vps solutions do the same thing. The issue here is that people with experience in system administration change the default password and set up key based authentication and try not to rely on password management from others. It's a shame your box got hacked, but immediately jumping to the conclusion the whole of digitalocean has a root exploit in the wild is a bit much imho...
andyhmltn 14y ago
Is it not possible your email was hacked? I know from my experience with DO, they email you the root password.
- goodwink 14y ago
  The article mentions that this was verified to not be the problem.
goodwink 14y ago
To answer some common questions about this article: Yes, I know my setup was very flawed. It was a test machine and in the midst of having automated config scripts written for it which were being tested. This is why the machine was vulnerable to the attack, but this does not diminish the importance of the underlying compromise it exposes in DigitialOcean's setup. The point of the article is that someone is able to gain access to DigitalOcean password reset emails or a database of root passwords which shouldn't exist, but seems to given that they set your root password back to a previously reset value after you rebuild your server from the base OS image.
brokentone 14y ago
TL;DR: Dude's server got hacked with a password-based SSH login. Doesn't definitively find source of hack. Doesn't like host security procedures he could have circumvented (changing password, disabling password-login). Posts on HN: ZOMG, ROOT VULN IN WILD! Edit: Disclosure, I met one of the co-founders at SXSW. Seem like cool dudes.
- goodwink 14y ago
  I think you missed the part where they retained a root password over a machine rebuild.
brianbreslin 14y ago
Off Topic: I met the founders at SXSW, and was wondering what you guys thought of their product? $5/month for a simple ssd vps seems like a good deal (was thinking of running a single wordpress site off of it).

38 points by goodwink 14y ago

I think the lesson we're all learning here is that if you don't do your own Sysadmin/devops then you're almost guaranteed to be dealing with imperfect systems.

gpcz 14y ago
I've got bad news for you about your perfect system...
ohboyacomment 14y ago
If you do your own sysadmin/devops, you are certainly guaranteed to be dealing with imperfect systems. (It's very likely worse to do your own unless you have the resources to do it right. Hosting companies have scale. Not defending DigitalOcean here, just challenging your assertion.)

Goranek 14y ago

This makes sense, i got my account blocked for spamming via SMTP, and i didn't send a single mail. It took 5 support letters to get my account unblocked, and the solution was to block SMTP ports... I didn't have anything installed !!! Clear Ubuntu image

deleted 14y ago
[deleted]
- ohboyacomment 14y ago
  What is it, exactly, that you expect to be running and 'default open vulnerable' on a fresh Ubuntu install? On my fresh Ubuntu deploys at Linode, the only network service listening is SSH. (Lest we forget that an open port is not a security vulnerability until a vulnerable service is listening on it, but I'll leave that assertion alone.) EDIT: The cowardly deleted comment chided the poster for not securing his system and mentioned 'default open vulnerable' services.
  - deleted 14y ago
    [deleted]
    - ohboyacomment 14y ago
      Oh, your comment is specific to this DigitalOcean scenario now? Because when you left it, it was phrased as a generality for systems administration, where you "FTFY"'d (God, I hate that) the person you are replying to in order to mock him for not securing his base Linux install. As if there's a hell of a lot to do out of the box. You're probably a -P INPUT DROP kind of guy, aren't you? Path MTU discovery and ping replies be damned? The person you are replying to is not the original poster of this scenario, and the person you are replying to very likely did not trigger the exact same scenario in his dealings. I do not believe one is related to the other, but nor did you, because you were careful not to mention anything related to this scenario at all in your dim, mean-spirited reply. So you're assuming that his root password was flown over e-mail with no evidence to that effect. (A root password flown over e-mail is a vulnerability, but an external one, and has nothing to do with securing a Linux install, which was the gist of your comment.) Couldn't even finish my comment before you turned gray and deleted out. Sad. I wish you the best of luck in your cheap karma adventure.
      - Goranek 14y ago
        Wait what!? I was talking about Digital Ocean.

zalew 14y ago

> it is likely that if you reset the root password via the web interface and don't change it afterwards that you are vulnerable shocker.

goodwink 14y ago
The shocker is that this is true even if you rebuild the box from the scratch image.

namidark 14y ago

You left root login enabled and kept passwords on. And you're upset you got compromised? You can also enable keys in the control panel and you won't have to deal with passwords being emailed. Those are the first few things you should be checking when setting up a new server (disable password logins and only allow keys).

goodwink 14y ago
I'm not upset I got compromised, it was very likely given my configuration, I acknowledge that. I did have keys enabled so the password wasn't email initially, it was only emailed as a result of a root password reset. The article is about what the compromise means about DigitalOcean not about my one box.

gregd 14y ago

Aren't we jumping the gun a bit by saying, "DigitalOcean Root Vulnerability in the Wild"? I'm not yet comfortable with that conclusion...

readme 14y ago
I agree. To me it sounds more like root vulnerability on OPs system. He did not change his password after using the automated password reset, knowing the password was sent in plain text.
- goodwink 14y ago
  No I didn't change the password. Instead I rebuilt the machine from the base OS image. Would you expect that a new system with the disk wiped would retain the old root password? I certainly wouldn't. Where were they keeping it?
  - bpicolo 14y ago
    Whenever I do a rebuild on a droplet it sends me a new root password in the email.
    - goodwink 14y ago
      I have ssh keys set up so it doesn't do this. Instead what it seems to do is reuse your old root password for the new image.
  - readme 14y ago
    Sorry -- my fault. I skimmed your article and I guess I missed this part. That is a bit ridiculous that they retained your password. Still though, logging in after rebuilding and configuring your system would have been a sane move.
  - wfn 14y ago
    > No I didn't change the password. Instead I rebuilt the machine from the base OS image. Probably not strictly related (it would seem the attack in question was via a compromised (on whoever's end) ssh password), but in any case good practice: always regenerate ssh keys after installation from a base image. Some OS images may contain pregenerated keypairs (private key is usually at ~/.ssh/id_rsa ), which should always be regenerated. (This bit is most probably not actually related to the issue at hand, but IAC should not be forgotten.)

thomseddon 14y ago

As soon as I saw the text "Your root password will be emailed to you" on the bottom of the droplet create page I opened a ticket, here's how it went: "Your root password will be emailed to you" -- start -- Me Just seen this at the bottom of the "Create a droplet" page. You're kidding right? Them The root password is sent via email because it is the easiest and fastest way to get a user online and running a virtual server. We strongly recommend updating the root password after you login for the first time. We also have SSH keys support so you can add your SSH key to the server during creation in which case no email is sent and instead the SSH keys are added under the root user for more secure access. Thanks, Me Just added an SSH key and you're quite right, I retract my blunt reaction. I must say however that I still think emailing them is a fairly terrible idea and I'm surprised your not worried about being found liable for a subsequent server hack. That aside, thank you for your swift response and for pointing out I can use my SSH key. I look forward to using DO more -- end -- N.B. I actually received their response twice from different agents suggesting it was a canned response Frankly I don't even think passwords should be an option, as on AWS (not that they're perfect)

goodwink 14y ago
But if you reset the root password it's still emailed, regardless of ssh key.
danielweber 14y ago
For lots of users, they have to send _something_. And while they could build a PGP-whatzits system for the small fraction of their people who use PGP, those people are already going to be immediately changing their passwords anyway. Heuristics like "keys are better than passwords" or "don't email passwords" are good heuristics, but they shouldn't become absolute rules.
- viraptor 14y ago
  They don't have to send the password. Allowing users to submit their password over https and storing only the hashed version is another option.

martinced 14y ago

"The successful root login followed only one unsuccessful attempt" Once a system is compromised by a root exploit what makes you think that any information that this system is giving is true? While it may be likely seen the circumstances it is by no way certain. For all we know it may have been a bruteforce attempt which, once in, got disguised as a known-root-password attack. As long as people are going to think that a compromised system is actually giving true information about what happened we'll be in big trouble. Or OP tells us that SSH login and SSH login attemps are logged automatically on another server which hasn't been compromised and then it's a different story...

goodwink 14y ago
The brute force attack would have been disallowed due to fail2ban being enabled.
- martinced 14y ago
  fail2ban only blocks brute-force attacks from similar IPs right? What if a botnet is used to do the brute-force, does fail2ban lock everybody out, effectively making fail2ban a tool that can be used for Denial of Service? All I'd have to do now is to hammer your system with SSH attempts from my botnet and you can't log in anymore...
  - brokentone 14y ago
    If my IP is on your botnet than I've got bigger issues than not getting into my server. Namely you stealing my password and you getting into my server.
    - andybak 14y ago
      I think he means that fail2ban can't be effective against botnets attacks because if it didn't take IPs into account then it becomes a DDOS tool. Therefore it must take IPs into account. Therefore it's not protection against brute-force password guessing from a botnet.

kamme 14y ago

Too bad the author makes it sound like digitalocean has severe problems while his setup was clearly flawed. The really sad part is that this kind problem will unfortunately only grow as vps systems become cheaper and cheaper. People with less knowledge will set up their own stack and not think of the consequences... I wonder what hosting providers will come up with to tackle this problem.

gregd 14y ago
Force a root password change on first login?
goodwink 14y ago
My setup was clearly flawed and I acknowledged that in the message. Their system /does/ have serious problems though since someone is likely intercepting their password reset emails or else accessing their root password database which shouldn't even exist. I'd say those are real problems regardless of the poor config of the machine (which I readily accept my mea culpas for).
- DanOWar 14y ago
  Was fail2ban even working? After a reformat, did you install fail2ban manually, or from a repo? Are you using syslog or rsyslog? Each's log format is slightly different, meaning you have to edit the filter to accomodate. The base install filter didn't even work correctly for me on a fresh CentOS re-image. Also, the latest version on the website is v0.8.7.1, but on CentOS epel it's v0.8.4.
- kamme 14y ago
  Well, to be honest, every setup has serious problems, it just depends how far you want it to go. 100% Secure doesn't exist and while it may look like bad service from your point of view, truth is I've seen other vps solutions do the same thing. The issue here is that people with experience in system administration change the default password and set up key based authentication and try not to rely on password management from others. It's a shame your box got hacked, but immediately jumping to the conclusion the whole of digitalocean has a root exploit in the wild is a bit much imho...

andyhmltn 14y ago

Is it not possible your email was hacked? I know from my experience with DO, they email you the root password.

goodwink 14y ago
The article mentions that this was verified to not be the problem.

goodwink 14y ago

To answer some common questions about this article: Yes, I know my setup was very flawed. It was a test machine and in the midst of having automated config scripts written for it which were being tested. This is why the machine was vulnerable to the attack, but this does not diminish the importance of the underlying compromise it exposes in DigitialOcean's setup. The point of the article is that someone is able to gain access to DigitalOcean password reset emails or a database of root passwords which shouldn't exist, but seems to given that they set your root password back to a previously reset value after you rebuild your server from the base OS image.

brokentone 14y ago

TL;DR: Dude's server got hacked with a password-based SSH login. Doesn't definitively find source of hack. Doesn't like host security procedures he could have circumvented (changing password, disabling password-login). Posts on HN: ZOMG, ROOT VULN IN WILD! Edit: Disclosure, I met one of the co-founders at SXSW. Seem like cool dudes.

goodwink 14y ago
I think you missed the part where they retained a root password over a machine rebuild.

brianbreslin 14y ago

Off Topic: I met the founders at SXSW, and was wondering what you guys thought of their product? $5/month for a simple ssd vps seems like a good deal (was thinking of running a single wordpress site off of it).