11 ms·

Identical Droplets in the DigitalOcean: Regenerate your Ubuntu SSH Host Keys now

285 points by jlund 13y ago

mey 13y ago
I must say, I'm impressed with how this was handled both by the original researcher and DigitalOcean.
- ihsw 13y ago
  Responsible disclosure can benefit us all, unfortunately some vendors -- SaaS, PaaS, physical, or otherwise -- use their legal departments as blunt weapons to needlessly attack well-meaning security researchers. In case anybody is wondering, I'm referring to Volkswagen. http://www.engadget.com/2013/07/29/uk-court-volkswagen-megamos-crypto/ http://www.engadget.com/2013/07/29/uk-court-volkswagen-megam... Personally I'm an advocate of full, anonymous, and public disclosure.
  - suhailpatel 13y ago
    I too am an advocate of full public disclosure but the concerns by Volkswagen are warranted and fixes to the issue can not be deployed as easily as DigitalOcean has done here. I don't agree with the gag order from the UK courts but it isn't totally unjustified.
  - philtar 13y ago
    That's not a fair comparison. DigitalOcean just needs to update some stuff to fix this issue. VW would need to recall millions of vehicles (10s billions $). You would do the same thing if you were in their position and had shareholders to worry about.
    - mikeash 13y ago
      No, I certainly would not do the same thing, and to suggest otherwise is an insult. Let's not excuse bad behavior with this misguided idea that we're all equally bad.
      - smtddr 13y ago
        So what is VW suppose to do? I'm actually truly curious. This flaw apparently will unlock many expensive cars. These cars' system cannot be replaced as quickly/cheaply as an sshd binary in a linux OS. I'm very curious, what other action could they have taken. They need him to be silent so they can figure out how to fix it before he make it public right? Is letting the public know the detailed exploit more important than the potential problems of the info being public?
        mikeash 13y ago
        VW needs to come up with an immediate workaround that they can publish to owners or allow dealers to quickly hack in, then come up with a permanent fix after that buys them some time. The immediate workaround may not be possible. In that case, they're just screwed. A company is not entitled to be able to save themselves from the consequences of their past fuckups in all situations. Sometimes, a mistake costs a lot of money or even kills the company. Perhaps this is one. I find it unlikely that it's impossible to disable the keyless entry system on the cars in question. Surely there is some fuse or wire that can be pulled to shut it off. But it ultimately doesn't matter. Finding a workaround quickly is what they need to do, and if they can't do it, that's not his problem.
        vacri 13y ago
        Your new solution of "fuck the company" actually harms the enduser even more. Some of them get the updated lock... then the company goes out of business, and now none of them can get official parts for their vehicles. It seems a solution for an ideal world, not the actual world.
        mikeash 13y ago
        You'll note that I proposed other solutions first. To repeat myself: it is highly unlikely that there is not some possible workaround that temporarily disables the system, even if it's something as brute as snipping a wire. Even in the absolute worst case that the vulnerability is somehow built into the very fabric of the car, you can still secure it by removing all valuables from the interior and then clamping a wheel with a boot. Remove the boot once you figure out a fix. Inconvenient to the owner to be sure, but not impossible to deal with. Harm to the end user is not my priority. Harm to society is, and it's clear to me that the long-term chilling effects on academic research far outweigh any temporary harm from VM issuing a recall or even going bankrupt. The alternative is to say that entity B should suffer from a restriction on their free speech simply because entity A, due to their own negligence, finds it excessively costly. There are so many different ways this could be handled other than "threaten to throw the researcher in jail if he doesn't shut up". But they are all more inconvenient and costly to VW. One can understand why, then, VW would go for the "threaten" option, if we think of VW as a sort of non-moral profit-optimizing organism. But I certainly can't understand why anyone would defend it, let alone say that we would do the same thing.
    - glesica 13y ago
      I would absolutely not do what they have done. But at the same time, I will probably never be the CEO of a large company. I suspect there is at least a weak causal relationship at work here...
    - ihsw 13y ago
      It's the cost of doing business. They're putting insecure software into hundreds of thousands of cars, and every owner of that car has no control over the software that's running on them. Imagine if we applied the same logic to phones and other devices -- I wouldn't be surprised if you personally would be offended at the idea that you have little recourse over your phone being hacked remotely and you can't do a damn thing about it because the mobile handset manufacturer locked it down. Thankfully phones are subsidized, ubiquitous, and cheap, so you can take your phone anywhere and get it fixed/replaced. This is the future folks: locked down devices that you have no control over. Caveat: we're all plugging our phones into these insecure systems too. Wrap your brain around that for a second to see where I'm going with this.
- specto 13y ago
  Except I informed them of this issue in January of this year http://imgur.com/GTi2UxJ http://imgur.com/GTi2UxJ Apparently they ignored it :)
  - kintamanimatt 13y ago
    That worries me far more than the actual security issue. Security issues happen to everybody, but so long as too many don't occur, it's the response that shapes my ongoing confidence in that company or product. What's happening on the left side? Is that you or the rep?
    - specto 13y ago
      I blocked out my name, the rep is the one with a picture. edit: actually I didn't realize there was a skype window up when I took the screenshot, thanks for warning me...
  - mey 13y ago
    Ok, that is disturbing...
    - specto 13y ago
      Honestly, I figured they would realize how important it would be to fix this so I didn't follow up on it once I fixed my own images.
      - coldpie 13y ago
        It's a reality of doing tech support. You get a flood of garbage information ("Hi, I can't access your web page, I get a 404 error. My system has 8 GB of RAM and an Intel 4700K and blah blah blah..."), and have to do your best to sort through and solve the user's problem. Your ticket had two problems described. The tech probably didn't understand the significance of the first problem, and so just discarded the information. Then she answered your second question. When you've got 100 tickets to sort through in your 8 hour day, you simply have to make some compromises on the thoroughness of your response. To get her attention, it would have been better to explain a little about what the consequences are, and request that she have a developer follow up. Make it clear that it's a major security failure and could lead to compromised VMs. Then open a second ticket for your other issue.
- ceol 13y ago
  Except I don't see an email from DO or a notification when I log in to the admin panel. So if I didn't check HN at this exact time and saw this article, I would have no idea. It's not a huge deal to me, but if Linode did the same thing, you all would be foaming at the mouth. Just thought I would point this out.
  - tallanvor 13y ago
    Do you have any Ubuntu instances running or saved? If not, then they would have no reason to notify you of the issue.
    - hexedpackets 13y ago
      I have a few Ubuntu instance and didn't receive any notification.
    - damncabbage 13y ago
      I've had a Ubuntu instance running for three months now; no notification.
    - rschmitty 13y ago
      running an Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-23-virtual x86_64) for 2 months, no notice yet for me
makomk 13y ago
This is now one of the first things I check when setting up a new VPS or other VM instance, because it's really common.
- voltagex_ 13y ago
  I think I need a list of things to check. This sounds pretty scriptable though: * SSH Host Key * SSH Authorized Keys * SSH PermitRootLogin * Disabling password auth in favour of keys * Security updates from the distro * SELinux (maybe?) Anything else?
  - icebraining 13y ago
    The difference is that with most of those, you can create a snapshot of an instance and then duplicate the configuration, but Host keys are special, since they need to be re-created for each instance. In any case, there was discussion on those issues not long ago: https://news.ycombinator.com/item?id=5316093 https://news.ycombinator.com/item?id=5316093
  - patio11 13y ago
    a) Move SSH off port 22. (Really limited security gain when implementing the rest of the suggestions, but saves spam in your logs.) b) Software firewall via e.g. IPtables is generally the first thing I turn on after rebooting SSHd with the new settings. c) (Optional) Consider using an architecture where you have N boxes and SSH only listens on a local interface on N-1 boxes, with the Nth box running nothing but your VPN. (This is also a good architecture choice for admin consoles, folks. www.example.com resolves to a public IP, admin.example.com resolves to a private IP, so even if they're technically speaking on the same box/boxes you won't lose the admin console if someone unwisely uses the same password for a WordPress blog somewhere.)
    - dsl 13y ago
      My honeypots have been seeing scans on 2022, 2222, 3022, etc. for years now. You should be setting proper ACLs for 22 and not moving to another port.
schappim 13y ago
Props to the way you handled this. That's how you do responsible vulnerability disclosures!
joshmn 13y ago
Now that it's said, I did notice something strange once. I had loaded up an Ubuntu Desktop droplet with the purpose of checking something out through the browser on the node. The startup page was https://www.americanexpress.com/ https://www.americanexpress.com/ Since when is that default? Didn't think much of it at the time, but now... whoa.
davidhollander 13y ago
> After you have run those commands, simply restart the SSH daemon so it starts up with the new keys in place I believe if your version of OpenSSH is up to date, sshd will read the host key each time a session is opened and does not need to be restarted.
throwawayh4xor 13y ago
Just verified this is also the case with at least some AWS-hosted servers. Coupled with the fact that many people simply ignore the MITM warning that SSH throws, this is scary stuff.
- vacri 13y ago
  Curious. I just upgraded a small server to a medium, detaching the existing volume and reattaching it to the new instance, and it has given me a new fingerprint.
- novaleaf 13y ago
  unfortunately, devops (like me) need to ignore these as part of our work system. i need to automate deployment / reprovisioning of 30 digital ocean servers. as reprovisionined servers frequently use the same ip address, i always run into this. for me I had to disable the check :(
  - Titanous 13y ago
    By disabling that check, you are destroying a huge component of the security that SSH provides. Perhaps you could clean up the authorized_keys file as part of your teardown script?
  - gregf 13y ago
    I ran into this issue as well. As the person below mentioned, make this part of your teardown script. You can use ssh-keygen -R hostname to remove host from ~/.ssh/known_hosts.
  - karlkatzke 13y ago
    I've ended up in this situation too. We migrate between datacenters with duplicate hosts fairly frequently at my primary place of employment, and best practice is to use the cname of the service to access the server currently acting as the primary for it (e.g. foo.bar.com as opposed to foo.datacenter.bar.com). That leaves me with a lot of foo.bar.com entries to clean out of known_hosts, or a lot of spurious MITM errors. What I've done is added a whitelist to my .ssh/config to disable the alerts only for those hosts. The foo.datacenter.bar.com address (which I use often enough, usually when migrating it between datacenters) still alerts. And yes, I know I'm living in the 90s what with my datacenters and whatnot. They're kind of like regions... what? Oh... why, you... You kids, get off my lawn!
scottlinux 13y ago
I suspect this kind of thing happens with other companies, but can only speculate. Somewhat related: chicagovps gave me a 'fresh' gentoo vps, and the default provided root password was identical to the original one from several months ago. I assume it is one gentoo image with the same password (for all customers)?
druiid 13y ago
One good thing to note is that any VM image using cloud-init (a package for debian/rhel systems) should automagically generate a new host_key set for any new system image. Basically if you build a system image for EC2 or any system that uses the EC2 data format (like Openstack) for host instantiation, then you should install cloud-init. It would prevent something like this.
agwa 13y ago
SSH host keys are problematic on cloud servers, not just because of this problem, but also because if the cloud provider does the right thing and generates the SSH host key on the first boot, the key is generated when the system has very little entropy available. The primary sources of entropy on Linux are key/mouse input, disk latency, and network interrupts. There's obviously no keyboard/mouse on a server, and in an SSD environment like DigitalOcean, disk latency is quite uniform and thus useless as a source of entropy. Linux distros mitigate the cold boot entropy problem by saving some state from the RNG on shutdown (on Debian, it's saved in /var/lib/urandom/random-seed) and using it to seed the RNG on the next boot. On physical servers this obviously isn't available on the first boot, and on cloud servers, the provider often bakes the same random-seed file into all their images, so everyone gets the same seed on first boot (fortunately this doesn't harm security any more than having no random-seed file at all, but it doesn't help either). What cloud providers should really do is generate (from a good source of randomness) a distinct random-seed file for every server that's created, but I haven't seen any providers do this.
- niels_olson 13y ago
  It's slow, and manual, but this gives me one more reason to like prgmr, at least for my little side projects. You generate the keys, and send the public key to Luke. Your only access to the control layer is via public_key auth from that key you generated. Presumably on a laptop, with gobs of entropy available.
  - agwa 13y ago
    I believe you're confusing SSH host keys with the keys used for user authentication. The private SSH host key has to reside on the server, so it's either being generated on the server or you're sending Luke a private key. But I agree this sounds like a good way to handle user authentication.
  - lsc 13y ago
    as much as I appreciate the plug, that key I ask you for? that's to get you into the admin interface on my side. I don't mess with the ssh keys within your image. (host keys are auto-generated on first boot, just like a real server, which has the 'generating keys in a low-entropy environment' problem described above. The image doesn't come with anything in the authorized_keys file; though actually, I could pretty easily adjust the creation script to put the key you sent me in the authorized_keys file of root... but I don't really know the 'right' way to do that; everyone has preferences, so I leave it up to the user.) We've actually talked a lot about trying to write a xen driver to share a hardware entropy device, but it hasn't gone anywhere. (I mean, xen has 'vtpm' - a virtualized Trusted Platform Module. And nobody uses that, so why not a vrandom?)
foxhop 13y ago
So you are the reason I started getting these error messages, I noticed the change on June 2, great work. If you are still reviewing salt, I just wrote a post about salt-cloud and DigitalOcean that you should check out - Create your own fleet of servers with Digital Ocean and salt-cloud: http://russell.ballestrini.net/create-your-own-fleet-of-servers-with-digital-ocean-and-salt-cloud/ http://russell.ballestrini.net/create-your-own-fleet-of-serv...
sehrope 13y ago
Generating fresh keys aside, one thing I do with our AWS setup is whitelist the IPs that can connect to our SSH bastion host. This completely eliminates scripted port scans of the SSH server and makes the auth logs much more manageable. If our IP address changes (eg. ISP assigns a new one for the cable modem) then we just update the whitelist (and remove the old address). It's very infrequent. I could probably count the number of times I've done it on one hand. It might not be the most scalable setup but at our small size with everybody working from home it works great. The only slight hitch is updating it when traveling but even that isn't much of a problem. It takes a minute or two from the AWS console and its good to go. I recently took a look at digital ocean ($5 servers gives me ideas...) but didn't see a firewall option similar to the security group setup in AWS. If it does exist then I highly recommend it.
- matthewbadeau 13y ago
  This is a very good idea. It could be script-able using the AWS API[1] (though I haven't tried it yet). [1] http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-AuthorizeSecurityGroupIngress.html#ApiReference-query-AuthorizeSecurityGroupIngress-Examples http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiRef... [EDIT] The relevant example is 4th in the list.
- werkshy 13y ago
  With Digital Ocean you'd need to install a software firewall on the servers themselves, there is no API-configurable network-level firewall. I used 'ufw' which was quite easy to get started with (on Ubuntu), and replicated my AWS security group config pretty quickly. I added the ufw config to my host setup scripts so it happens automatically.
  - sehrope 13y ago
    The problem with the software one is you need a way to modify it when you can't access the instance. With AWS you do it from the admin console or APIs. If it's on the machine itself you'd have to know the IP to open up in advance or have someone at home base do it.
rwmj 13y ago
They should be using cloud-init or virt-sysprep[1] on new instances. In particular, it is vital that you give your new instances a unique random seed (which virt-sysprep can do). Also that you provide the virtio-rng to guests that support it. [1] http://libguestfs.org/virt-sysprep.1.html http://libguestfs.org/virt-sysprep.1.html
deleted 13y ago
[deleted]
joeblau 13y ago
Great find. I came from a heavy security background and moved to SV where it seems like security is an after thought. I spent many long days and nights STIGing RHEL boxes so I can appreciate this find. Also thanks for letting me know about Digital Ocean, their VPS looks promising and I think I might start using it.
Nux 13y ago
This is not the last of the problems we'll have with "the cloud", but I guess it's part of what makes it so exciting. :-) Many people, especially beginners, make the mistake of leaving the same SSH keys in a certain template or in a snapshot of a virtual machine that they later use as a template. There are a few files that you really, really need to wipe out from a wannabe image template: - /etc/ssh/* key* (for reasons explained in the parent article. stupid autoformatting, remove the space after the first asterisk) - /var/lib/random-seed (the seed used to initialise the random number generator. this is the location on CentOS) - /etc/udev/rules.d/70-persistent-net.rules (so that the VM's new NIC - with a new MAC - can use the same "eth0" name) People who want to do this more exhaustively can have a look at libguestfs and it's program virt-sysprep which does all of the above and more! http://libguestfs.org/virt-sysprep.1.html http://libguestfs.org/virt-sysprep.1.html
rlpb 13y ago
To avoid this kind of security problem, use providers that use official Ubuntu Cloud images only. If Canonical haven't certified the Ubuntu images you're using, then your provider could have done anything to them. You'll need some other way to determine their competence. Cowboy images like this are exactly the reason trademarks exist. Commercial providers who don't get certification are in fact violating Ubuntu's trademark by telling you that you are getting Ubuntu, when in fact you are getting a modified image which is possibly compromised (such as in this case).
- regularfry 13y ago
  How do I validate that my provider is actually providing an official Ubuntu Cloud image?
  - rlpb 13y ago
    Technically? I'm not sure that's possible. They own the hypervisor, so you have to trust them. This is why the Ubuntu trademark is so important.
    - regularfry 13y ago
      If I have to trust them because they own the hypervisor, the fact they claim to deliver a "certified" image buys me nothing in terms of security.
      - rlpb 13y ago
        That's demonstrably not the case. In this situation, it was only the modified image that accidentally introduced a vulnerability. The official image does not have this vulnerability. Had you been using an Ubuntu certified cloud, you wouldn't have been vulnerable.
        regularfry 13y ago
        I've got no way to check I am using an official image, other than knowing that at some point in the past, my provider bunged Canonical some cash for the "Ubuntu Certified(tm)" sticker. Without knowing a lot more about how certification works, that's not a particularly strong proof.
stevekemp 13y ago
We ran into similar problems on the hosting side; another surprise can be the debian-sys-maint password configure by the Debian mysql-server package.

285 points by jlund 13y ago

mey 13y ago

I must say, I'm impressed with how this was handled both by the original researcher and DigitalOcean.

ihsw 13y ago
Responsible disclosure can benefit us all, unfortunately some vendors -- SaaS, PaaS, physical, or otherwise -- use their legal departments as blunt weapons to needlessly attack well-meaning security researchers. In case anybody is wondering, I'm referring to Volkswagen. http://www.engadget.com/2013/07/29/uk-court-volkswagen-megamos-crypto/ http://www.engadget.com/2013/07/29/uk-court-volkswagen-megam... Personally I'm an advocate of full, anonymous, and public disclosure.
- suhailpatel 13y ago
  I too am an advocate of full public disclosure but the concerns by Volkswagen are warranted and fixes to the issue can not be deployed as easily as DigitalOcean has done here. I don't agree with the gag order from the UK courts but it isn't totally unjustified.
- philtar 13y ago
  That's not a fair comparison. DigitalOcean just needs to update some stuff to fix this issue. VW would need to recall millions of vehicles (10s billions $). You would do the same thing if you were in their position and had shareholders to worry about.
  - mikeash 13y ago
    No, I certainly would not do the same thing, and to suggest otherwise is an insult. Let's not excuse bad behavior with this misguided idea that we're all equally bad.
    - smtddr 13y ago
      So what is VW suppose to do? I'm actually truly curious. This flaw apparently will unlock many expensive cars. These cars' system cannot be replaced as quickly/cheaply as an sshd binary in a linux OS. I'm very curious, what other action could they have taken. They need him to be silent so they can figure out how to fix it before he make it public right? Is letting the public know the detailed exploit more important than the potential problems of the info being public?
      - mikeash 13y ago
        VW needs to come up with an immediate workaround that they can publish to owners or allow dealers to quickly hack in, then come up with a permanent fix after that buys them some time. The immediate workaround may not be possible. In that case, they're just screwed. A company is not entitled to be able to save themselves from the consequences of their past fuckups in all situations. Sometimes, a mistake costs a lot of money or even kills the company. Perhaps this is one. I find it unlikely that it's impossible to disable the keyless entry system on the cars in question. Surely there is some fuse or wire that can be pulled to shut it off. But it ultimately doesn't matter. Finding a workaround quickly is what they need to do, and if they can't do it, that's not his problem.
        vacri 13y ago
        Your new solution of "fuck the company" actually harms the enduser even more. Some of them get the updated lock... then the company goes out of business, and now none of them can get official parts for their vehicles. It seems a solution for an ideal world, not the actual world.
        mikeash 13y ago
        You'll note that I proposed other solutions first. To repeat myself: it is highly unlikely that there is not some possible workaround that temporarily disables the system, even if it's something as brute as snipping a wire. Even in the absolute worst case that the vulnerability is somehow built into the very fabric of the car, you can still secure it by removing all valuables from the interior and then clamping a wheel with a boot. Remove the boot once you figure out a fix. Inconvenient to the owner to be sure, but not impossible to deal with. Harm to the end user is not my priority. Harm to society is, and it's clear to me that the long-term chilling effects on academic research far outweigh any temporary harm from VM issuing a recall or even going bankrupt. The alternative is to say that entity B should suffer from a restriction on their free speech simply because entity A, due to their own negligence, finds it excessively costly. There are so many different ways this could be handled other than "threaten to throw the researcher in jail if he doesn't shut up". But they are all more inconvenient and costly to VW. One can understand why, then, VW would go for the "threaten" option, if we think of VW as a sort of non-moral profit-optimizing organism. But I certainly can't understand why anyone would defend it, let alone say that we would do the same thing.
  - glesica 13y ago
    I would absolutely not do what they have done. But at the same time, I will probably never be the CEO of a large company. I suspect there is at least a weak causal relationship at work here...
  - ihsw 13y ago
    It's the cost of doing business. They're putting insecure software into hundreds of thousands of cars, and every owner of that car has no control over the software that's running on them. Imagine if we applied the same logic to phones and other devices -- I wouldn't be surprised if you personally would be offended at the idea that you have little recourse over your phone being hacked remotely and you can't do a damn thing about it because the mobile handset manufacturer locked it down. Thankfully phones are subsidized, ubiquitous, and cheap, so you can take your phone anywhere and get it fixed/replaced. This is the future folks: locked down devices that you have no control over. Caveat: we're all plugging our phones into these insecure systems too. Wrap your brain around that for a second to see where I'm going with this.
specto 13y ago
Except I informed them of this issue in January of this year http://imgur.com/GTi2UxJ http://imgur.com/GTi2UxJ Apparently they ignored it :)
- kintamanimatt 13y ago
  That worries me far more than the actual security issue. Security issues happen to everybody, but so long as too many don't occur, it's the response that shapes my ongoing confidence in that company or product. What's happening on the left side? Is that you or the rep?
  - specto 13y ago
    I blocked out my name, the rep is the one with a picture. edit: actually I didn't realize there was a skype window up when I took the screenshot, thanks for warning me...
- mey 13y ago
  Ok, that is disturbing...
  - specto 13y ago
    Honestly, I figured they would realize how important it would be to fix this so I didn't follow up on it once I fixed my own images.
    - coldpie 13y ago
      It's a reality of doing tech support. You get a flood of garbage information ("Hi, I can't access your web page, I get a 404 error. My system has 8 GB of RAM and an Intel 4700K and blah blah blah..."), and have to do your best to sort through and solve the user's problem. Your ticket had two problems described. The tech probably didn't understand the significance of the first problem, and so just discarded the information. Then she answered your second question. When you've got 100 tickets to sort through in your 8 hour day, you simply have to make some compromises on the thoroughness of your response. To get her attention, it would have been better to explain a little about what the consequences are, and request that she have a developer follow up. Make it clear that it's a major security failure and could lead to compromised VMs. Then open a second ticket for your other issue.
ceol 13y ago
Except I don't see an email from DO or a notification when I log in to the admin panel. So if I didn't check HN at this exact time and saw this article, I would have no idea. It's not a huge deal to me, but if Linode did the same thing, you all would be foaming at the mouth. Just thought I would point this out.
- tallanvor 13y ago
  Do you have any Ubuntu instances running or saved? If not, then they would have no reason to notify you of the issue.
  - hexedpackets 13y ago
    I have a few Ubuntu instance and didn't receive any notification.
  - damncabbage 13y ago
    I've had a Ubuntu instance running for three months now; no notification.
  - rschmitty 13y ago
    running an Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-23-virtual x86_64) for 2 months, no notice yet for me

makomk 13y ago

This is now one of the first things I check when setting up a new VPS or other VM instance, because it's really common.

voltagex_ 13y ago
I think I need a list of things to check. This sounds pretty scriptable though: * SSH Host Key * SSH Authorized Keys * SSH PermitRootLogin * Disabling password auth in favour of keys * Security updates from the distro * SELinux (maybe?) Anything else?
- icebraining 13y ago
  The difference is that with most of those, you can create a snapshot of an instance and then duplicate the configuration, but Host keys are special, since they need to be re-created for each instance. In any case, there was discussion on those issues not long ago: https://news.ycombinator.com/item?id=5316093 https://news.ycombinator.com/item?id=5316093
- patio11 13y ago
  a) Move SSH off port 22. (Really limited security gain when implementing the rest of the suggestions, but saves spam in your logs.) b) Software firewall via e.g. IPtables is generally the first thing I turn on after rebooting SSHd with the new settings. c) (Optional) Consider using an architecture where you have N boxes and SSH only listens on a local interface on N-1 boxes, with the Nth box running nothing but your VPN. (This is also a good architecture choice for admin consoles, folks. www.example.com resolves to a public IP, admin.example.com resolves to a private IP, so even if they're technically speaking on the same box/boxes you won't lose the admin console if someone unwisely uses the same password for a WordPress blog somewhere.)
  - dsl 13y ago
    My honeypots have been seeing scans on 2022, 2222, 3022, etc. for years now. You should be setting proper ACLs for 22 and not moving to another port.

schappim 13y ago

Props to the way you handled this. That's how you do responsible vulnerability disclosures!

joshmn 13y ago

Now that it's said, I did notice something strange once. I had loaded up an Ubuntu Desktop droplet with the purpose of checking something out through the browser on the node. The startup page was https://www.americanexpress.com/ https://www.americanexpress.com/ Since when is that default? Didn't think much of it at the time, but now... whoa.

davidhollander 13y ago

> After you have run those commands, simply restart the SSH daemon so it starts up with the new keys in place I believe if your version of OpenSSH is up to date, sshd will read the host key each time a session is opened and does not need to be restarted.

throwawayh4xor 13y ago

Just verified this is also the case with at least some AWS-hosted servers. Coupled with the fact that many people simply ignore the MITM warning that SSH throws, this is scary stuff.

vacri 13y ago
Curious. I just upgraded a small server to a medium, detaching the existing volume and reattaching it to the new instance, and it has given me a new fingerprint.
novaleaf 13y ago
unfortunately, devops (like me) need to ignore these as part of our work system. i need to automate deployment / reprovisioning of 30 digital ocean servers. as reprovisionined servers frequently use the same ip address, i always run into this. for me I had to disable the check :(
- Titanous 13y ago
  By disabling that check, you are destroying a huge component of the security that SSH provides. Perhaps you could clean up the authorized_keys file as part of your teardown script?
- gregf 13y ago
  I ran into this issue as well. As the person below mentioned, make this part of your teardown script. You can use ssh-keygen -R hostname to remove host from ~/.ssh/known_hosts.
- karlkatzke 13y ago
  I've ended up in this situation too. We migrate between datacenters with duplicate hosts fairly frequently at my primary place of employment, and best practice is to use the cname of the service to access the server currently acting as the primary for it (e.g. foo.bar.com as opposed to foo.datacenter.bar.com). That leaves me with a lot of foo.bar.com entries to clean out of known_hosts, or a lot of spurious MITM errors. What I've done is added a whitelist to my .ssh/config to disable the alerts only for those hosts. The foo.datacenter.bar.com address (which I use often enough, usually when migrating it between datacenters) still alerts. And yes, I know I'm living in the 90s what with my datacenters and whatnot. They're kind of like regions... what? Oh... why, you... You kids, get off my lawn!

scottlinux 13y ago

I suspect this kind of thing happens with other companies, but can only speculate. Somewhat related: chicagovps gave me a 'fresh' gentoo vps, and the default provided root password was identical to the original one from several months ago. I assume it is one gentoo image with the same password (for all customers)?

druiid 13y ago

One good thing to note is that any VM image using cloud-init (a package for debian/rhel systems) should automagically generate a new host_key set for any new system image. Basically if you build a system image for EC2 or any system that uses the EC2 data format (like Openstack) for host instantiation, then you should install cloud-init. It would prevent something like this.

agwa 13y ago

SSH host keys are problematic on cloud servers, not just because of this problem, but also because if the cloud provider does the right thing and generates the SSH host key on the first boot, the key is generated when the system has very little entropy available. The primary sources of entropy on Linux are key/mouse input, disk latency, and network interrupts. There's obviously no keyboard/mouse on a server, and in an SSD environment like DigitalOcean, disk latency is quite uniform and thus useless as a source of entropy. Linux distros mitigate the cold boot entropy problem by saving some state from the RNG on shutdown (on Debian, it's saved in /var/lib/urandom/random-seed) and using it to seed the RNG on the next boot. On physical servers this obviously isn't available on the first boot, and on cloud servers, the provider often bakes the same random-seed file into all their images, so everyone gets the same seed on first boot (fortunately this doesn't harm security any more than having no random-seed file at all, but it doesn't help either). What cloud providers should really do is generate (from a good source of randomness) a distinct random-seed file for every server that's created, but I haven't seen any providers do this.

niels_olson 13y ago
It's slow, and manual, but this gives me one more reason to like prgmr, at least for my little side projects. You generate the keys, and send the public key to Luke. Your only access to the control layer is via public_key auth from that key you generated. Presumably on a laptop, with gobs of entropy available.
- agwa 13y ago
  I believe you're confusing SSH host keys with the keys used for user authentication. The private SSH host key has to reside on the server, so it's either being generated on the server or you're sending Luke a private key. But I agree this sounds like a good way to handle user authentication.
- lsc 13y ago
  as much as I appreciate the plug, that key I ask you for? that's to get you into the admin interface on my side. I don't mess with the ssh keys within your image. (host keys are auto-generated on first boot, just like a real server, which has the 'generating keys in a low-entropy environment' problem described above. The image doesn't come with anything in the authorized_keys file; though actually, I could pretty easily adjust the creation script to put the key you sent me in the authorized_keys file of root... but I don't really know the 'right' way to do that; everyone has preferences, so I leave it up to the user.) We've actually talked a lot about trying to write a xen driver to share a hardware entropy device, but it hasn't gone anywhere. (I mean, xen has 'vtpm' - a virtualized Trusted Platform Module. And nobody uses that, so why not a vrandom?)

foxhop 13y ago

So you are the reason I started getting these error messages, I noticed the change on June 2, great work. If you are still reviewing salt, I just wrote a post about salt-cloud and DigitalOcean that you should check out - Create your own fleet of servers with Digital Ocean and salt-cloud: http://russell.ballestrini.net/create-your-own-fleet-of-servers-with-digital-ocean-and-salt-cloud/ http://russell.ballestrini.net/create-your-own-fleet-of-serv...

sehrope 13y ago

Generating fresh keys aside, one thing I do with our AWS setup is whitelist the IPs that can connect to our SSH bastion host. This completely eliminates scripted port scans of the SSH server and makes the auth logs much more manageable. If our IP address changes (eg. ISP assigns a new one for the cable modem) then we just update the whitelist (and remove the old address). It's very infrequent. I could probably count the number of times I've done it on one hand. It might not be the most scalable setup but at our small size with everybody working from home it works great. The only slight hitch is updating it when traveling but even that isn't much of a problem. It takes a minute or two from the AWS console and its good to go. I recently took a look at digital ocean ($5 servers gives me ideas...) but didn't see a firewall option similar to the security group setup in AWS. If it does exist then I highly recommend it.

matthewbadeau 13y ago
This is a very good idea. It could be script-able using the AWS API[1] (though I haven't tried it yet). [1] http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-AuthorizeSecurityGroupIngress.html#ApiReference-query-AuthorizeSecurityGroupIngress-Examples http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiRef... [EDIT] The relevant example is 4th in the list.
werkshy 13y ago
With Digital Ocean you'd need to install a software firewall on the servers themselves, there is no API-configurable network-level firewall. I used 'ufw' which was quite easy to get started with (on Ubuntu), and replicated my AWS security group config pretty quickly. I added the ufw config to my host setup scripts so it happens automatically.
- sehrope 13y ago
  The problem with the software one is you need a way to modify it when you can't access the instance. With AWS you do it from the admin console or APIs. If it's on the machine itself you'd have to know the IP to open up in advance or have someone at home base do it.

rwmj 13y ago

They should be using cloud-init or virt-sysprep[1] on new instances. In particular, it is vital that you give your new instances a unique random seed (which virt-sysprep can do). Also that you provide the virtio-rng to guests that support it. [1] http://libguestfs.org/virt-sysprep.1.html http://libguestfs.org/virt-sysprep.1.html

deleted 13y ago

[deleted]

joeblau 13y ago

Great find. I came from a heavy security background and moved to SV where it seems like security is an after thought. I spent many long days and nights STIGing RHEL boxes so I can appreciate this find. Also thanks for letting me know about Digital Ocean, their VPS looks promising and I think I might start using it.

Nux 13y ago

This is not the last of the problems we'll have with "the cloud", but I guess it's part of what makes it so exciting. :-) Many people, especially beginners, make the mistake of leaving the same SSH keys in a certain template or in a snapshot of a virtual machine that they later use as a template. There are a few files that you really, really need to wipe out from a wannabe image template: - /etc/ssh/* key* (for reasons explained in the parent article. stupid autoformatting, remove the space after the first asterisk) - /var/lib/random-seed (the seed used to initialise the random number generator. this is the location on CentOS) - /etc/udev/rules.d/70-persistent-net.rules (so that the VM's new NIC - with a new MAC - can use the same "eth0" name) People who want to do this more exhaustively can have a look at libguestfs and it's program virt-sysprep which does all of the above and more! http://libguestfs.org/virt-sysprep.1.html http://libguestfs.org/virt-sysprep.1.html

rlpb 13y ago

To avoid this kind of security problem, use providers that use official Ubuntu Cloud images only. If Canonical haven't certified the Ubuntu images you're using, then your provider could have done anything to them. You'll need some other way to determine their competence. Cowboy images like this are exactly the reason trademarks exist. Commercial providers who don't get certification are in fact violating Ubuntu's trademark by telling you that you are getting Ubuntu, when in fact you are getting a modified image which is possibly compromised (such as in this case).

regularfry 13y ago
How do I validate that my provider is actually providing an official Ubuntu Cloud image?
- rlpb 13y ago
  Technically? I'm not sure that's possible. They own the hypervisor, so you have to trust them. This is why the Ubuntu trademark is so important.
  - regularfry 13y ago
    If I have to trust them because they own the hypervisor, the fact they claim to deliver a "certified" image buys me nothing in terms of security.
    - rlpb 13y ago
      That's demonstrably not the case. In this situation, it was only the modified image that accidentally introduced a vulnerability. The official image does not have this vulnerability. Had you been using an Ubuntu certified cloud, you wouldn't have been vulnerable.
      - regularfry 13y ago
        I've got no way to check I am using an official image, other than knowing that at some point in the past, my provider bunged Canonical some cash for the "Ubuntu Certified(tm)" sticker. Without knowing a lot more about how certification works, that's not a particularly strong proof.

stevekemp 13y ago

We ran into similar problems on the hosting side; another surprise can be the debian-sys-maint password configure by the Debian mysql-server package.