14 ms·

Non-Unique SSH Host Keys Ed25519 on Hetzner

127 points by jonaslejon 11y ago

zx2c4 11y ago
Yet another reason not to use images provided by your hosting company. In anycase, if you are relying on hosting company images (complete with their root backdoor for "performance monitoring control panel!"), it's not like you care very much about your servers' security in the first place...
- Ded7xSEoPKYNsDd 11y ago
  If you really believe your hosting provider puts root backdoors in your systems, you really shouldn't run anything in a hypervisor controlled by them.
  - TimWolla 11y ago
    http://help.ovh.com/InstallOVHkey http://help.ovh.com/InstallOVHkey > In order to be able to intervene on your dedicated server without your root password, the automatic installation of ssh key is done. Only authorized employees of OVH will use it. It is not a gap in security, contrary, thanks to this OVH has root rights to your server and may identify the problems with your server. When you request an intervention, we need to have access to ssh.
    - mschuster91 11y ago
      Sounds to me like what is needed for a managed root-server. Actually it's kinda cool because you can enjoy timely upgrades, security fixes and response to ddos attacks while at the same time you also have the flexibility to install needed software without going through a change request workflow...
    - pquerna 11y ago
      A SSH private key is just some bytes. Once you have access to it, you have access to it forever in the future. RSA doesn't care if you are an "authorized employee". A static SSH key for a single human is already risky over a long time period, but for a key that is shared between multiple humans... wow.
    - niij 11y ago
      This is an opt-in process, not an opt-out. They don't have authorized keys on your server by default, only instructions to add them if needed.
      - Wilya 11y ago
        Unless it has changed recently, it's opt-out. The instructions are mostly for re-adding the key if you have removed it.
- dozzie 11y ago
  > Yet another reason not to use images provided by your hosting company. Wait, you mean you clone your images instead of installing them from scratch with some automated system? How barbarous!
- otterley 11y ago
  It's not as though it's impossible to make a secure image. Mistakes can and do happen, though. Better to own up to it and fix it than to hide the problem.
jjuhl 11y ago
At least they are aware of the issue and warning their customers. A lot of companies would just fix the issue (or not even that) and try to silently sweep it under the rug.
- kuschku 11y ago
  Their business is being a trustworthy, high quality service. Sweeping it under the rug would destroy their business. (Not that competitors wouldn’t have done it anyway)
  - chx 11y ago
    > Their business is being a trustworthy, high quality service. Really. Here I thought their business is renting out servers made from desktop parts for insanely low amounts of money. I have no problems with this strategy, I enjoy my 42EUR a month 2x3TB HDD + 2x120GB SSD, 16GB i7 2600 server, thanks much. It runs hobby sites and such. I know what I bought and what I can expect.
    - 5ersi 11y ago
      They also offer reasonably priced server-class dedicated servers with Xeon processors, ECC memory and datacenter-series SSDs.
      - merb 11y ago
        "reasonably priced". You never needed a private network with 3 nodes.
        corford 11y ago
        €12,61/month extra per server and a one off ~€25 for the switch is hardly extortionate given what they charge for servers...
        merb 11y ago
ymse 11y ago
This is probably pretty common. When I first deployed virtualization at a previous job ~5 years ago, it took almost two years before I realized all images used the same host keys. Can someone versed in cryptography comment on whether this allows passive eavesdropping? Active MITM sure, but I thought session keys would be unique.
- brians 11y ago
  You are right: it allows active impersonation, but not passive eavesdropping.
  - pflanze 11y ago
    > impersonation "Impersonification" of the server, you mean (inservication if that's a word). It would allow an attacker to operate a machine under their control that poses as mine. But then, for password based logins, that will actually be enough to impersonify the user as well (the attacker will create an ssh connection to the real server with the given password). But I wonder how things work with key based logins. I've ordered a server from Hetzner recently, I guess I'll just ditch it (and perhaps order a new one). BTW I haven't gotten any email notification from Hetzner about this yet.
    - snuxoll 11y ago
      Given that key based logins require signing a per-session token to verify the owner of the key is trying to connect (otherwise they would be useless since someone could just perform a replay attack) MITM'ing a key-authenticated session would allow you to perform attacks with that session, but it would not allow you to re-authenticate against that server later.
      - Buge 11y ago
        Would active MITM be possible with key login? I would think it should be impossible. If the symmetric session key is chosen by the server and encrypted with the client's public key, the attacker won't know it and cannot mess with the session. The only thing the attacker can do is completely impersonate the server. But the attacker cannot communicate with the actual server at all.
zymhan 11y ago
Wasn't there a cloud hosting provider that didn't even bother cleaning out the SSH known_hosts file?
KenanSulayman 11y ago
Companies that let their interns do the crypto stuff ¯\_(ツ)_/¯
sarciszewski 11y ago
If anyone works for a hosting provider, please write a script that checks all of your customers' SSH ports for identical fingerprints. If you have any collisions, you have a problem.
- jedisct1 11y ago
  Except that performing network scans, even on your own network, is illegal in many countries. Even in countries where this is legal, providers cannot take the risk of running a scan that could possibly crash a customer application, or they could be sued for that.
  - switch007 11y ago
    Which countries and under which laws? Which T&Cs do not have a blanket clause to indemnify the hosting company in such situations? If I accidentally reboot the virtualisation host, can my company be sued for making the application "crash"?
    - AdamGibbins 11y ago
      There's multiple examples of legal issues mentioned on https://nmap.org/book/legal-issues.html https://nmap.org/book/legal-issues.html
  - sarciszewski 11y ago
    Huh. I hadn't even thought of that. If connecting to port 22 and slurping the host fingerprint crashes a customer application... I don't even know. Like, how is their app even running?
    - devonkim 11y ago
      I can imagine a scenario where a user that logs into a machine and tries to read a very sensitive file that they shouldn't have access to (should only be read by sshd and a specific automation user) trips an alarm via hooks in SELinux or something similar that causes the host to shut down immediately and security admins notified. It should just outright deny it I'd argue instead of shutting down, but I've seen really weird security policies before that make little practical sense and wouldn't be too surprised if someone created an IDS enforcement policy similar to this.
tshtf 11y ago
This is very widespread. I reported a similar issue on Chunkhost (https://chunkhost.com/ https://chunkhost.com/) back in 2011... They never responded to my email about remediation, but hopefully the issue was fixed.
wumwufwurd 11y ago
Affected customers have received an email about the issue. I got mine about 2 hours ago. It would be great not to publish an article like that until some time later, so that everyone gets a chance to fix the problem on their machines before it goes fully public - especially since it's the holiday season and some people don't check email or news that often.
- mschuster91 11y ago
  The problem is that as soon as you send the email, someone will leak it, setting the usual internet machine of FUD, "this must be a fake" etc at work.
- Tepix 11y ago
  The impact isn't that great anyway...
rwmj 11y ago
Use virt-sysprep to prepare your templates. http://libguestfs.org/virt-sysprep.1.html http://libguestfs.org/virt-sysprep.1.html
r0muald 11y ago
I do hope this page is not the only source for the announcement, since it's a wiki page on a non-HTTPS MediaWiki instance that runs an abysmally old and unsupported (= unsafe) version of the software.
ThomasAH 11y ago
Some additional details and their mail to customers here: http://blogs.intevation.de/thomas/hetzner-duplicate-ed25519-ssh-host-keys/ http://blogs.intevation.de/thomas/hetzner-duplicate-ed25519-... "I must say, I’m impressed. Especially at this time of the year I would have expected a slower reaction or a less detailed announcement."
hannob 11y ago
May be helpful for others: https://github.com/hannob/ed25519hetzner https://github.com/hannob/ed25519hetzner (script to check host key and known_hosts file for vulnerable hetzner keys)

127 points by jonaslejon 11y ago

zx2c4 11y ago

Yet another reason not to use images provided by your hosting company. In anycase, if you are relying on hosting company images (complete with their root backdoor for "performance monitoring control panel!"), it's not like you care very much about your servers' security in the first place...

Ded7xSEoPKYNsDd 11y ago
If you really believe your hosting provider puts root backdoors in your systems, you really shouldn't run anything in a hypervisor controlled by them.
- TimWolla 11y ago
  http://help.ovh.com/InstallOVHkey http://help.ovh.com/InstallOVHkey > In order to be able to intervene on your dedicated server without your root password, the automatic installation of ssh key is done. Only authorized employees of OVH will use it. It is not a gap in security, contrary, thanks to this OVH has root rights to your server and may identify the problems with your server. When you request an intervention, we need to have access to ssh.
  - mschuster91 11y ago
    Sounds to me like what is needed for a managed root-server. Actually it's kinda cool because you can enjoy timely upgrades, security fixes and response to ddos attacks while at the same time you also have the flexibility to install needed software without going through a change request workflow...
  - pquerna 11y ago
    A SSH private key is just some bytes. Once you have access to it, you have access to it forever in the future. RSA doesn't care if you are an "authorized employee". A static SSH key for a single human is already risky over a long time period, but for a key that is shared between multiple humans... wow.
  - niij 11y ago
    This is an opt-in process, not an opt-out. They don't have authorized keys on your server by default, only instructions to add them if needed.
    - Wilya 11y ago
      Unless it has changed recently, it's opt-out. The instructions are mostly for re-adding the key if you have removed it.
dozzie 11y ago
> Yet another reason not to use images provided by your hosting company. Wait, you mean you clone your images instead of installing them from scratch with some automated system? How barbarous!
otterley 11y ago
It's not as though it's impossible to make a secure image. Mistakes can and do happen, though. Better to own up to it and fix it than to hide the problem.

jjuhl 11y ago

At least they are aware of the issue and warning their customers. A lot of companies would just fix the issue (or not even that) and try to silently sweep it under the rug.

kuschku 11y ago
Their business is being a trustworthy, high quality service. Sweeping it under the rug would destroy their business. (Not that competitors wouldn’t have done it anyway)
- chx 11y ago
  > Their business is being a trustworthy, high quality service. Really. Here I thought their business is renting out servers made from desktop parts for insanely low amounts of money. I have no problems with this strategy, I enjoy my 42EUR a month 2x3TB HDD + 2x120GB SSD, 16GB i7 2600 server, thanks much. It runs hobby sites and such. I know what I bought and what I can expect.
  - 5ersi 11y ago
    They also offer reasonably priced server-class dedicated servers with Xeon processors, ECC memory and datacenter-series SSDs.
    - merb 11y ago
      "reasonably priced". You never needed a private network with 3 nodes.
      - corford 11y ago
        €12,61/month extra per server and a one off ~€25 for the switch is hardly extortionate given what they charge for servers...
        merb 11y ago

ymse 11y ago

This is probably pretty common. When I first deployed virtualization at a previous job ~5 years ago, it took almost two years before I realized all images used the same host keys. Can someone versed in cryptography comment on whether this allows passive eavesdropping? Active MITM sure, but I thought session keys would be unique.

brians 11y ago
You are right: it allows active impersonation, but not passive eavesdropping.
- pflanze 11y ago
  > impersonation "Impersonification" of the server, you mean (inservication if that's a word). It would allow an attacker to operate a machine under their control that poses as mine. But then, for password based logins, that will actually be enough to impersonify the user as well (the attacker will create an ssh connection to the real server with the given password). But I wonder how things work with key based logins. I've ordered a server from Hetzner recently, I guess I'll just ditch it (and perhaps order a new one). BTW I haven't gotten any email notification from Hetzner about this yet.
  - snuxoll 11y ago
    Given that key based logins require signing a per-session token to verify the owner of the key is trying to connect (otherwise they would be useless since someone could just perform a replay attack) MITM'ing a key-authenticated session would allow you to perform attacks with that session, but it would not allow you to re-authenticate against that server later.
    - Buge 11y ago
      Would active MITM be possible with key login? I would think it should be impossible. If the symmetric session key is chosen by the server and encrypted with the client's public key, the attacker won't know it and cannot mess with the session. The only thing the attacker can do is completely impersonate the server. But the attacker cannot communicate with the actual server at all.

zymhan 11y ago

Wasn't there a cloud hosting provider that didn't even bother cleaning out the SSH known_hosts file?

KenanSulayman 11y ago

Companies that let their interns do the crypto stuff ¯\_(ツ)_/¯

sarciszewski 11y ago

If anyone works for a hosting provider, please write a script that checks all of your customers' SSH ports for identical fingerprints. If you have any collisions, you have a problem.

jedisct1 11y ago
Except that performing network scans, even on your own network, is illegal in many countries. Even in countries where this is legal, providers cannot take the risk of running a scan that could possibly crash a customer application, or they could be sued for that.
- switch007 11y ago
  Which countries and under which laws? Which T&Cs do not have a blanket clause to indemnify the hosting company in such situations? If I accidentally reboot the virtualisation host, can my company be sued for making the application "crash"?
  - AdamGibbins 11y ago
    There's multiple examples of legal issues mentioned on https://nmap.org/book/legal-issues.html https://nmap.org/book/legal-issues.html
- sarciszewski 11y ago
  Huh. I hadn't even thought of that. If connecting to port 22 and slurping the host fingerprint crashes a customer application... I don't even know. Like, how is their app even running?
  - devonkim 11y ago
    I can imagine a scenario where a user that logs into a machine and tries to read a very sensitive file that they shouldn't have access to (should only be read by sshd and a specific automation user) trips an alarm via hooks in SELinux or something similar that causes the host to shut down immediately and security admins notified. It should just outright deny it I'd argue instead of shutting down, but I've seen really weird security policies before that make little practical sense and wouldn't be too surprised if someone created an IDS enforcement policy similar to this.

tshtf 11y ago

This is very widespread. I reported a similar issue on Chunkhost (https://chunkhost.com/ https://chunkhost.com/) back in 2011... They never responded to my email about remediation, but hopefully the issue was fixed.

wumwufwurd 11y ago

Affected customers have received an email about the issue. I got mine about 2 hours ago. It would be great not to publish an article like that until some time later, so that everyone gets a chance to fix the problem on their machines before it goes fully public - especially since it's the holiday season and some people don't check email or news that often.

mschuster91 11y ago
The problem is that as soon as you send the email, someone will leak it, setting the usual internet machine of FUD, "this must be a fake" etc at work.
Tepix 11y ago
The impact isn't that great anyway...

rwmj 11y ago

Use virt-sysprep to prepare your templates. http://libguestfs.org/virt-sysprep.1.html http://libguestfs.org/virt-sysprep.1.html

r0muald 11y ago

I do hope this page is not the only source for the announcement, since it's a wiki page on a non-HTTPS MediaWiki instance that runs an abysmally old and unsupported (= unsafe) version of the software.

ThomasAH 11y ago

Some additional details and their mail to customers here: http://blogs.intevation.de/thomas/hetzner-duplicate-ed25519-ssh-host-keys/ http://blogs.intevation.de/thomas/hetzner-duplicate-ed25519-... "I must say, I’m impressed. Especially at this time of the year I would have expected a slower reaction or a less detailed announcement."

hannob 11y ago

May be helpful for others: https://github.com/hannob/ed25519hetzner https://github.com/hannob/ed25519hetzner (script to check host key and known_hosts file for vulnerable hetzner keys)