23 ms·

Golang SSH Security

426 points by transitorykris 9y ago

3pt14159 9y ago
I love Digital Ocean, but they do the same thing with their API. I wrote to them about it years ago, even talked to some developers there, and the general explanation is the same: Screw around with cloud-init to get the public key. If you use the DO API to provision servers my feature request is here: https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/9307569-return-the-droplet-s-ssh-public-key-as-part-of-api https://digitalocean.uservoice.com/forums/136585-digitalocea... Please upvote it or at the very least copy the cloud-init script to help provision your servers.
- problems 9y ago
  For hostkeys on DO you can probably get a script to run that'll request a signed certificate from a server you own. The signed cert can be validated fully by clients. Or go with a convergence style system and probe it from multiple locations. Or just give up and go with TOFU - if you never get an error even on different connections, you probably haven't been mitm'd.
  - DorothySim 9y ago
    > For hostkeys on DO you can probably get a script to run that'll request a signed certificate from a server you own. Or just embed the signed host certificate in cloud-init.
mbertschler 9y ago
While I really love the stable nature of Go and its standard library, I am happy that this breaking change was put out there in the interest of security. This issue hit me while building a tool for internal use at my employer. I am using the glide vendoring manager for this project, added another dependency which triggered an update of all other dependencies. At that point my tool broke and forced me to actually think about host key verification.
- TheDong 9y ago
  This isn't the standard library though, and if it were then it wouldn't have been changed.
  - mbertschler 9y ago
    I didn't want to imply that. I meant that this kind of fix is a good reason to break something, and I am happy that they quickly reacted to this issue, and don't change everything all the time even though it is in the x/ packages and not covered by the standard library stability promise. In general I am very happy that the big emphasis on a stable APIs was taken up by the community, and that we have a lot of stable packages out now (even though they might not be 100% stable like the standard library). Since I also have to work with NodeJs where changing APIs and packages are much more common, I came to really appreciate that fact about the ecosystem.
  - syscomet 9y ago
    It's a special case though. The golang.org/x/ packages are experimental but also candidates for promotion to the standard library. Eg, "context". But this sort of issue is exactly the sort of real-world review and hardening which justifies having a namespace for stuff to go _before_ it becomes stdlib.
    - kardianos 9y ago
      No, x repos are just eXtra. /x/exp is expiramental.
  - bradfitz 9y ago
    We have broken compatibility once before in the standard library for security reasons. The go1compat doc says we're allowed to: https://golang.org/doc/go1compat https://golang.org/doc/go1compat > Security. A security issue in the specification or implementation may come to light whose resolution requires breaking compatibility. We reserve the right to address such security issues.
adtac 9y ago
I'm really impressed with the quick response from the golang team. The fact that they didn't mind introducing breaking changes shows that their priorities are right.
niftich 9y ago
Golang seems to follow the 80/20 rule from the outset (or perhaps an even smaller proportion), which is perfectly fine. Some other languages' standardlibs try to offer a complete treatment of a particular problemspace from the start which is tricky to get right on first attempt. Those are the instances where developers complain about complex APIs, uneven abstractions, or the like. However, one of the artifacts of a popular language having a lean-and-mean standard library is that custom code proliferates, and the Go community's distaste for frameworks (as opposed to libraries) means that the it's not just the business-specific edges of the code that's unique in each implementation (as you'd expect), but also a good amount of the plumbing and domain-specific control code and their immediate callers. In some other languages, where there's more of a culture for using a dependency to intentionally simplify your problem space in exchange for ceding control, this style would be derided as NIH. The vendor's response here is a function of not only the vendor's own rationale and priorities, but also of the above developer philosophy. This is surprising to me, given that Go is an opinionated language, and yet opinionated third-party code driving your logic is frequently discouraged by its community. On the other hand, the language maintainers' response was measured, proper, and commendable. They made a breaking change to an experimental API, and improved their product in the process.
- deleted 9y ago
  [deleted]
- e12e 9y ago
  This isn't 80/20 - this is broken. If you trust your network, use rsh or telnet.
  - comex 9y ago
    Telnet does not support port forwarding, file transfer, host key authentication (trusted network ≠ trusted clients), etc., and is not installed by default on most systems. rsh only supports file transfer out of those, and is dead enough that it's not even in Homebrew. Nor is there any reason not to use SSH just because you trust your network, except possibly performance of huge file transfers.
risyasin 9y ago
Well. I haven't really started to learn golang yet. But sure that this breaking change indeed convinced me to do it. I have implemented an automated ssh session in another language there was absolutely no host key checking or tofu implementation even worse that they designed the api not to allow that manually. That was frustrating. But obviously the golang language designers and the entry owner and myself sharing the same concerns obviously. Thanks for writing about this
avar 9y ago
Oh man: > I am bemused by an approach to accepting > security reports which is to go through the > motions of having PGP public keys available > for people to use to report issues but upon > receiving such a request ask for it to be > submitted without PGP because digging out > the keys is too much of a hassle.
- pm215 9y ago
  Advertising that you use pgp and then in practice not using it is kind of daft, but is there actually much benefit to using encryption to initially report vulnerabilities?
  - Godel_unicode 9y ago
    Using pgp is good for +30 cryptolluminati cool points. It also reduces your kibitz surface, which is useful for keeping the guardian from writing incorrect hit pieces about how you don't care about security. Other than that, not really.
  - openasocket 9y ago
    If we're talking serious vulnerabilities (e.g. can allow remote code execution) for a project used by a large number of people, then absolutely. It can be weeks between initial report and a fix being pushed depending on the vulnerability, and even longer for people to update. It's a pretty good niche: intercept emails to maintainers for important projects and either sell the vulnerabilities or use them yourself in that small window. It'll net you a pretty steady stream of 0days if you can pull it off and you choose the right target. Encrypting communication makes that strategy a lot more difficult, and more invasive.
    - pm215 9y ago
      It just seems to me that the window is small (and you have to write a weaponised exploit in that time too), and the only people likely to be able to reliably intercept the email are nation-state actors who do bulk email slurping, and they're probably sitting on a pile of zerodays anyway. But that's just gut feeling so it's likely wrong.
aceperry 9y ago
Really wonderful and thorough report. I'm not at all a security expert but manage to learn quite a lot from reading this post. Kudos to the author for giving context and background on the issues. If more security reports are written like this, the whole industry would benefit greatly.
eropple 9y ago
I am...not a fan of Golang, as I have made pretty clear around here on occasion. But I'll give credit where credit's due, and this is a good decision on the part of the people maintaining x/crypto/ssh. Not the tooling vendor's awful response--I'm pretty sure I know who it is, and if not there's two of 'em because I've had these conversations before--but the maintainers are doing the right thing. This probably shouldn't have gotten out the door without host key verification in the first place (and that ties back into the reasons why I do not like or trust Golang or its community when it comes to tools that I have to consume), but it's better to bite the bullet and fix this now instead of letting it fester. (The "PGP is too hard for discussing security issues" thing, though, is total nonsense. Can't be doing that.)
- grey-area 9y ago
  Doesn't the writer call out hashicorp specifically by name in the article? I used to respect this vendor and recommend their tools to others. I am having to rethink this a lot. I am no longer happy to recommend Hashicorp products to others.
  - eropple 9y ago
    Oh, ha, I glazed over that paragraph! But yeah, it was totally Hashicorp I was thinking about; this sort of reliability-thoroughly-optional thing shoots through a lot of their tools and keeps me pretty far away from any of the ones that might touch live environments. (Packer and Vagrant are fine.)
    - RRRA 9y ago
      Which one of their tools are we talking about here? Terraform?
      - syscomet 9y ago
        blog-post author here: all of them written in Golang which use SSH. Packer, Terraform, Vault, all of them. Using a bastion host in your configurations doesn't help if you invoke the tooling on your laptop, since the connection to the bastion host is done with the SSH package, again with no host-key verification.
joneholland 9y ago
Why do people keep saying "the vendor"? It's hashicorp.
- fancy_pantser 9y ago
  Calling software companies "vendors" is a fun little lost piece of Americana. Like calling marijuana "grass".
YZF 9y ago
It's amazing how many people out there consider MITM as something they don't have to defend against. If you're a developer you have to assume your system will be MITMed. It doesn't matter if you're on the Internet or behind a firewall. Trust on first use is not a good solution because someone can tailor their attack against that first use.
- ge96 9y ago
  What are some clear signs that this is happening to you? I track the url requested on my server(s) and some of them don't make sense/looking for exploits like wordpress login exploits. I don't know I have SSL/A+ according to Qualys, I'm kind of drawing a blank where MITM happens. I've seen/read about it before. Heartbleed? No I don't know... really tired, but interested, gots to Google. edit: I'm still on LAMP stack for clarification. Too bad to hear about Golang though I'm still looking to learn it I hear a lot of great things about it.
  - F30 9y ago
    MITM attacks are designed to not be detectable, so the solution is to have tooling which prohibits them – exactly what is lacking in this case. For web, using TLS (SSL) is a good start. This could be improved further by using HSTS, HPKP, DANE etc. (not sure if A+ already implies them anyway). For SSH, you need to have an out-of-band way to get the host keys or use something like an SSH CA.
    - ge96 9y ago
      What does that mean out-of-band? I'm not good with SSH, I'm not using 2-factor key based, also in general it makes sense to have separate servers right? Like one server that transfers request to a server closer to another country. Not Cloudflare but your own thing assuming you rented from different datacenters in the world. Sorry not related to the question. edit: literally band? Like another wavelength/connection?
      - js2 9y ago
DanielDent 9y ago
I wrote about this a while back, and also proposed a solution which doesn't involve parsing console output: https://www.danieldent.com/blog/ssh-requires-a-chain-of-trust/ https://www.danieldent.com/blog/ssh-requires-a-chain-of-trus...
baby 9y ago
A few things: 1. how can an experimental library (x/) get a CVE? 2. what is "hostkey verification"? Probably the fingerprint check you usually get when you ssh into a machine + the blocking warning you get when the fingerprint of the machine suddenly changes. 3. if this is what "hostkey verification" is. How is it so hard to implement? create some sort of fingerprint out of the server's public key; prompt the user for input; cache the result.
- tptacek 9y ago
  What does a CVE number have to do with whether the target is experimental? A bug is a bug.
  - baby 9y ago
    I'd expect an experimental library to possibly be filled with bugs and constantly evolve (in a breaking way) its API while having a "USE AT YOUR OWN RISK" message. Although that might not be how the namespace golang.org/x/ is defined. Looking here: https://golang.org/doc/go1compat https://golang.org/doc/go1compat > may be developed under looser compatibility requirements looks like that might explain why it's a CVE. Also another good answer from Zaki: https://twitter.com/zmanian/status/853342150272008192 https://twitter.com/zmanian/status/853342150272008192
    - ymse 9y ago
      CVEs can be thought of more as a public security advisory. Many vendors and distributions follow the CVE list. If there was no CVE, this issue might have gone unrecognised in your package manager or paid software product for a long time.
      - baby 9y ago
        > your package manager or paid software product I'd hope these don't use experimental libraries. Although as others pointed out to me, this is not an experimental library :) (which was the answer I was looking for)
chrisper 9y ago
I wonder if the devops company was this one: https://gravitational.com/teleport/index.html https://gravitational.com/teleport/index.html EDIT: Actually, it looks like it's Hashicorp.
bogomipz 9y ago
Could SSHFP records not have been an option here? Especially combined with something like DNSSEC?
- F30 9y ago
  Maybe, but then the library would have to support those. This isn't as much about the concrete technology as it is about the willingness to implement any host-key checking.
bradknowles 9y ago
Phil may be grumpy, but he is not a troll.
mitchellh 9y ago
Hello! As the blog post clearly states, the vendor is HashiCorp. As the founder of HashiCorp and someone who participated in the initial report we received on this topic, I'd like to state our point of view from my own mouth. I'd first like to be up front about exactly which of our software doesn't perform host key verification, since we have a lot of software and this CVE doesn't apply to most. There are three places that were identified as affected: Packer and Terraform with SSH provisioners, which both create a machine resource and can perform SSH connections to setup the machine; and Vault’s SSH backend in Dynamic Key mode performs SSH connections from the Vault server to hosts (other modes do not). Any other usage of our software is unaffected. We’ll discuss each of these cases in detail, since the details matter to understand our thought process and response. Vault: The SSH secret backend has three modes that can be used for generating SSH credentials: certificates, one-time passwords, and dynamic keys. Only the dynamic key mode ever actually makes connections to other machines, but more importantly, our documentation has always recommended that the dynamic key mode only be used as a last resort because of its various (documented) drawbacks compared to the other modes. With the addition of the ability to generate SSH certificates (which was on our roadmap for a long time and added in 0.7, prior to both the original report and the blog post), we did not explicitly mark the dynamic key mode as deprecated in our documentation, but we probably should do so. Given that it is not recommended for usage (but maintained for backwards compatibility), we chose to warn users of this additional drawback of the dynamic key method, and documented the lack of host key verification (https://github.com/hashicorp/vault/commit/251da1bcdc27678feaa477f087c8d010223d7e8c https://github.com/hashicorp/vault/commit/251da1bcdc27678fea...). As we stated in our response to the reporter, "It isn’t something we want to hide (and we’re not trying to) and we will document this." Terraform/Packer: Terraform and Packer support the ability to use "provisioners" to bootstrap a machine. In both, the provisioner is run very shortly after the machine is initially created, representing an extremely small window of attack. Neither support connecting to a pre-existing machine via SSH under normal use cases (you can make it happen through some advanced configuration trickery with Terraform, but it's abnormal). Because of this, we didn't register this as a high-priority issue. However, we admit that this can be improved and we likely should've been more reactionary in our response. I apologize for that. We have added plans to improve this to our roadmap, covered in a couple paragraphs. As the blog post states, the reporter suggested parsing console logs to determine the host key. And, as the blog post correctly says, we don't want to do this. There is a combinatorial explosion of complexity in supporting this, we have experience with this (due to Vagrant supporting this type of behavior), and we've found maintenance of this sort of functionality to be difficult to support over time. We came to this conclusion though only because there is a viable alternative: SSH certificate authentication. If a viable alternative didn't exist, we may have been forced to take the more complex route. SSH certificate authentication was introduced many years ago and is broadly supported. This type of auth also provides authenticity to a first-use connection. We mentioned in our response email that this is something we're open to doing instead. I admit that in our response to the reporter, we explicitly said this "is not a priority" but shortly after decided to schedule this work for the next major TF release. We should've followed up again, but didn't. And that's where we're at currently! I hope this helps make our response to the report and our future roadmap around this issue more clear.
deleted 9y ago
[deleted]
asveikau 9y ago
Good thing they didn't write this in C, or their library would have real security trouble. /s
jakewins 9y ago
When we built the new set of drivers for Neo4j, we decided to allow three modes: No encryption, Trust on First Use and Trusted Signature - there's no way to establish an connection without trust. This was a terrifying decision, because of ease of use concerns. Having done so and shipped it, TL;DR: It worked awesome, outside of some early kinks in TOFU that we worked out - and now everyone can sleep well knowing there's not a single install that thinks they are running an encrypted setup when they really aren't. Anyone that came back asking for a flag to disable host key verification seemed happy with our argument for why that's not really much different from just disabling encryption. See "Trust" here: https://neo4j.com/docs/developer-manual/current/drivers/configure-connect/ https://neo4j.com/docs/developer-manual/current/drivers/conf... If you're interested in doing this as well, we wrote code to do it in Python, JS, Java and C#, it's all Apache licensed: JS: https://github.com/neo4j/neo4j-javascript-driver/blob/1.2/src/v1/internal/ch-node.js#L106 https://github.com/neo4j/neo4j-javascript-driver/blob/1.2/sr... Python: https://github.com/neo4j/neo4j-python-driver/blob/1.2/neo4j/bolt/connection.py#L463 https://github.com/neo4j/neo4j-python-driver/blob/1.2/neo4j/... Java: https://github.com/neo4j/neo4j-java-driver/blob/1.3/driver/src/main/java/org/neo4j/driver/internal/security/TLSSocketChannel.java#L68 https://github.com/neo4j/neo4j-java-driver/blob/1.3/driver/s... C#: https://github.com/neo4j/neo4j-dotnet-driver/blob/1.3/Neo4j.Driver/Neo4j.Driver/Internal/Connector/ITrustStrategy.cs#L28 https://github.com/neo4j/neo4j-dotnet-driver/blob/1.3/Neo4j....
ak217 9y ago
FYI, you can also instruct cloud-init to use a particular key pair by supplying it in instance metadata/user-data. This avoids the need for hacky scripts extracting public keys from console output (which may also be delayed by a few minutes after the instance starts).

I love Digital Ocean, but they do the same thing with their API. I wrote to them about it years ago, even talked to some developers there, and the general explanation is the same: Screw around with cloud-init to get the public key. If you use the DO API to provision servers my feature request is here: https://digitalocean.uservoice.com/forums/136585-digitalocean/suggestions/9307569-return-the-droplet-s-ssh-public-key-as-part-of-api https://digitalocean.uservoice.com/forums/136585-digitalocea... Please upvote it or at the very least copy the cloud-init script to help provision your servers.

problems 9y ago
For hostkeys on DO you can probably get a script to run that'll request a signed certificate from a server you own. The signed cert can be validated fully by clients. Or go with a convergence style system and probe it from multiple locations. Or just give up and go with TOFU - if you never get an error even on different connections, you probably haven't been mitm'd.
- DorothySim 9y ago
  > For hostkeys on DO you can probably get a script to run that'll request a signed certificate from a server you own. Or just embed the signed host certificate in cloud-init.

mbertschler 9y ago

While I really love the stable nature of Go and its standard library, I am happy that this breaking change was put out there in the interest of security. This issue hit me while building a tool for internal use at my employer. I am using the glide vendoring manager for this project, added another dependency which triggered an update of all other dependencies. At that point my tool broke and forced me to actually think about host key verification.

TheDong 9y ago
This isn't the standard library though, and if it were then it wouldn't have been changed.
- mbertschler 9y ago
  I didn't want to imply that. I meant that this kind of fix is a good reason to break something, and I am happy that they quickly reacted to this issue, and don't change everything all the time even though it is in the x/ packages and not covered by the standard library stability promise. In general I am very happy that the big emphasis on a stable APIs was taken up by the community, and that we have a lot of stable packages out now (even though they might not be 100% stable like the standard library). Since I also have to work with NodeJs where changing APIs and packages are much more common, I came to really appreciate that fact about the ecosystem.
- syscomet 9y ago
  It's a special case though. The golang.org/x/ packages are experimental but also candidates for promotion to the standard library. Eg, "context". But this sort of issue is exactly the sort of real-world review and hardening which justifies having a namespace for stuff to go _before_ it becomes stdlib.
  - kardianos 9y ago
    No, x repos are just eXtra. /x/exp is expiramental.
- bradfitz 9y ago
  We have broken compatibility once before in the standard library for security reasons. The go1compat doc says we're allowed to: https://golang.org/doc/go1compat https://golang.org/doc/go1compat > Security. A security issue in the specification or implementation may come to light whose resolution requires breaking compatibility. We reserve the right to address such security issues.

adtac 9y ago

I'm really impressed with the quick response from the golang team. The fact that they didn't mind introducing breaking changes shows that their priorities are right.

niftich 9y ago

Golang seems to follow the 80/20 rule from the outset (or perhaps an even smaller proportion), which is perfectly fine. Some other languages' standardlibs try to offer a complete treatment of a particular problemspace from the start which is tricky to get right on first attempt. Those are the instances where developers complain about complex APIs, uneven abstractions, or the like. However, one of the artifacts of a popular language having a lean-and-mean standard library is that custom code proliferates, and the Go community's distaste for frameworks (as opposed to libraries) means that the it's not just the business-specific edges of the code that's unique in each implementation (as you'd expect), but also a good amount of the plumbing and domain-specific control code and their immediate callers. In some other languages, where there's more of a culture for using a dependency to intentionally simplify your problem space in exchange for ceding control, this style would be derided as NIH. The vendor's response here is a function of not only the vendor's own rationale and priorities, but also of the above developer philosophy. This is surprising to me, given that Go is an opinionated language, and yet opinionated third-party code driving your logic is frequently discouraged by its community. On the other hand, the language maintainers' response was measured, proper, and commendable. They made a breaking change to an experimental API, and improved their product in the process.

deleted 9y ago
[deleted]
e12e 9y ago
This isn't 80/20 - this is broken. If you trust your network, use rsh or telnet.
- comex 9y ago
  Telnet does not support port forwarding, file transfer, host key authentication (trusted network ≠ trusted clients), etc., and is not installed by default on most systems. rsh only supports file transfer out of those, and is dead enough that it's not even in Homebrew. Nor is there any reason not to use SSH just because you trust your network, except possibly performance of huge file transfers.

risyasin 9y ago

Well. I haven't really started to learn golang yet. But sure that this breaking change indeed convinced me to do it. I have implemented an automated ssh session in another language there was absolutely no host key checking or tofu implementation even worse that they designed the api not to allow that manually. That was frustrating. But obviously the golang language designers and the entry owner and myself sharing the same concerns obviously. Thanks for writing about this

avar 9y ago

Oh man: > I am bemused by an approach to accepting > security reports which is to go through the > motions of having PGP public keys available > for people to use to report issues but upon > receiving such a request ask for it to be > submitted without PGP because digging out > the keys is too much of a hassle.

pm215 9y ago
Advertising that you use pgp and then in practice not using it is kind of daft, but is there actually much benefit to using encryption to initially report vulnerabilities?
- Godel_unicode 9y ago
  Using pgp is good for +30 cryptolluminati cool points. It also reduces your kibitz surface, which is useful for keeping the guardian from writing incorrect hit pieces about how you don't care about security. Other than that, not really.
- openasocket 9y ago
  If we're talking serious vulnerabilities (e.g. can allow remote code execution) for a project used by a large number of people, then absolutely. It can be weeks between initial report and a fix being pushed depending on the vulnerability, and even longer for people to update. It's a pretty good niche: intercept emails to maintainers for important projects and either sell the vulnerabilities or use them yourself in that small window. It'll net you a pretty steady stream of 0days if you can pull it off and you choose the right target. Encrypting communication makes that strategy a lot more difficult, and more invasive.
  - pm215 9y ago
    It just seems to me that the window is small (and you have to write a weaponised exploit in that time too), and the only people likely to be able to reliably intercept the email are nation-state actors who do bulk email slurping, and they're probably sitting on a pile of zerodays anyway. But that's just gut feeling so it's likely wrong.

aceperry 9y ago

Really wonderful and thorough report. I'm not at all a security expert but manage to learn quite a lot from reading this post. Kudos to the author for giving context and background on the issues. If more security reports are written like this, the whole industry would benefit greatly.

eropple 9y ago

I am...not a fan of Golang, as I have made pretty clear around here on occasion. But I'll give credit where credit's due, and this is a good decision on the part of the people maintaining x/crypto/ssh. Not the tooling vendor's awful response--I'm pretty sure I know who it is, and if not there's two of 'em because I've had these conversations before--but the maintainers are doing the right thing. This probably shouldn't have gotten out the door without host key verification in the first place (and that ties back into the reasons why I do not like or trust Golang or its community when it comes to tools that I have to consume), but it's better to bite the bullet and fix this now instead of letting it fester. (The "PGP is too hard for discussing security issues" thing, though, is total nonsense. Can't be doing that.)

grey-area 9y ago
Doesn't the writer call out hashicorp specifically by name in the article? I used to respect this vendor and recommend their tools to others. I am having to rethink this a lot. I am no longer happy to recommend Hashicorp products to others.
- eropple 9y ago
  Oh, ha, I glazed over that paragraph! But yeah, it was totally Hashicorp I was thinking about; this sort of reliability-thoroughly-optional thing shoots through a lot of their tools and keeps me pretty far away from any of the ones that might touch live environments. (Packer and Vagrant are fine.)
  - RRRA 9y ago
    Which one of their tools are we talking about here? Terraform?
    - syscomet 9y ago
      blog-post author here: all of them written in Golang which use SSH. Packer, Terraform, Vault, all of them. Using a bastion host in your configurations doesn't help if you invoke the tooling on your laptop, since the connection to the bastion host is done with the SSH package, again with no host-key verification.

joneholland 9y ago

Why do people keep saying "the vendor"? It's hashicorp.

fancy_pantser 9y ago
Calling software companies "vendors" is a fun little lost piece of Americana. Like calling marijuana "grass".

YZF 9y ago

It's amazing how many people out there consider MITM as something they don't have to defend against. If you're a developer you have to assume your system will be MITMed. It doesn't matter if you're on the Internet or behind a firewall. Trust on first use is not a good solution because someone can tailor their attack against that first use.

ge96 9y ago
What are some clear signs that this is happening to you? I track the url requested on my server(s) and some of them don't make sense/looking for exploits like wordpress login exploits. I don't know I have SSL/A+ according to Qualys, I'm kind of drawing a blank where MITM happens. I've seen/read about it before. Heartbleed? No I don't know... really tired, but interested, gots to Google. edit: I'm still on LAMP stack for clarification. Too bad to hear about Golang though I'm still looking to learn it I hear a lot of great things about it.
- F30 9y ago
  MITM attacks are designed to not be detectable, so the solution is to have tooling which prohibits them – exactly what is lacking in this case. For web, using TLS (SSL) is a good start. This could be improved further by using HSTS, HPKP, DANE etc. (not sure if A+ already implies them anyway). For SSH, you need to have an out-of-band way to get the host keys or use something like an SSH CA.
  - ge96 9y ago
    What does that mean out-of-band? I'm not good with SSH, I'm not using 2-factor key based, also in general it makes sense to have separate servers right? Like one server that transfers request to a server closer to another country. Not Cloudflare but your own thing assuming you rented from different datacenters in the world. Sorry not related to the question. edit: literally band? Like another wavelength/connection?
    - js2 9y ago

DanielDent 9y ago

I wrote about this a while back, and also proposed a solution which doesn't involve parsing console output: https://www.danieldent.com/blog/ssh-requires-a-chain-of-trust/ https://www.danieldent.com/blog/ssh-requires-a-chain-of-trus...

baby 9y ago

A few things: 1. how can an experimental library (x/) get a CVE? 2. what is "hostkey verification"? Probably the fingerprint check you usually get when you ssh into a machine + the blocking warning you get when the fingerprint of the machine suddenly changes. 3. if this is what "hostkey verification" is. How is it so hard to implement? create some sort of fingerprint out of the server's public key; prompt the user for input; cache the result.

tptacek 9y ago
What does a CVE number have to do with whether the target is experimental? A bug is a bug.
- baby 9y ago
  I'd expect an experimental library to possibly be filled with bugs and constantly evolve (in a breaking way) its API while having a "USE AT YOUR OWN RISK" message. Although that might not be how the namespace golang.org/x/ is defined. Looking here: https://golang.org/doc/go1compat https://golang.org/doc/go1compat > may be developed under looser compatibility requirements looks like that might explain why it's a CVE. Also another good answer from Zaki: https://twitter.com/zmanian/status/853342150272008192 https://twitter.com/zmanian/status/853342150272008192
  - ymse 9y ago
    CVEs can be thought of more as a public security advisory. Many vendors and distributions follow the CVE list. If there was no CVE, this issue might have gone unrecognised in your package manager or paid software product for a long time.
    - baby 9y ago
      > your package manager or paid software product I'd hope these don't use experimental libraries. Although as others pointed out to me, this is not an experimental library :) (which was the answer I was looking for)

chrisper 9y ago

I wonder if the devops company was this one: https://gravitational.com/teleport/index.html https://gravitational.com/teleport/index.html EDIT: Actually, it looks like it's Hashicorp.

bogomipz 9y ago

Could SSHFP records not have been an option here? Especially combined with something like DNSSEC?

F30 9y ago
Maybe, but then the library would have to support those. This isn't as much about the concrete technology as it is about the willingness to implement any host-key checking.

bradknowles 9y ago

Phil may be grumpy, but he is not a troll.

mitchellh 9y ago

Hello! As the blog post clearly states, the vendor is HashiCorp. As the founder of HashiCorp and someone who participated in the initial report we received on this topic, I'd like to state our point of view from my own mouth. I'd first like to be up front about exactly which of our software doesn't perform host key verification, since we have a lot of software and this CVE doesn't apply to most. There are three places that were identified as affected: Packer and Terraform with SSH provisioners, which both create a machine resource and can perform SSH connections to setup the machine; and Vault’s SSH backend in Dynamic Key mode performs SSH connections from the Vault server to hosts (other modes do not). Any other usage of our software is unaffected. We’ll discuss each of these cases in detail, since the details matter to understand our thought process and response. Vault: The SSH secret backend has three modes that can be used for generating SSH credentials: certificates, one-time passwords, and dynamic keys. Only the dynamic key mode ever actually makes connections to other machines, but more importantly, our documentation has always recommended that the dynamic key mode only be used as a last resort because of its various (documented) drawbacks compared to the other modes. With the addition of the ability to generate SSH certificates (which was on our roadmap for a long time and added in 0.7, prior to both the original report and the blog post), we did not explicitly mark the dynamic key mode as deprecated in our documentation, but we probably should do so. Given that it is not recommended for usage (but maintained for backwards compatibility), we chose to warn users of this additional drawback of the dynamic key method, and documented the lack of host key verification (https://github.com/hashicorp/vault/commit/251da1bcdc27678feaa477f087c8d010223d7e8c https://github.com/hashicorp/vault/commit/251da1bcdc27678fea...). As we stated in our response to the reporter, "It isn’t something we want to hide (and we’re not trying to) and we will document this." Terraform/Packer: Terraform and Packer support the ability to use "provisioners" to bootstrap a machine. In both, the provisioner is run very shortly after the machine is initially created, representing an extremely small window of attack. Neither support connecting to a pre-existing machine via SSH under normal use cases (you can make it happen through some advanced configuration trickery with Terraform, but it's abnormal). Because of this, we didn't register this as a high-priority issue. However, we admit that this can be improved and we likely should've been more reactionary in our response. I apologize for that. We have added plans to improve this to our roadmap, covered in a couple paragraphs. As the blog post states, the reporter suggested parsing console logs to determine the host key. And, as the blog post correctly says, we don't want to do this. There is a combinatorial explosion of complexity in supporting this, we have experience with this (due to Vagrant supporting this type of behavior), and we've found maintenance of this sort of functionality to be difficult to support over time. We came to this conclusion though only because there is a viable alternative: SSH certificate authentication. If a viable alternative didn't exist, we may have been forced to take the more complex route. SSH certificate authentication was introduced many years ago and is broadly supported. This type of auth also provides authenticity to a first-use connection. We mentioned in our response email that this is something we're open to doing instead. I admit that in our response to the reporter, we explicitly said this "is not a priority" but shortly after decided to schedule this work for the next major TF release. We should've followed up again, but didn't. And that's where we're at currently! I hope this helps make our response to the report and our future roadmap around this issue more clear.

deleted 9y ago

[deleted]

asveikau 9y ago

Good thing they didn't write this in C, or their library would have real security trouble. /s

jakewins 9y ago

When we built the new set of drivers for Neo4j, we decided to allow three modes: No encryption, Trust on First Use and Trusted Signature - there's no way to establish an connection without trust. This was a terrifying decision, because of ease of use concerns. Having done so and shipped it, TL;DR: It worked awesome, outside of some early kinks in TOFU that we worked out - and now everyone can sleep well knowing there's not a single install that thinks they are running an encrypted setup when they really aren't. Anyone that came back asking for a flag to disable host key verification seemed happy with our argument for why that's not really much different from just disabling encryption. See "Trust" here: https://neo4j.com/docs/developer-manual/current/drivers/configure-connect/ https://neo4j.com/docs/developer-manual/current/drivers/conf... If you're interested in doing this as well, we wrote code to do it in Python, JS, Java and C#, it's all Apache licensed: JS: https://github.com/neo4j/neo4j-javascript-driver/blob/1.2/src/v1/internal/ch-node.js#L106 https://github.com/neo4j/neo4j-javascript-driver/blob/1.2/sr... Python: https://github.com/neo4j/neo4j-python-driver/blob/1.2/neo4j/bolt/connection.py#L463 https://github.com/neo4j/neo4j-python-driver/blob/1.2/neo4j/... Java: https://github.com/neo4j/neo4j-java-driver/blob/1.3/driver/src/main/java/org/neo4j/driver/internal/security/TLSSocketChannel.java#L68 https://github.com/neo4j/neo4j-java-driver/blob/1.3/driver/s... C#: https://github.com/neo4j/neo4j-dotnet-driver/blob/1.3/Neo4j.Driver/Neo4j.Driver/Internal/Connector/ITrustStrategy.cs#L28 https://github.com/neo4j/neo4j-dotnet-driver/blob/1.3/Neo4j....

ak217 9y ago

FYI, you can also instruct cloud-init to use a particular key pair by supplying it in instance metadata/user-data. This avoids the need for hacky scripts extracting public keys from console output (which may also be delayed by a few minutes after the instance starts).