21 ms·

Show HN: Caddy-SSH

280 points by m_sahaf 5y ago

m_sahaf 5y ago
Hi everyone! Author here This has been my stress-reliever for the past ~2 years. I'm sticking around, so feel free to ask any questions. Github Repo: https://github.com/mohammed90/caddy-ssh https://github.com/mohammed90/caddy-ssh
- explorigin 5y ago
  Cool! (not a security professional) One of the reasons in my understanding that projects don't deviate from older languages was because they give you some control around compilation that would be necessary to thwart timing-based attacks. Does this consider timing-based attacks or is that at a lower-level library?
  - gunapologist99 5y ago
    Excellent point; it does appear that the author did consider timing attacks in at least one location: https://github.com/mohammed90/caddy-ssh/blob/master/internal/ssh/ssh.go#L126 https://github.com/mohammed90/caddy-ssh/blob/master/internal...
    - m_sahaf 5y ago
      To be fair, this bit is borrowed/forked from github.com/gliderlabs/ssh.
      - thefreeman 5y ago
        Curious why you would just copy the entire gilderlabs/ssh package into your repo instead of referencing it as a module? How do you plan to keep up to date with bug / security fixes?
        francislavoie 5y ago
        Some of the changes necessary were too invasive/breaking to gliderlabs/ssh, such as https://github.com/gliderlabs/ssh/pull/161 https://github.com/gliderlabs/ssh/pull/161 so making a copy ended up making more sense, I think. That's my understanding from what Mohammed's told me, anyways.
        m_sahaf 5y ago
        That's right. Moreover, the activity no the repo has been a bit stagnant (no disrespect to the maintainers, they are likely busy with other projects or life). Other projects have opted to fork the repo, rewrite the module repo path, and maintain their fork (e.g. https://github.com/tailscale/ssh/ https://github.com/tailscale/ssh/, but I've seen many others too). I didn't want to maintain a fork, so a copy under internal/ in my own repo means it's firm-fork (between soft and hard) where I can confidently break its APIs I'm the only one depending on without worry. It will slowly morph to fit the needs of the project only. The maintainers gliderlabs/ssh have plans for a new version with more ergonomic APIs but the work hasn't started yet. I didn't want to wait either.
  - m_sahaf 5y ago
    Not very too level. I had to take that into consideration at some parts. For example, the static username_password provider calls a hasher defined in Caddy which uses `subtle.ConstantTimeCompare` function used. At other places, I don't return early (when possible) on auth failures to avoid timing attacks. That said, I'd love to know if there are places where I fell short.
  - enneff 5y ago
    I have reviewed some of the crypto code in the Go standard library that this is built with, and there is use of constant time primitives in there so it is at least possible and some attention has been spent on it.
- XzAeRosho 5y ago
  Congratulations! Looks like a really cool project. So, if I get this right, this should be a drop-in replacement for current SSH servers? I get that an adapter has to be built to support sshd config files, but is that the ultimate goal here? Is security the main selling point of this project?
  - m_sahaf 5y ago
    Thank you! Indeed, you're right. The ultimate goal is to be drop-in replacement. There is a PR hanging waiting to be taken up. I might look at it once I finish furnishing the foundation, but others are welcome to take it up!
- DHowett 5y ago
  Hey! This is awesome. I’m the engineering manager for the Windows console subsystem team. I’d be happy to help out with your ConPTY issues! Feel free to file a discussion issue over on GitHub at microsoft/terminal, or email me at duhowett@(corporate domain name). I suspect what you need is CreatePseudoConsoleAsUser… which we should have offered as a public API.
  - password4321 5y ago
    Can you point the right person to this internally for the real OpenSSH shipped with Windows? I'm actually curious how licensing works out for 3rd party servers like Caddy-SSH. Licensing / Multi-user access / CAL | https://github.com/PowerShell/Win32-OpenSSH/issues/926 https://github.com/PowerShell/Win32-OpenSSH/issues/926 (Oct 2017)
- nodesocket 5y ago
  Am I understanding that the public keys used for authorization (authorized_keys) can come from a centralized source? Being able to add and remove authorized keys from say Redis or Consul centrally would be extremely useful from a management perspective. Obviously, security of that Redis or Consul would have to be tight and prevent public access.
  - frutiger 5y ago
    Slightly off-topic, but OpenSSH already supports this via `AuthorizedKeysCommand`: https://man.openbsd.org/sshd_config#AuthorizedKeysCommand https://man.openbsd.org/sshd_config#AuthorizedKeysCommand
    - nodesocket 5y ago
      Oh really interesting. Any good guides you know of how to implement AuthorizedKeysCommand with Redis for example?
      - macno 5y ago
        Hi @nodesocket, you can take a look how we solved it with Theo https://theoapp.readthedocs.io/en/latest/index.html https://theoapp.readthedocs.io/en/latest/index.html It supports fine hosts/users grants - i.e. I can connect as "dev" user on servers "node1" and "node2" but not on "node3" - and it leverages asymmetric key signing to validate the public SSH keys. Theo supports mysql/mariadb/sqlite/postgresql(experimental) for storing data and redis/memecached for caching. Happy to answer any further questions!
      - frutiger 5y ago
        I'm not really an expert on redis. Perhaps one can fashion something with `redis-cli`? Or write a program that links the redis client library.
  - m_sahaf 5y ago
    That's right! Currently such (static) keys are loaded at provision-time, when the server is first booting up, but there's nothing preventing it from being lazy-loaded at authentication-time. Of course loading them at authentication-time incurs latency as tax. Nothing prevents two separate modules from existing: one eager-loads the keys, and another lazy-loads them.
- jamal-kumar 5y ago
  Hmm interesting. Considering memory corruption vulnerabilities have been pretty much the vast rarity in the OpenSSH codebase, and other classes of vulnerabilties have been more common, I'm curious to know what you're doing to address those classes of vulnerabilities as well as support for other best practices like SSH certificates? [2] I think I see the appeal of an entirely memory-safe OS running an entirely memory-safe SSH implementation because if we're talking about eliminating that class of vulnerability, you might as well go the whole hog - I just don't see that close to the point of being battle tested yet nor taking care of stuff like side channel attacks or tricking people into thinking their TOFU isn't smelly. I like your project because I think that this stuff is important, I think it's just like "how do you improve on the original" may actually be more than just eliminating one class of vuln. [1] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=openssh https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=openssh [2] https://smallstep.com/blog/use-ssh-certificates/ https://smallstep.com/blog/use-ssh-certificates/
yjftsjthsd-h 5y ago
> The ISRG estimates ~80% of the vulnerabilities exploited in the wild are memory safety bugs. Okay, but 1. How many vulnerabilities has openssh shipped, and 2. How many of those were memory issues? I would usually be tentatively on board, but you're competing with the OpenBSD folks, who have a shockingly good track record regardless of using C. No offense, but you could write in a formally verified Ada subset and I'd still hesitate to replace my SSH daemon. (FWIW, I say all of this hoping to be wrong; an alternative implementation, if equally secure, would be great to have.)
- voxic11 5y ago
  Not sure about the statistics but one of the most memorable vulnerabilities in recent history, Heartbleed, was exactly this, a OpenSSH memory safety issue.
  - lawl 5y ago
    No. Heartbleed was OpenSSL, not OpenSSH.
  - photon-torpedo 5y ago
    Heartbleed was OpenSSL, not OpenSSH.
  - spockz 5y ago
    That was in OpenSSL not OpenSSH, two entirely different beasts.
  - deleted 5y ago
    [deleted]
  - lucideer 5y ago
    Sibling commenters have already pointed out you're confusing OpenSSH with OpenSSL. For additional context, since the gp mentioned that: > the OpenBSD folks, who have a shockingly good track record Not only is OpenSSH a BSD project, but so is LibreSSL (the OpenSSL fork that was a response to Heartbleed). Before LibreSSL, BSD folk were also working on assl - similarly an OpenSSL alternative motivated by the BSD guys having concerns about OpenSSL's codebase. So they very much have a solid track record of being pro-actively on top of this type of stuff.
GordonS 5y ago
Pluggable auth providers for SSH sounds awesome, though I can't think of a particular use for it right now!
- gunapologist99 5y ago
  Ironically, these already exist for OpenSSH and have been developed for Linux and UNIX for decades. It's funny how people keep reinventing things. You can use them in your /etc/ssh/sshd_config with "UsePam yes", but, to your point!, generally the built-in auth is a better option for a smaller surface area, unless you specifically know you need PAM. They are literally called Pluggable Authentication Modules (PAM) and it's a core part of any Linux distribution. (I'm not aware of any modern UNIX or Linux that do not offer PAM, but perhaps some embedded distros.) They can be configured for many types of server software, not just OpenSSH. (For example, mail servers like Dovecot and Postfix support PAM, database servers like Postgresql and MySQL, etc.) https://tldp.org/HOWTO/User-Authentication-HOWTO/x115.html https://tldp.org/HOWTO/User-Authentication-HOWTO/x115.html
  - francislavoie 5y ago
    > It's funny how people keep reinventing things. That's not exactly fair. The entire point of this exercise is to move away from C code, by implementing it in a memory safe language (Go). Since PAM uses shared-libraries to operate, that's fundamentally incompatible here since Go is statically compiled (unless you use some CGO like in https://github.com/msteinert/pam https://github.com/msteinert/pam) so implementing auth via Caddy's module system is the way to go for this project.
    - candiddevmike 5y ago
      Memory safety won't save you from logic bugs, which is extremely prevalent in authentication exploits.
      - francislavoie 5y ago
        No arguments there, but it's still much safer to start with a memory-safe language as a base.
jon-wood 5y ago
What's the benefit of this being built on top of Caddy, which unless I'm mistaken has historically been an HTTP(S) server? Maybe there's someone intrinsically great about how Caddy works for this, but on first glance it feels like the scope creep in servers like Apache, or Curl's support for querying every data source under the sun. It can be handy having a single tool that does everything, but at the same time that gives an ever larger attack surface where suddenly your machine has been compromised via instructions from an MQTT server because Curl was installed.
- francislavoie 5y ago
  Caddy v2 is actually designed as an "app platform". The HTTP app is just one such type of app, and is what Caddy ships with by default. But it can be extended to do _anything_. See https://caddyserver.com/docs/architecture https://caddyserver.com/docs/architecture You don't have to run both the HTTP app and the SSH app in the same process, you could have two separate Caddy processes configured to each do their own thing, if you're at all concerned.
  - pantulis 5y ago
    Isn't this like a super-server in the spirit of inetd?
    - francislavoie 5y ago
      I guess, except Caddy doesn't spawn any processes, it just "starts apps" which are configured in-process, pure Go. Another example Caddy app is https://github.com/mholt/caddy-l4 https://github.com/mholt/caddy-l4 which lets you do arbitrary TCP/UDP handling/proxying. There's a list of available apps here https://caddyserver.com/docs/json/apps/ https://caddyserver.com/docs/json/apps/ (the SSH app is not there yet, should be soon)
      - jrockway 5y ago
        I think it's fine for a conceptual understanding. The Go runtime looks a lot like an OS, mapping G (goroutines) to Ms (worker threads), removing goroutines from the workers when they block in syscalls, waking up goroutines when a lock they want is available or a syscall finishes, etc. In the end it really depends on what you think of inetd as. If you think of it as "I install this one server and now I have finger and gopher!" then that seems similar to Caddy's applications. If you think of it as "whenever a TCP connection arrives, the fork() syscall is used to create a process to handle that session, file descriptors 0 and 1 are connected to the TCP socket, and then exec() according to the global config is run to handle the protocol", then ... no it's not the same as inetd.
foxtrottbravo 5y ago
First and foremost, congratulations on bringing the project to this stage - I think it's an impressive piece of work. I am in no way qualified to trample on your parade but two things came to my mind that pinch a personal nerve of mine and I would really like to have alleviated by you or the folks who know that stuff: - if your Goal was "secure by default", why did you allow passwords in the first place? Following Caddys recipe would be more like SSH-Keys only, wouldn't it? Is there a reason other than compatibility? - In that same avenue? Why allow such a thing as downloading authorized keys from a third party? Domain takeovers or account compromises on say Github are a thing - so again while it may be a nice usability aspect isn't that contrary to the secure by default pradigm? Again thank you for your work and congratulations on the project - those above are just honest questions that came to mind which I would really like to be educated on
- getcrunk 5y ago
  Passwords are not insecure. Certs have tradeoffs.
  - dewey 5y ago
    I don't read it as "passwords are insecure" but as having defaults that force people into using it in ways that are harder to mess up. Re-using an insecure password is easy, using a key wrong is harder.
    - moondev 5y ago
      re-using an insecure key is just as easy https://github.com/mikalv/anything2ed25519 https://github.com/mikalv/anything2ed25519
    - LinuxBender 5y ago
      I understand what you are saying but in my experience the opposite can be equally true. Unless every system in your org is managing the authorized_keys for every account with automation and hourly validation one can end up with rogue keys, forgotten keys, unmanaged keys. SSH has no concept of identity in this regard. If I temporarily have sudo on a system I can append any random public key into the authorized_keys of any account. It gets even worse if that system allows passwordless sudo. Now there is an audit trail that showed you made a reckless change when it was really me. This risk gets exponentially worse if the system I append a random key into is a command and control or configuration management system. One doesn't even really need sudo for this to be a problem given how much power non root accounts have over applications, deployments, builds and secret management.
g_p 5y ago
This looks very interesting. Is there any support (or plans) for SSH certificates? They help to manage some of the revocation and access control challenges, as well as the issues around trust-on-first-use and similar. (And also the fragility of syncing around authorized-keys files, or relying on LDAP for login, in infrastructure type environments). See also - https://news.ycombinator.com/item?id=30788544 https://news.ycombinator.com/item?id=30788544
- eliaspro 5y ago
  I had to think of exactly this post as well and I started to wonder, whether Caddy-SSH couldn't even handle this using Caddy's built-in ACME support?
  - mholt 5y ago
    It could, if there was an ACME server doling out SSH certificates. And Caddy can get certificates via more than just ACME, it's completely extensible, so it will work with... anything I guess.
  - tialaramex 5y ago
    Although the word "Certificate" is in the name of this feature and the expansion of ACME, they don't really have much in common. ACME is similar to, and different from the Internet's other certificate protocols, both in ways that are not helpful for a SSH server. Firstly ACME is similar in that it's about X.509 certificates roughly PKIX flavoured, whereas certificates in SSH don't use X.509 and thus don't use PKIX Then it's different in that ACME solves the proof-of-control problem for the Web PKI, it was actually built in parallel with what became the "10 Blessed Methods" for the Web PKI (but today there are not ten of them). But almost nobody wants public SSH servers for random people from the Internet to access, so the Web PKI is largely irrelevant to certificates for SSH servers. [ The Web PKI is the name for the PKI that makes your web browser work, unlike SSH you actually do use web browsers to connect to random public stuff - lots of other things rely on the Web PKI, not just the Web but there are good reasons it is called the Web PKI anyway ]
- m_sahaf 5y ago
sneak 5y ago
Note that outsourcing your key list to a live GitHub URL gives Microsoft unfettered access to your box, should they (or anyone who can compel them, such as the US armed forces) ever want or need it. If you wouldn't use Microsoft SSO for local login, you should not thus configure your sshd that way.
- zufallsheld 5y ago
  How does outsourcing your key list give ms access to my box? Presumably you mean because they also have the private key?
  - yjftsjthsd-h 5y ago
    Because they can serve any list of public keys they want; they can return a blank list and lock you out, or append another key at will.
  - sneak 5y ago
    No, the keys on GitHub are only the public portion. If your sshd consults Microsoft servers to check a key, Microsoft can serve it any file they wish (or are forced to), such as a list of Equation Group pubkeys.
donatj 5y ago
I'm not sure I fully understand, what's the advantage of getting Caddy involved here versus a fully standalone app? Given the name I'd at first figured it was an official Caddy project, but that does not seem to be the case.
- m_sahaf 5y ago
  Names are hard. I did consider changing the name but couldn't find another one. Feel free to pitch names on GitHub. Caddy provides the module system and the config management backed by powerful REST API for config query and patching. If it wasn't for Caddy, I would've had to build all of that from scratch. That's too much to own.
  - pronoiac 5y ago
    I filed an issue to suggest caddysshack.
achairapart 5y ago
Speaking of sane defaults, I wonder how many people are aware that Caddy serves dotfiles by default.
- mholt 5y ago
  That's incorrect. Caddy does not serve any files by default. You have to explicitly enable file serving in your config.
  - achairapart 5y ago
    Sorry Matt, didn't want to be rude. But yes, it does. Using the file_serve directive, dotfiles are served by default. Here are some random domains taken from the Caddy Forum threads: https://lastfree.space/.git/HEAD https://lastfree.space/.git/HEAD https://www.aspecthq.com/.git/HEAD https://www.aspecthq.com/.git/HEAD Those people are unaware of serving their git (or .env files or other secrets) over the internet, probably because are used to other webservers that just block this behaviour by default. And it's ok to not do or be like the others, but this should at least be mentioned in the docs.
    - SirGiggles 5y ago
      What you initially stated implies a completely different meaning. You initially made the implication that Caddy would enable the file server implicitly when it does not. That was what I interpreted upon first reading. Your correction states that dotfiles are served using the file_server directive which is correct and and example is given in hiding the .git folder [1] [1] https://caddyserver.com/docs/caddyfile/directives/file_server https://caddyserver.com/docs/caddyfile/directives/file_serve...
    - mholt 5y ago
      With all due respect, you're wrong. If Caddy is serving any files, the user explicitly enabled that functionality. And when enabled, it only serves files within the root directory specified by the user, or the current working directory, and the docs are very clear about this: https://caddyserver.com/docs/modules/http.handlers.file_server#root https://caddyserver.com/docs/modules/http.handlers.file_serv...
elischleifer 5y ago
Biggest obstacle to proper SSH usage tends to be the creation and management of people's private keys. The number of times people have to google for instructions for doing this right is insane. Let's get someone working on that.
- dapirian 5y ago
  true story
- the_angry_angel 5y ago
  Teleport I feel solves this well, from an organisation perspective
  - elischleifer 5y ago
    Cool - was not aware of that service. Will have to check it out.
  - AtlasBarfed 5y ago
    At my org, if I use teleport, I have to MFA in via a UI, and they offer zero other alternatives. The teleport daemon falls over due to memory stress and other issues that sshd doesn't fall over during, and when do you often need CLI access to a node? When it's under stress.... Goodbye automation. It's a good idea, but it's made ssh and the problems I solve with ssh worse for me.
- deleted 5y ago
  [deleted]

280 points by m_sahaf 5y ago

Hi everyone! Author here This has been my stress-reliever for the past ~2 years. I'm sticking around, so feel free to ask any questions. Github Repo: https://github.com/mohammed90/caddy-ssh https://github.com/mohammed90/caddy-ssh

explorigin 5y ago
Cool! (not a security professional) One of the reasons in my understanding that projects don't deviate from older languages was because they give you some control around compilation that would be necessary to thwart timing-based attacks. Does this consider timing-based attacks or is that at a lower-level library?
- gunapologist99 5y ago
  Excellent point; it does appear that the author did consider timing attacks in at least one location: https://github.com/mohammed90/caddy-ssh/blob/master/internal/ssh/ssh.go#L126 https://github.com/mohammed90/caddy-ssh/blob/master/internal...
  - m_sahaf 5y ago
    To be fair, this bit is borrowed/forked from github.com/gliderlabs/ssh.
    - thefreeman 5y ago
      Curious why you would just copy the entire gilderlabs/ssh package into your repo instead of referencing it as a module? How do you plan to keep up to date with bug / security fixes?
      - francislavoie 5y ago
        Some of the changes necessary were too invasive/breaking to gliderlabs/ssh, such as https://github.com/gliderlabs/ssh/pull/161 https://github.com/gliderlabs/ssh/pull/161 so making a copy ended up making more sense, I think. That's my understanding from what Mohammed's told me, anyways.
        m_sahaf 5y ago
        That's right. Moreover, the activity no the repo has been a bit stagnant (no disrespect to the maintainers, they are likely busy with other projects or life). Other projects have opted to fork the repo, rewrite the module repo path, and maintain their fork (e.g. https://github.com/tailscale/ssh/ https://github.com/tailscale/ssh/, but I've seen many others too). I didn't want to maintain a fork, so a copy under internal/ in my own repo means it's firm-fork (between soft and hard) where I can confidently break its APIs I'm the only one depending on without worry. It will slowly morph to fit the needs of the project only. The maintainers gliderlabs/ssh have plans for a new version with more ergonomic APIs but the work hasn't started yet. I didn't want to wait either.
- m_sahaf 5y ago
  Not very too level. I had to take that into consideration at some parts. For example, the static username_password provider calls a hasher defined in Caddy which uses `subtle.ConstantTimeCompare` function used. At other places, I don't return early (when possible) on auth failures to avoid timing attacks. That said, I'd love to know if there are places where I fell short.
- enneff 5y ago
  I have reviewed some of the crypto code in the Go standard library that this is built with, and there is use of constant time primitives in there so it is at least possible and some attention has been spent on it.
XzAeRosho 5y ago
Congratulations! Looks like a really cool project. So, if I get this right, this should be a drop-in replacement for current SSH servers? I get that an adapter has to be built to support sshd config files, but is that the ultimate goal here? Is security the main selling point of this project?
- m_sahaf 5y ago
  Thank you! Indeed, you're right. The ultimate goal is to be drop-in replacement. There is a PR hanging waiting to be taken up. I might look at it once I finish furnishing the foundation, but others are welcome to take it up!
DHowett 5y ago
Hey! This is awesome. I’m the engineering manager for the Windows console subsystem team. I’d be happy to help out with your ConPTY issues! Feel free to file a discussion issue over on GitHub at microsoft/terminal, or email me at duhowett@(corporate domain name). I suspect what you need is CreatePseudoConsoleAsUser… which we should have offered as a public API.
- password4321 5y ago
  Can you point the right person to this internally for the real OpenSSH shipped with Windows? I'm actually curious how licensing works out for 3rd party servers like Caddy-SSH. Licensing / Multi-user access / CAL | https://github.com/PowerShell/Win32-OpenSSH/issues/926 https://github.com/PowerShell/Win32-OpenSSH/issues/926 (Oct 2017)
nodesocket 5y ago
Am I understanding that the public keys used for authorization (authorized_keys) can come from a centralized source? Being able to add and remove authorized keys from say Redis or Consul centrally would be extremely useful from a management perspective. Obviously, security of that Redis or Consul would have to be tight and prevent public access.
- frutiger 5y ago
  Slightly off-topic, but OpenSSH already supports this via `AuthorizedKeysCommand`: https://man.openbsd.org/sshd_config#AuthorizedKeysCommand https://man.openbsd.org/sshd_config#AuthorizedKeysCommand
  - nodesocket 5y ago
    Oh really interesting. Any good guides you know of how to implement AuthorizedKeysCommand with Redis for example?
    - macno 5y ago
      Hi @nodesocket, you can take a look how we solved it with Theo https://theoapp.readthedocs.io/en/latest/index.html https://theoapp.readthedocs.io/en/latest/index.html It supports fine hosts/users grants - i.e. I can connect as "dev" user on servers "node1" and "node2" but not on "node3" - and it leverages asymmetric key signing to validate the public SSH keys. Theo supports mysql/mariadb/sqlite/postgresql(experimental) for storing data and redis/memecached for caching. Happy to answer any further questions!
    - frutiger 5y ago
      I'm not really an expert on redis. Perhaps one can fashion something with `redis-cli`? Or write a program that links the redis client library.
- m_sahaf 5y ago
  That's right! Currently such (static) keys are loaded at provision-time, when the server is first booting up, but there's nothing preventing it from being lazy-loaded at authentication-time. Of course loading them at authentication-time incurs latency as tax. Nothing prevents two separate modules from existing: one eager-loads the keys, and another lazy-loads them.
jamal-kumar 5y ago
Hmm interesting. Considering memory corruption vulnerabilities have been pretty much the vast rarity in the OpenSSH codebase, and other classes of vulnerabilties have been more common, I'm curious to know what you're doing to address those classes of vulnerabilities as well as support for other best practices like SSH certificates? [2] I think I see the appeal of an entirely memory-safe OS running an entirely memory-safe SSH implementation because if we're talking about eliminating that class of vulnerability, you might as well go the whole hog - I just don't see that close to the point of being battle tested yet nor taking care of stuff like side channel attacks or tricking people into thinking their TOFU isn't smelly. I like your project because I think that this stuff is important, I think it's just like "how do you improve on the original" may actually be more than just eliminating one class of vuln. [1] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=openssh https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=openssh [2] https://smallstep.com/blog/use-ssh-certificates/ https://smallstep.com/blog/use-ssh-certificates/

yjftsjthsd-h 5y ago

> The ISRG estimates ~80% of the vulnerabilities exploited in the wild are memory safety bugs. Okay, but 1. How many vulnerabilities has openssh shipped, and 2. How many of those were memory issues? I would usually be tentatively on board, but you're competing with the OpenBSD folks, who have a shockingly good track record regardless of using C. No offense, but you could write in a formally verified Ada subset and I'd still hesitate to replace my SSH daemon. (FWIW, I say all of this hoping to be wrong; an alternative implementation, if equally secure, would be great to have.)

voxic11 5y ago
Not sure about the statistics but one of the most memorable vulnerabilities in recent history, Heartbleed, was exactly this, a OpenSSH memory safety issue.
- lawl 5y ago
  No. Heartbleed was OpenSSL, not OpenSSH.
- photon-torpedo 5y ago
  Heartbleed was OpenSSL, not OpenSSH.
- spockz 5y ago
  That was in OpenSSL not OpenSSH, two entirely different beasts.
- deleted 5y ago
  [deleted]
- lucideer 5y ago
  Sibling commenters have already pointed out you're confusing OpenSSH with OpenSSL. For additional context, since the gp mentioned that: > the OpenBSD folks, who have a shockingly good track record Not only is OpenSSH a BSD project, but so is LibreSSL (the OpenSSL fork that was a response to Heartbleed). Before LibreSSL, BSD folk were also working on assl - similarly an OpenSSL alternative motivated by the BSD guys having concerns about OpenSSL's codebase. So they very much have a solid track record of being pro-actively on top of this type of stuff.

GordonS 5y ago

Pluggable auth providers for SSH sounds awesome, though I can't think of a particular use for it right now!

gunapologist99 5y ago
Ironically, these already exist for OpenSSH and have been developed for Linux and UNIX for decades. It's funny how people keep reinventing things. You can use them in your /etc/ssh/sshd_config with "UsePam yes", but, to your point!, generally the built-in auth is a better option for a smaller surface area, unless you specifically know you need PAM. They are literally called Pluggable Authentication Modules (PAM) and it's a core part of any Linux distribution. (I'm not aware of any modern UNIX or Linux that do not offer PAM, but perhaps some embedded distros.) They can be configured for many types of server software, not just OpenSSH. (For example, mail servers like Dovecot and Postfix support PAM, database servers like Postgresql and MySQL, etc.) https://tldp.org/HOWTO/User-Authentication-HOWTO/x115.html https://tldp.org/HOWTO/User-Authentication-HOWTO/x115.html
- francislavoie 5y ago
  > It's funny how people keep reinventing things. That's not exactly fair. The entire point of this exercise is to move away from C code, by implementing it in a memory safe language (Go). Since PAM uses shared-libraries to operate, that's fundamentally incompatible here since Go is statically compiled (unless you use some CGO like in https://github.com/msteinert/pam https://github.com/msteinert/pam) so implementing auth via Caddy's module system is the way to go for this project.
  - candiddevmike 5y ago
    Memory safety won't save you from logic bugs, which is extremely prevalent in authentication exploits.
    - francislavoie 5y ago
      No arguments there, but it's still much safer to start with a memory-safe language as a base.

jon-wood 5y ago

What's the benefit of this being built on top of Caddy, which unless I'm mistaken has historically been an HTTP(S) server? Maybe there's someone intrinsically great about how Caddy works for this, but on first glance it feels like the scope creep in servers like Apache, or Curl's support for querying every data source under the sun. It can be handy having a single tool that does everything, but at the same time that gives an ever larger attack surface where suddenly your machine has been compromised via instructions from an MQTT server because Curl was installed.

francislavoie 5y ago
Caddy v2 is actually designed as an "app platform". The HTTP app is just one such type of app, and is what Caddy ships with by default. But it can be extended to do _anything_. See https://caddyserver.com/docs/architecture https://caddyserver.com/docs/architecture You don't have to run both the HTTP app and the SSH app in the same process, you could have two separate Caddy processes configured to each do their own thing, if you're at all concerned.
- pantulis 5y ago
  Isn't this like a super-server in the spirit of inetd?
  - francislavoie 5y ago
    I guess, except Caddy doesn't spawn any processes, it just "starts apps" which are configured in-process, pure Go. Another example Caddy app is https://github.com/mholt/caddy-l4 https://github.com/mholt/caddy-l4 which lets you do arbitrary TCP/UDP handling/proxying. There's a list of available apps here https://caddyserver.com/docs/json/apps/ https://caddyserver.com/docs/json/apps/ (the SSH app is not there yet, should be soon)
    - jrockway 5y ago
      I think it's fine for a conceptual understanding. The Go runtime looks a lot like an OS, mapping G (goroutines) to Ms (worker threads), removing goroutines from the workers when they block in syscalls, waking up goroutines when a lock they want is available or a syscall finishes, etc. In the end it really depends on what you think of inetd as. If you think of it as "I install this one server and now I have finger and gopher!" then that seems similar to Caddy's applications. If you think of it as "whenever a TCP connection arrives, the fork() syscall is used to create a process to handle that session, file descriptors 0 and 1 are connected to the TCP socket, and then exec() according to the global config is run to handle the protocol", then ... no it's not the same as inetd.

foxtrottbravo 5y ago

First and foremost, congratulations on bringing the project to this stage - I think it's an impressive piece of work. I am in no way qualified to trample on your parade but two things came to my mind that pinch a personal nerve of mine and I would really like to have alleviated by you or the folks who know that stuff: - if your Goal was "secure by default", why did you allow passwords in the first place? Following Caddys recipe would be more like SSH-Keys only, wouldn't it? Is there a reason other than compatibility? - In that same avenue? Why allow such a thing as downloading authorized keys from a third party? Domain takeovers or account compromises on say Github are a thing - so again while it may be a nice usability aspect isn't that contrary to the secure by default pradigm? Again thank you for your work and congratulations on the project - those above are just honest questions that came to mind which I would really like to be educated on

getcrunk 5y ago
Passwords are not insecure. Certs have tradeoffs.
- dewey 5y ago
  I don't read it as "passwords are insecure" but as having defaults that force people into using it in ways that are harder to mess up. Re-using an insecure password is easy, using a key wrong is harder.
  - moondev 5y ago
    re-using an insecure key is just as easy https://github.com/mikalv/anything2ed25519 https://github.com/mikalv/anything2ed25519
  - LinuxBender 5y ago
    I understand what you are saying but in my experience the opposite can be equally true. Unless every system in your org is managing the authorized_keys for every account with automation and hourly validation one can end up with rogue keys, forgotten keys, unmanaged keys. SSH has no concept of identity in this regard. If I temporarily have sudo on a system I can append any random public key into the authorized_keys of any account. It gets even worse if that system allows passwordless sudo. Now there is an audit trail that showed you made a reckless change when it was really me. This risk gets exponentially worse if the system I append a random key into is a command and control or configuration management system. One doesn't even really need sudo for this to be a problem given how much power non root accounts have over applications, deployments, builds and secret management.

g_p 5y ago

This looks very interesting. Is there any support (or plans) for SSH certificates? They help to manage some of the revocation and access control challenges, as well as the issues around trust-on-first-use and similar. (And also the fragility of syncing around authorized-keys files, or relying on LDAP for login, in infrastructure type environments). See also - https://news.ycombinator.com/item?id=30788544 https://news.ycombinator.com/item?id=30788544

eliaspro 5y ago
I had to think of exactly this post as well and I started to wonder, whether Caddy-SSH couldn't even handle this using Caddy's built-in ACME support?
- mholt 5y ago
  It could, if there was an ACME server doling out SSH certificates. And Caddy can get certificates via more than just ACME, it's completely extensible, so it will work with... anything I guess.
- tialaramex 5y ago
  Although the word "Certificate" is in the name of this feature and the expansion of ACME, they don't really have much in common. ACME is similar to, and different from the Internet's other certificate protocols, both in ways that are not helpful for a SSH server. Firstly ACME is similar in that it's about X.509 certificates roughly PKIX flavoured, whereas certificates in SSH don't use X.509 and thus don't use PKIX Then it's different in that ACME solves the proof-of-control problem for the Web PKI, it was actually built in parallel with what became the "10 Blessed Methods" for the Web PKI (but today there are not ten of them). But almost nobody wants public SSH servers for random people from the Internet to access, so the Web PKI is largely irrelevant to certificates for SSH servers. [ The Web PKI is the name for the PKI that makes your web browser work, unlike SSH you actually do use web browsers to connect to random public stuff - lots of other things rely on the Web PKI, not just the Web but there are good reasons it is called the Web PKI anyway ]
m_sahaf 5y ago

sneak 5y ago

Note that outsourcing your key list to a live GitHub URL gives Microsoft unfettered access to your box, should they (or anyone who can compel them, such as the US armed forces) ever want or need it. If you wouldn't use Microsoft SSO for local login, you should not thus configure your sshd that way.

zufallsheld 5y ago
How does outsourcing your key list give ms access to my box? Presumably you mean because they also have the private key?
- yjftsjthsd-h 5y ago
  Because they can serve any list of public keys they want; they can return a blank list and lock you out, or append another key at will.
- sneak 5y ago
  No, the keys on GitHub are only the public portion. If your sshd consults Microsoft servers to check a key, Microsoft can serve it any file they wish (or are forced to), such as a list of Equation Group pubkeys.

donatj 5y ago

I'm not sure I fully understand, what's the advantage of getting Caddy involved here versus a fully standalone app? Given the name I'd at first figured it was an official Caddy project, but that does not seem to be the case.

m_sahaf 5y ago
Names are hard. I did consider changing the name but couldn't find another one. Feel free to pitch names on GitHub. Caddy provides the module system and the config management backed by powerful REST API for config query and patching. If it wasn't for Caddy, I would've had to build all of that from scratch. That's too much to own.
- pronoiac 5y ago
  I filed an issue to suggest caddysshack.

achairapart 5y ago

Speaking of sane defaults, I wonder how many people are aware that Caddy serves dotfiles by default.

mholt 5y ago
That's incorrect. Caddy does not serve any files by default. You have to explicitly enable file serving in your config.
- achairapart 5y ago
  Sorry Matt, didn't want to be rude. But yes, it does. Using the file_serve directive, dotfiles are served by default. Here are some random domains taken from the Caddy Forum threads: https://lastfree.space/.git/HEAD https://lastfree.space/.git/HEAD https://www.aspecthq.com/.git/HEAD https://www.aspecthq.com/.git/HEAD Those people are unaware of serving their git (or .env files or other secrets) over the internet, probably because are used to other webservers that just block this behaviour by default. And it's ok to not do or be like the others, but this should at least be mentioned in the docs.
  - SirGiggles 5y ago
    What you initially stated implies a completely different meaning. You initially made the implication that Caddy would enable the file server implicitly when it does not. That was what I interpreted upon first reading. Your correction states that dotfiles are served using the file_server directive which is correct and and example is given in hiding the .git folder [1] [1] https://caddyserver.com/docs/caddyfile/directives/file_server https://caddyserver.com/docs/caddyfile/directives/file_serve...
  - mholt 5y ago
    With all due respect, you're wrong. If Caddy is serving any files, the user explicitly enabled that functionality. And when enabled, it only serves files within the root directory specified by the user, or the current working directory, and the docs are very clear about this: https://caddyserver.com/docs/modules/http.handlers.file_server#root https://caddyserver.com/docs/modules/http.handlers.file_serv...

elischleifer 5y ago

Biggest obstacle to proper SSH usage tends to be the creation and management of people's private keys. The number of times people have to google for instructions for doing this right is insane. Let's get someone working on that.

dapirian 5y ago
true story
the_angry_angel 5y ago
Teleport I feel solves this well, from an organisation perspective
- elischleifer 5y ago
  Cool - was not aware of that service. Will have to check it out.
- AtlasBarfed 5y ago
  At my org, if I use teleport, I have to MFA in via a UI, and they offer zero other alternatives. The teleport daemon falls over due to memory stress and other issues that sshd doesn't fall over during, and when do you often need CLI access to a node? When it's under stress.... Goodbye automation. It's a good idea, but it's made ssh and the problems I solve with ssh worse for me.
deleted 5y ago
[deleted]