12 ms·

Fearless SSH: Short-lived certificates bring Zero Trust to infrastructure

151 points by mfrw 2y ago

koutsie 2y ago
How is trusting Cloudflare "zero-trust" ?
EthanHeilman 2y ago
I'm a member of the team that worked on this happy to answer any questions. We (BastionZero) recently got bought by Cloudflare and it is exciting bringing our SSH ideas to Cloudflare.
- mdaniel 2y ago
  I just wanted to offer my congratulations on the acquisition. I don't know any details about your specific one, but I have been around enough to know that it's still worth celebrating o/
- lenova 2y ago
  I'd love to hear about the acquisition story with Cloudflare.
  - FlyingSnake 2y ago
    You can read the details here: https://blog.cloudflare.com/cloudflare-acquires-bastionzero/ https://blog.cloudflare.com/cloudflare-acquires-bastionzero/
  - EthanHeilman 2y ago
    Are particular questions? So far my experience with joining and working at Cloudflare has been fantastic. Coming from a background of startups and academia, the size and scope of what Cloudflare is building and currently runs is overwhelming. In academia I've seen lots of excellent academic computer science papers that never benefit anyone because they never get turned into a tool that someone can just pick up and use. Ideas have inherent value, even useless ideas, but it feels good to see great ideas have impact. What appealed to me the most about getting acquired by Cloudflare is seeing research applied directly to products and used by people. Cloudflare does an excellent job both inventing innovative ideas and then actually making them real. There used to be a lot of companies that did this 10 years ago, but Cloudflare now seems rare in that respect.
mdaniel 2y ago
I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For context, I'm also one of those "immutable OS or GTFO" chaps because in my experience the next thing that happens after some rando ssh-es into a machine is they launch vi or apt-get or whatever and now it's a snowflake with zero auditing of the actions taken to it I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again
- namxam 2y ago
  But what is the alternative?
  - mdaniel 2y ago
    There's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-of-settings https://github.com/bottlerocket-os/bottlerocket#description-...> In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box
    - cyberax 2y ago
      Be careful with SSM, it can provide pretty much unlimited access: https://github.com/Cyberax/gimlet https://github.com/Cyberax/gimlet You can use it to tunnel arbitrary traffic inside your VPC.
antoniomika 2y ago
I wrote a system that did this >5 years ago (luckily was able to open source it before the startup went under[0]). The bastion would record ssh sessions in asciicast v2 format and store those for later playback directly from a control panel. The main issue that still isn't solved by a solution like this is user management on the remote (ssh server) side. In a more recent implementation, integration with LDAP made the most sense and allows for separation of user and login credentials. A single integrated solution is likely the holy grail in this space. [0] https://github.com/notion/bastion https://github.com/notion/bastion
- mdaniel 2y ago
  Out of curiosity, why ignore this PR? https://github.com/notion/bastion/pull/13 https://github.com/notion/bastion/pull/13 I would think even a simple "sorry, this change does not align with the project's goals" -> closed would help the submitter (and others) have some clarity versus the PR limbo it's currently in That aside, thanks so much for pointing this out: it looks like good fun, especially the Asciicast support!
  - antoniomika 2y ago
    Honestly never had a chance to merge it/review it. Once the company wound down, I had to move onto other things (find a new job, work on other priorities, etc) and lost access to be able to do anything with it after. I thought about forking it and modernizing it but never came to fruition.
edelbitter 2y ago
Why does the title say "Zero Trust", when the article explains that this only works as long as every involved component of the Cloudflare MitM keylogger and its CA can be trusted? If hosts keys are worthless because you do not know in advance what key the proxy will have.. than this scheme is back to trusting servers merely because they are in Cloudflare address space, no?
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- varenc 2y ago
  https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i... Zero Trust just means you stop inherently trusting your private network and verify every user/device/request regardless. If you opt in to using Cloudflare to do this then it requires running Cloudflare software.
  - bdd8f1df777b 2y ago
    But with public key auth I'm already distrusting everyone on my private network.
    - resoluteteeth 2y ago
      Technically I guess that's "zero trust" in the sense of meeting the requirement of not trusting internal connections more than external ones, but in practice I guess "zero trust" also typically entails making every connection go through the same user-based authentication system, which uploading specific keys to specific servers manually definitely doesn't achieve.
  - PLG88 2y ago
    Thats one interpretation... ZT also posits assuming the network is compromised and hostile, that also applies to CF and their cloud/network. It blows my mind that so many solutions claim ZT while mandating TLS to their infra/cloud, you can trust their decryption of your date, and worst IMHO, they will MITM your OICD/SAML key to ensure the endpoint can authenticate and access services... that is a hell of a lot of implicit trust in them, not least of them being served a court order to decrypt your data. Zero trust done correctly done not have those same drawbacks.
johnklos 2y ago
So... don't trust long lived ssh keys, but trust Cloudflare's CA. Why? What has Cloudflare done to earn trust? If that alone weren't reason enough to dismiss this, the article has marketing BS throughout. For instance, "SSH access to a server often comes with elevated privileges". Ummm... Every authentication system ever has whatever privileges that come with that authentication system. This is the kind of bull you say / write when you want to snow someone who doesn't know any better. To those of us who do understand this, this is almost AI level bullshit. The same is true of their supposed selling points: > Author fine-grained policy to govern who can SSH to your servers and through which SSH user(s) they can log in as. That's exactly what ssh does. You set up precisely which authentication methods you accept, you set up keys for exactly that purpose, and you set up individual accounts. Do Cloudflare really think we're setting up a single user account and giving access to lots of different people, and we need them to save us? (now that I think about it, I bet some people do this, but this is still a ridiculous selling point) > Monitor infrastructure access with Access and SSH command logs So they're MITM all of our connections? We're supposed to trust them, even though they have a long history of not only working with scammers and malicious actors, but protecting them? I suppose there's a sucker born every minute, so Cloudflare will undoubtedly sell some people on this silliness, but to me it just looks like yet another way that Cloudflare wants to recentralize the Internet around them. If they had their way, then in a few years, were they to go down, a majority of the Internet would literally stop working. That should scare everyone.
- deleted 2y ago
  [deleted]
tptacek 2y ago
I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA. When I'm selling people on SSH CA's, the first thing I usually have to convince them of is that I'm not saying they should trust some third party. You know where all your servers are. External CAs exist to solve the counterparty introduction problem, which is a problem SSH servers do not have.
- xyst 2y ago
  Same reasons for companies still buying “CrowdStrike” and installing that crapware. It’s all for regulatory checkboxes (ie, fedramp cert).
  - tptacek 2y ago
    I do not believe you in fact need any kind of SSH CA, let alone one run by a third party, to be FedRAMP-compliant.
- kevin_nisbet 2y ago
  I'm with you, I imagine it's mostly people just drawing parallels, they can figure out how to get a web certificate so think SSH is the same thing. The second order problem I've found is when you dig in there are plenty of people who ask for certs but when push comes to shove really want functionality where when user access is cancelled all active sessions get torn down immediatly as well.
- michaelt 2y ago
  > I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA. I think the sales pitch for these sorts of service is: "Get an SSH-like experience, but it integrates with your corporate single-sign-on system, has activity logs that can't be deleted even if you're root on the target, sorts out your every-ephemeral-cloud-instance-has-a-different-fingerprint issues, and we'll sort out all the reverse-tunnelling-through-NAT and bastion-host-for-virtual-private-cloud stuff too" Big businesses pursuing SOC2 compliance love this sort of thing.
xyst 2y ago
Underlying tech is “Openpubkey”. https://github.com/openpubkey/openpubkey https://github.com/openpubkey/openpubkey BastionZero just builds on top of that to provide a “seamless” UX for ssh sessions and some auditing/fedramp certification. Personally, not a fan of relying on CF. Need less centralization/consolidation into a few companies. It’s bad enough with MS dominating the OS (consumer) space. AWS dominating cloud computing. And CF filling the gaps between the stack.
- ranger_danger 2y ago
  Completely agree. I also don't want to trust certificate authorities for my SSH connections let alone CF. Would not be surprised if it/they were compromised.
  - yjftsjthsd-h 2y ago
    > I also don't want to trust certificate authorities for my SSH connections let alone CF. Would not be surprised if it/they were compromised. OpenPubkey, or in general? Normal SSH CAs don't do PKI like browsers use, you make and trust your own CA(s). And if an attacker can compromise your CA private key, why can't they compromise your SSH private key directly?
    - looofooo0 2y ago
      https://www.usenix.org/system/files/login/articles/105484-Gutmann.pdf https://www.usenix.org/system/files/login/articles/105484-Gu... + People just don't check ssh keys normally.
      - yjftsjthsd-h 2y ago
        That's about host keys, not user keys. And... I'm struggling to think of a threat model where that problem manifests in a compromise? Like, what's your threat model? That said, CAs actually really help with that problem, because if a server has its host keys signed with a CA and then the user trusts that CA then they don't have to TOFU the host keys.
shermantanktop 2y ago
I didn’t understand the marketing term “zero trust” and I still don’t. In practice, I get it - a network zone shouldn’t require a lower authn/z bar on the implicit assumption that admission to that zone must have required a higher bar. But all these systems are built on trust, and if it isn’t based on network zoning, it’s based on something else. Maybe that other thing is better, maybe not. But it exists and it needs to be understood. An actual zero trust system is the proverbial unpowered computer in a bunker.
- ngneer 2y ago
  With you there. The marketing term makes Zero Sense to me.
- wmf 2y ago
  The something else is specifically user/service identity. Not machine identity, not IP address. It is somewhat silly to have a buzzword that means "no, actually authenticate users" but here we are.
- athorax 2y ago
  It means there is zero trust of a device/service/user on your network until they have been fully authenticated. It is about having zero trust in something just because it is inside your network perimeter.
  - shermantanktop 2y ago
    Maybe it should be called "zero trust of a device/service/user on your network until they have been fully authenticated." But that wouldn't sell high-dollar consulting services.
- acdha 2y ago
  Yeah, it’s not a great name. Twenty years ago we called it “end to end authentication” and I think that’s better because it focuses on the most important aspect, but it probably doesn’t sound as cool for marketing purposes. I also like how that makes it easier to understand how variation is normal: for example, authentication comes in various flavors and that’s okay whereas some of that zero trust vendors will try to claim that something is or isn’t ZT based on feature gaps in their competitors’ and it’s just so tedious to play that game.
cyberax 2y ago
Hah. I did pretty much all the same stuff in my previous company. One thing that we did a bit better: we used AWS SSM to provision our SSH-CA certificates onto the running AWS EC2 instances during the first connection. It would be even better if AWS allowed to use SSH CA certs as keys, but alas...
- pugz 2y ago
  FYI I love your work with Gimlet, etc. I too would love "native" support for SSH CAs in EC2. What I ended up doing is adding a line to every EC2 userdata script that would rewrite the /home/ec2-user/.ssh/authorized_keys file to treat the provided EC2 keypair as a CA instead of a regular pubkey.
dangsux 2y ago
[dead]
arianvanp 2y ago
Zero trust. But they don't solve the more interesting problem: host key authentication. Would be nice if they can replace TOFU access with SSH CA as well. Ideally based on device posture of the server (e.g. TPM2 attestation)
- mdaniel 2y ago
  While not applicable for all situations <https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record> and its <https://man.openbsd.org/ssh_config.5#VerifyHostKeyDNS https://man.openbsd.org/ssh_config.5#VerifyHostKeyDNS> friend may interest you
  - udev4096 2y ago
    I was not aware of using random[0] ASCII art as a way to check if the host key has changed [0] - https://man.openbsd.org/ssh.1#random https://man.openbsd.org/ssh.1#random
- anilakar 2y ago
  As far as I know you can CA sign host keys the same way you can sign users' public keys. As always, the main issue is that certificate chaining is not possible in SSH PK"I", so you need to have absolute trust in the machine that does the signing.
advael 2y ago
You know you can just do this with keyauth and a cron job, right?
- wmf 2y ago
  And Dropbox is a wrapper around rsync.
  - advael 2y ago
    Generally speaking a lot of "essential tools" in "cloud computing" are available as free, boring operating system utilities.
    - kkielhofner 2y ago
      It’s a joke from a famous moment in HN history: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
      - advael 2y ago
        That is pretty funny, and the whole idea that you can't make money packaging open-source software in a way that's more appealing to people is definitely funny given that this is the business model of a lot of successful companies I do however think this leads to a lot of problems when those companies try to protect their business models, as we are seeing a lot of today
curben 2y ago
Cloudflare has been offering SSH CA-based authentication for more than 2 years [1], I wrote a guide back in feb '23 [2]. The announcement is more about offering new features, such as more granular user control. [1]:https://web.archive.org/web/20210418143636/https://developers.cloudflare.com/cloudflare-one/identity/users/short-lived-certificates https://web.archive.org/web/20210418143636/https://developer... [2]: https://mdleom.com/blog/2023/02/13/ssh-certificate-cloudflare-tunnel/ https://mdleom.com/blog/2023/02/13/ssh-certificate-cloudflar...
keepamovin 2y ago
Does this give CloudFlare a backdoor to all your servers? That would not strictly be ZT, as some identify in the comments here.
- knallfrosch 2y ago
  Everything rests on CloudFlare's key.
- ChoHag 2y ago
  [dead]
- udev4096 2y ago
  For cloudflare, all their fancy ZT excludes themselves. It's just like the well known MiTM they perform while using their CA
  - megous 2y ago
    Sounds like their modus operandi for most of their products, incl. the original one.
    - keepamovin 2y ago
      And if China hacks CloudFlare? I guess we're all fucked.
anilakar 2y ago
Every now and then a new SSH key management solution emerges and every time it is yet another connection-terminating proxy and not a real PKI solution.
INTPenis 2y ago
Properly setup IaC, that treats Linux as an appliance instead, could get rid of SSH altogether. I'm only saying this because after 20+ years as a sysadmin I feel like there have been no decent solutions presented. On the other hand, to protect my IaC and Gitops I have seen very decent and mature solutions.
- otabdeveloper4 2y ago
  I don't know what exactly you mean by "IaC" here, but the ones I know use SSH under the hood somewhere. (Except with some sort of "bot admin" key now, which is strictly worse.)
  - INTPenis 2y ago
    I mean that you treat Linux servers as appliances, you do everything in IaC at provisioning and you never login over SSH.
    - otabdeveloper4 2y ago
      "IaC at provisioning" means (in practice) a webapp and an eternal root access token that does login over SSH for you behind the scenes. That's sctrictly worse from a security point of view. In an ideal world we would have private CAs and short-lived certificates that get bubbled through all the layers of the software stack. Going back to webapps and tokens is a step backwards.
      - INTPenis 2y ago
        That's a bad practice. I have better security experience from the infrastructure around IaC than SSH. Because for IaC we used Gitlab, hidden by a Keycloak, or connected to an Azure AD, protected by a MFA VPN. And for provisioning we used containers, no SSH required there either. The major revolution that allowed me to move away from SSH in server provisioning is container hosts, ignition (or cloud-init), and these days the cutting edge is bootc.
rdtsc 2y ago
By “ValidPrinciples” did they mean “ValidPrincipals”? And by ZeroTrust they really mean OneTrust: trust CF. A classic off-by-one error :-)
- deleted 2y ago
  [deleted]
deleted 2y ago
[deleted]
amar0c 2y ago
Is there anything similar ("central point of SSH access/keys management" ) that is not Cloudflare ? I know about Tailscale and it's SSH but recently it introduced so much latency (even tho they say it's P2P between A and B) that is unusable. Ideally something self hosted but not hard requirement
- udev4096 2y ago
  Like a ssh key management cli?
TechnicalVault 2y ago
The whole MITM just makes me deeply uncomfortable, it's introducing a single point of trust with the keys to the kingdom. If I want to log what someone is doing, I do it server side e.g. some kind of rsyslog. That way I can leverage existing log anomaly detection systems to pick up and isolate the server if we detect any bad behaviour.
- naikrovek 2y ago
  yeah the MITM thing is ... concerning. this just moves the trusted component from the SSH key to Cloudflare, and you still must trust something implicitly. except now it's a company that has agency and a will of its own instead of just some files on a filesystem. I'll stick to forced key rotation, thanks.
  - EthanHeilman 2y ago
    > you still must trust something implicitly. except now it's a company that has agency and a will of its own instead of just some files on a filesystem. Some keys on a file system on a large number of user endhosts is a security nightmare. At big companies user endhosts are compromised hourly. When you say forced key rotation how do you accomplish that and how often do you rotate? What if you want to disallow access to a user on a faster tempo then your rotation period? How do you ensure that you are giving out the new keys to only authorized people? My experience has been, when you really invest in building a highly secure key rotation system, you end up building something similar to our system. 1. You want SSO integration with policy to ensure only the right people get the right keys to ensure the right keys end up on the right hosts. This is a hard problem. 2. You end up using a SSH CA with short lived certificates because "key expires after 3 minutes" is far more secure than "key rotated every 90 days". 3. Compliance requirements typically require session recording and logging, do you end up creating a MITM SSH Proxy to do this? Building all this stuff is expensive and it needs to be kept up to date. Instead of building it in-house and hoping you build it right, buy a zero trust SSH product. For many companies the alternative isn't key rotation it just an endless growing set of keys that never expire. To quote Tatu Ylonen the inventor of SSH: > "In analyzing SSH keys for dozens of large enterprises, it has turned out that in many environments 90% of all authorized keys are no longer used. They represent access that was provisioned, but never terminated when the person left or the need for access ceased to exist. Some of the authorized keys are 10-20 years old, and typically about 10% of them grant root access or other privileged access. The vast majority of private user keys found in most enviroments do not have passphrases." Challenges in Managing SSH Keys – and a Call for Solutions https://ylonen.org/papers/ssh-key-challenges.pdf https://ylonen.org/papers/ssh-key-challenges.pdf
udev4096 2y ago
> You no longer need to manage long-lived SSH keys Well, now you are managing CAs. Sure, it's short lived but it's not different than having a policy for rotating your SSH keys
- acdha 2y ago
  It’s really important to understand why those are different. CAs are organizational and tightly restricted: I don’t use or have access to my CA’s private key but my SSH key is on every client I use. If I leave the company, you have to check every authorized key file on every server to ensure my keys are no longer present. In contrast, the CA doesn’t need to rotate since I never had access to it and since the CA will set an expiration time on each of the keys I do get it’s probably unusable shortly after my departure even if you missed something.
nanis 2y ago
> the SSH certificates issued by the Cloudflare CA include a field called ValidPrinciples Having implemented similar systems before, I was interested to read this post. Then I see this. Now I have to find out if that really is the field, if this was ChatGPT spellcheck, or something else entirely.
- blueflow 2y ago
  For the others: The correct naming is "principals".
  - jgrahamc 2y ago
    Sigh. I'll get that fixed and figure out how that happened.
    - nanis 2y ago
      This was corrected to: > ... SSH certificates issued by the Cloudflare CA include a field called valid_principals which indicates it wasn't just the spelling of `principals`.
      - jgrahamc 2y ago
        It depends... ssh-keygen -L displays the fields as Principals (which are set using the -n parameter) and internally a lot of the OpenSSH code talks about AuthorizedPrincipals...
blueflow 2y ago
Instead of stealing your password/keypair, the baddies will now have to spoof your authentication with cloudflare. If thats just a password, you gained nothing. If you have 2FA set up for that, you could equally use that for SSH directly, using a ssh key on a physical FIDO stick. OpenSSH already has native support for that (ecdsa-sk and ed25519-sk key formats). The gain here is minimal.
c-linkage 2y ago
Welcome to Kerberos[0] over HTTP. [0] https://www.geeksforgeeks.org/kerberos/ https://www.geeksforgeeks.org/kerberos/
nonameiguess 2y ago
Basic summary seems to be: * This has nothing to do with zero-trust. If you already require pubkey auth to every connection made to a server regardless of origin, that's already meeting the definition of zero trust. * What this actually gives you is a solution to the problem of centrally revoking long-lived keys by not having any and instead using certificate auth. Now the CA is the only long-lived key. * This is a reasonable thing large orgs should probably do. There is no reason the CA should be an external third-party like Cloudflare, however. * This also integrates with existing SSO providers so human users can be granted short-lived session certs based on whatever you use to authenticate them to the SSO provider. Also reasonable, also no reason this should be offered as a service from Cloudflare as opposed to something you can self-host like Kerberos. * This also provides ssh command logging by proxying the session and capturing all commands as they get relayed. Arguably not a bad idea in principle, but a log collector like rsyslogd sending to an aggregator accomplishes the same thing in practice, and again, I would think you'd want to self-host a proxy if you choose to go that route, not rent it from Cloudflare. All in all, good things a lot of orgs should do, but they should probably actually do. I get the "well, it's hard" angle, but you're usually looking at large, well-funded orgs when you're talking things like SOC and FedRamp compliance. If you want to be a bank or whatever, yeah, that's hard. It's supposed to be. As I understand it, at least part of the spirit of SOC and FedRamp and the like is your organization has processes, plans, procedures, and personnel in place with the expertise and care to take security seriously, not "we have no idea what any of this means, why it matters, and don't have the time, but we pay a subscription fee to Cloudflare and they say they take care of it."
singhrac 2y ago
I get that HN does not like Cloudflare and does not like the term “Zero Trust”, but geez these comments are repetitive. Can anyone compare to Tailscale SSH? Are they basically offering an (even more) enterprise version of Tailscale’s product line?
andriosr 2y ago
hoopdev here. Zero trust for SSH is just table stakes these days. Real challenge is getting devs to actually adopt better practices without the tooling getting in their way. Found in practice that certs > keys but you need to think beyond just SSH. Most teams have a mix of SSH, K8s, DBs etc. Using separate tools for each just creates more headache. Haven't tried Boundary but Teleport/hoop/Tailscale all handle the mixed protocol issue decently. Main difference is hoop focuses more on protocol-level DLP and automated reviews vs pure network access. Horses for courses though, they're all valid approaches. Key is picking something devs will actually use vs work around. Nothing worse than a "secure" solution that drives people to create workarounds.
dmuth 2y ago
Using CAs and signed certificates in SSH is definitely the way. If anyone wants to play around with that, without the risk of locking themselves out of a server, I built a little "playground" awhile back whihc is a series of Docker containers that can SSH to each other. Give it a try at https://github.com/dmuth/ssh-principal-and-ca-playground https://github.com/dmuth/ssh-principal-and-ca-playground (I haven't touched the project in awhile, so if there are any issues, please open an Issue and I'll gladly look at it!)
WesolyKubeczek 2y ago
I can’t wait for a bug to happen when you authenticate correctly but unexpectedly slide into someone else’s network.

151 points by mfrw 2y ago

How is trusting Cloudflare "zero-trust" ?

I'm a member of the team that worked on this happy to answer any questions. We (BastionZero) recently got bought by Cloudflare and it is exciting bringing our SSH ideas to Cloudflare.

mdaniel 2y ago
I just wanted to offer my congratulations on the acquisition. I don't know any details about your specific one, but I have been around enough to know that it's still worth celebrating o/
lenova 2y ago
I'd love to hear about the acquisition story with Cloudflare.
- FlyingSnake 2y ago
  You can read the details here: https://blog.cloudflare.com/cloudflare-acquires-bastionzero/ https://blog.cloudflare.com/cloudflare-acquires-bastionzero/
- EthanHeilman 2y ago
  Are particular questions? So far my experience with joining and working at Cloudflare has been fantastic. Coming from a background of startups and academia, the size and scope of what Cloudflare is building and currently runs is overwhelming. In academia I've seen lots of excellent academic computer science papers that never benefit anyone because they never get turned into a tool that someone can just pick up and use. Ideas have inherent value, even useless ideas, but it feels good to see great ideas have impact. What appealed to me the most about getting acquired by Cloudflare is seeing research applied directly to products and used by people. Cloudflare does an excellent job both inventing innovative ideas and then actually making them real. There used to be a lot of companies that did this 10 years ago, but Cloudflare now seems rare in that respect.

mdaniel 2y ago

I really enjoyed my time with Vault's ssh-ca (back when it had a sane license) but have now grown up and believe that any ssh access is an antipattern. For context, I'm also one of those "immutable OS or GTFO" chaps because in my experience the next thing that happens after some rando ssh-es into a machine is they launch vi or apt-get or whatever and now it's a snowflake with zero auditing of the actions taken to it I don't mean to detract from this, because short-lived creds are always better, but for my money I hope I never have sshd running on any machine again

namxam 2y ago
But what is the alternative?
- mdaniel 2y ago
  There's not one answer to your question, but here's mine: kubelet and AWS SSM (which, to the best of my knowledge will work on non-AWS infra it just needs to be provided creds). Bottlerocket <https://github.com/bottlerocket-os/bottlerocket#setup https://github.com/bottlerocket-os/bottlerocket#setup> comes batteries included with both of those things, and is cheaply provisioned with (ahem) TOML user-data <https://github.com/bottlerocket-os/bottlerocket#description-of-settings https://github.com/bottlerocket-os/bottlerocket#description-...> In that specific case, one can also have "systemd for normal people" via its support for static Pod definitions, so one can run containerized toys on boot even without being a formal member of a kubernetes cluster AWS SSM provides auditing of what a person might normally type via ssh, and kubelet similarly, just at a different abstraction level. For clarity, I am aware that it's possible via some sshd trickery one could get similar audit and log egress, but I haven't seen one of those in practice whereas kubelet and AWS SSM provide it out of the box
  - cyberax 2y ago
    Be careful with SSM, it can provide pretty much unlimited access: https://github.com/Cyberax/gimlet https://github.com/Cyberax/gimlet You can use it to tunnel arbitrary traffic inside your VPC.

antoniomika 2y ago

I wrote a system that did this >5 years ago (luckily was able to open source it before the startup went under[0]). The bastion would record ssh sessions in asciicast v2 format and store those for later playback directly from a control panel. The main issue that still isn't solved by a solution like this is user management on the remote (ssh server) side. In a more recent implementation, integration with LDAP made the most sense and allows for separation of user and login credentials. A single integrated solution is likely the holy grail in this space. [0] https://github.com/notion/bastion https://github.com/notion/bastion

mdaniel 2y ago
Out of curiosity, why ignore this PR? https://github.com/notion/bastion/pull/13 https://github.com/notion/bastion/pull/13 I would think even a simple "sorry, this change does not align with the project's goals" -> closed would help the submitter (and others) have some clarity versus the PR limbo it's currently in That aside, thanks so much for pointing this out: it looks like good fun, especially the Asciicast support!
- antoniomika 2y ago
  Honestly never had a chance to merge it/review it. Once the company wound down, I had to move onto other things (find a new job, work on other priorities, etc) and lost access to be able to do anything with it after. I thought about forking it and modernizing it but never came to fruition.

edelbitter 2y ago

Why does the title say "Zero Trust", when the article explains that this only works as long as every involved component of the Cloudflare MitM keylogger and its CA can be trusted? If hosts keys are worthless because you do not know in advance what key the proxy will have.. than this scheme is back to trusting servers merely because they are in Cloudflare address space, no?

deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
varenc 2y ago
https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/ https://www.cloudflare.com/learning/security/glossary/what-i... Zero Trust just means you stop inherently trusting your private network and verify every user/device/request regardless. If you opt in to using Cloudflare to do this then it requires running Cloudflare software.
- bdd8f1df777b 2y ago
  But with public key auth I'm already distrusting everyone on my private network.
  - resoluteteeth 2y ago
    Technically I guess that's "zero trust" in the sense of meeting the requirement of not trusting internal connections more than external ones, but in practice I guess "zero trust" also typically entails making every connection go through the same user-based authentication system, which uploading specific keys to specific servers manually definitely doesn't achieve.
- PLG88 2y ago
  Thats one interpretation... ZT also posits assuming the network is compromised and hostile, that also applies to CF and their cloud/network. It blows my mind that so many solutions claim ZT while mandating TLS to their infra/cloud, you can trust their decryption of your date, and worst IMHO, they will MITM your OICD/SAML key to ensure the endpoint can authenticate and access services... that is a hell of a lot of implicit trust in them, not least of them being served a court order to decrypt your data. Zero trust done correctly done not have those same drawbacks.

johnklos 2y ago

So... don't trust long lived ssh keys, but trust Cloudflare's CA. Why? What has Cloudflare done to earn trust? If that alone weren't reason enough to dismiss this, the article has marketing BS throughout. For instance, "SSH access to a server often comes with elevated privileges". Ummm... Every authentication system ever has whatever privileges that come with that authentication system. This is the kind of bull you say / write when you want to snow someone who doesn't know any better. To those of us who do understand this, this is almost AI level bullshit. The same is true of their supposed selling points: > Author fine-grained policy to govern who can SSH to your servers and through which SSH user(s) they can log in as. That's exactly what ssh does. You set up precisely which authentication methods you accept, you set up keys for exactly that purpose, and you set up individual accounts. Do Cloudflare really think we're setting up a single user account and giving access to lots of different people, and we need them to save us? (now that I think about it, I bet some people do this, but this is still a ridiculous selling point) > Monitor infrastructure access with Access and SSH command logs So they're MITM all of our connections? We're supposed to trust them, even though they have a long history of not only working with scammers and malicious actors, but protecting them? I suppose there's a sucker born every minute, so Cloudflare will undoubtedly sell some people on this silliness, but to me it just looks like yet another way that Cloudflare wants to recentralize the Internet around them. If they had their way, then in a few years, were they to go down, a majority of the Internet would literally stop working. That should scare everyone.

deleted 2y ago
[deleted]

tptacek 2y ago

I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA. When I'm selling people on SSH CA's, the first thing I usually have to convince them of is that I'm not saying they should trust some third party. You know where all your servers are. External CAs exist to solve the counterparty introduction problem, which is a problem SSH servers do not have.

xyst 2y ago
Same reasons for companies still buying “CrowdStrike” and installing that crapware. It’s all for regulatory checkboxes (ie, fedramp cert).
- tptacek 2y ago
  I do not believe you in fact need any kind of SSH CA, let alone one run by a third party, to be FedRAMP-compliant.
kevin_nisbet 2y ago
I'm with you, I imagine it's mostly people just drawing parallels, they can figure out how to get a web certificate so think SSH is the same thing. The second order problem I've found is when you dig in there are plenty of people who ask for certs but when push comes to shove really want functionality where when user access is cancelled all active sessions get torn down immediatly as well.
michaelt 2y ago
> I'm a fan of SSH certificates and cannot understand why anyone would set up certificate authentication with an external third-party CA. I think the sales pitch for these sorts of service is: "Get an SSH-like experience, but it integrates with your corporate single-sign-on system, has activity logs that can't be deleted even if you're root on the target, sorts out your every-ephemeral-cloud-instance-has-a-different-fingerprint issues, and we'll sort out all the reverse-tunnelling-through-NAT and bastion-host-for-virtual-private-cloud stuff too" Big businesses pursuing SOC2 compliance love this sort of thing.

xyst 2y ago

Underlying tech is “Openpubkey”. https://github.com/openpubkey/openpubkey https://github.com/openpubkey/openpubkey BastionZero just builds on top of that to provide a “seamless” UX for ssh sessions and some auditing/fedramp certification. Personally, not a fan of relying on CF. Need less centralization/consolidation into a few companies. It’s bad enough with MS dominating the OS (consumer) space. AWS dominating cloud computing. And CF filling the gaps between the stack.

ranger_danger 2y ago
Completely agree. I also don't want to trust certificate authorities for my SSH connections let alone CF. Would not be surprised if it/they were compromised.
- yjftsjthsd-h 2y ago
  > I also don't want to trust certificate authorities for my SSH connections let alone CF. Would not be surprised if it/they were compromised. OpenPubkey, or in general? Normal SSH CAs don't do PKI like browsers use, you make and trust your own CA(s). And if an attacker can compromise your CA private key, why can't they compromise your SSH private key directly?
  - looofooo0 2y ago
    https://www.usenix.org/system/files/login/articles/105484-Gutmann.pdf https://www.usenix.org/system/files/login/articles/105484-Gu... + People just don't check ssh keys normally.
    - yjftsjthsd-h 2y ago
      That's about host keys, not user keys. And... I'm struggling to think of a threat model where that problem manifests in a compromise? Like, what's your threat model? That said, CAs actually really help with that problem, because if a server has its host keys signed with a CA and then the user trusts that CA then they don't have to TOFU the host keys.

shermantanktop 2y ago

I didn’t understand the marketing term “zero trust” and I still don’t. In practice, I get it - a network zone shouldn’t require a lower authn/z bar on the implicit assumption that admission to that zone must have required a higher bar. But all these systems are built on trust, and if it isn’t based on network zoning, it’s based on something else. Maybe that other thing is better, maybe not. But it exists and it needs to be understood. An actual zero trust system is the proverbial unpowered computer in a bunker.

ngneer 2y ago
With you there. The marketing term makes Zero Sense to me.
wmf 2y ago
The something else is specifically user/service identity. Not machine identity, not IP address. It is somewhat silly to have a buzzword that means "no, actually authenticate users" but here we are.
athorax 2y ago
It means there is zero trust of a device/service/user on your network until they have been fully authenticated. It is about having zero trust in something just because it is inside your network perimeter.
- shermantanktop 2y ago
  Maybe it should be called "zero trust of a device/service/user on your network until they have been fully authenticated." But that wouldn't sell high-dollar consulting services.
acdha 2y ago
Yeah, it’s not a great name. Twenty years ago we called it “end to end authentication” and I think that’s better because it focuses on the most important aspect, but it probably doesn’t sound as cool for marketing purposes. I also like how that makes it easier to understand how variation is normal: for example, authentication comes in various flavors and that’s okay whereas some of that zero trust vendors will try to claim that something is or isn’t ZT based on feature gaps in their competitors’ and it’s just so tedious to play that game.

cyberax 2y ago

Hah. I did pretty much all the same stuff in my previous company. One thing that we did a bit better: we used AWS SSM to provision our SSH-CA certificates onto the running AWS EC2 instances during the first connection. It would be even better if AWS allowed to use SSH CA certs as keys, but alas...

pugz 2y ago
FYI I love your work with Gimlet, etc. I too would love "native" support for SSH CAs in EC2. What I ended up doing is adding a line to every EC2 userdata script that would rewrite the /home/ec2-user/.ssh/authorized_keys file to treat the provided EC2 keypair as a CA instead of a regular pubkey.

dangsux 2y ago

[dead]

arianvanp 2y ago

Zero trust. But they don't solve the more interesting problem: host key authentication. Would be nice if they can replace TOFU access with SSH CA as well. Ideally based on device posture of the server (e.g. TPM2 attestation)

mdaniel 2y ago
While not applicable for all situations <https://en.wikipedia.org/wiki/SSHFP_record https://en.wikipedia.org/wiki/SSHFP_record> and its <https://man.openbsd.org/ssh_config.5#VerifyHostKeyDNS https://man.openbsd.org/ssh_config.5#VerifyHostKeyDNS> friend may interest you
- udev4096 2y ago
  I was not aware of using random[0] ASCII art as a way to check if the host key has changed [0] - https://man.openbsd.org/ssh.1#random https://man.openbsd.org/ssh.1#random
anilakar 2y ago
As far as I know you can CA sign host keys the same way you can sign users' public keys. As always, the main issue is that certificate chaining is not possible in SSH PK"I", so you need to have absolute trust in the machine that does the signing.

advael 2y ago

You know you can just do this with keyauth and a cron job, right?

wmf 2y ago
And Dropbox is a wrapper around rsync.
- advael 2y ago
  Generally speaking a lot of "essential tools" in "cloud computing" are available as free, boring operating system utilities.
  - kkielhofner 2y ago
    It’s a joke from a famous moment in HN history: https://news.ycombinator.com/item?id=9224 https://news.ycombinator.com/item?id=9224
    - advael 2y ago
      That is pretty funny, and the whole idea that you can't make money packaging open-source software in a way that's more appealing to people is definitely funny given that this is the business model of a lot of successful companies I do however think this leads to a lot of problems when those companies try to protect their business models, as we are seeing a lot of today

curben 2y ago

Cloudflare has been offering SSH CA-based authentication for more than 2 years [1], I wrote a guide back in feb '23 [2]. The announcement is more about offering new features, such as more granular user control. [1]:https://web.archive.org/web/20210418143636/https://developers.cloudflare.com/cloudflare-one/identity/users/short-lived-certificates https://web.archive.org/web/20210418143636/https://developer... [2]: https://mdleom.com/blog/2023/02/13/ssh-certificate-cloudflare-tunnel/ https://mdleom.com/blog/2023/02/13/ssh-certificate-cloudflar...

keepamovin 2y ago

Does this give CloudFlare a backdoor to all your servers? That would not strictly be ZT, as some identify in the comments here.

knallfrosch 2y ago
Everything rests on CloudFlare's key.
ChoHag 2y ago
[dead]
udev4096 2y ago
For cloudflare, all their fancy ZT excludes themselves. It's just like the well known MiTM they perform while using their CA
- megous 2y ago
  Sounds like their modus operandi for most of their products, incl. the original one.
  - keepamovin 2y ago
    And if China hacks CloudFlare? I guess we're all fucked.

anilakar 2y ago

Every now and then a new SSH key management solution emerges and every time it is yet another connection-terminating proxy and not a real PKI solution.

INTPenis 2y ago

Properly setup IaC, that treats Linux as an appliance instead, could get rid of SSH altogether. I'm only saying this because after 20+ years as a sysadmin I feel like there have been no decent solutions presented. On the other hand, to protect my IaC and Gitops I have seen very decent and mature solutions.

otabdeveloper4 2y ago
I don't know what exactly you mean by "IaC" here, but the ones I know use SSH under the hood somewhere. (Except with some sort of "bot admin" key now, which is strictly worse.)
- INTPenis 2y ago
  I mean that you treat Linux servers as appliances, you do everything in IaC at provisioning and you never login over SSH.
  - otabdeveloper4 2y ago
    "IaC at provisioning" means (in practice) a webapp and an eternal root access token that does login over SSH for you behind the scenes. That's sctrictly worse from a security point of view. In an ideal world we would have private CAs and short-lived certificates that get bubbled through all the layers of the software stack. Going back to webapps and tokens is a step backwards.
    - INTPenis 2y ago
      That's a bad practice. I have better security experience from the infrastructure around IaC than SSH. Because for IaC we used Gitlab, hidden by a Keycloak, or connected to an Azure AD, protected by a MFA VPN. And for provisioning we used containers, no SSH required there either. The major revolution that allowed me to move away from SSH in server provisioning is container hosts, ignition (or cloud-init), and these days the cutting edge is bootc.

rdtsc 2y ago

By “ValidPrinciples” did they mean “ValidPrincipals”? And by ZeroTrust they really mean OneTrust: trust CF. A classic off-by-one error :-)

deleted 2y ago
[deleted]

deleted 2y ago

[deleted]

amar0c 2y ago

Is there anything similar ("central point of SSH access/keys management" ) that is not Cloudflare ? I know about Tailscale and it's SSH but recently it introduced so much latency (even tho they say it's P2P between A and B) that is unusable. Ideally something self hosted but not hard requirement

udev4096 2y ago
Like a ssh key management cli?

TechnicalVault 2y ago

The whole MITM just makes me deeply uncomfortable, it's introducing a single point of trust with the keys to the kingdom. If I want to log what someone is doing, I do it server side e.g. some kind of rsyslog. That way I can leverage existing log anomaly detection systems to pick up and isolate the server if we detect any bad behaviour.

naikrovek 2y ago
yeah the MITM thing is ... concerning. this just moves the trusted component from the SSH key to Cloudflare, and you still must trust something implicitly. except now it's a company that has agency and a will of its own instead of just some files on a filesystem. I'll stick to forced key rotation, thanks.
- EthanHeilman 2y ago
  > you still must trust something implicitly. except now it's a company that has agency and a will of its own instead of just some files on a filesystem. Some keys on a file system on a large number of user endhosts is a security nightmare. At big companies user endhosts are compromised hourly. When you say forced key rotation how do you accomplish that and how often do you rotate? What if you want to disallow access to a user on a faster tempo then your rotation period? How do you ensure that you are giving out the new keys to only authorized people? My experience has been, when you really invest in building a highly secure key rotation system, you end up building something similar to our system. 1. You want SSO integration with policy to ensure only the right people get the right keys to ensure the right keys end up on the right hosts. This is a hard problem. 2. You end up using a SSH CA with short lived certificates because "key expires after 3 minutes" is far more secure than "key rotated every 90 days". 3. Compliance requirements typically require session recording and logging, do you end up creating a MITM SSH Proxy to do this? Building all this stuff is expensive and it needs to be kept up to date. Instead of building it in-house and hoping you build it right, buy a zero trust SSH product. For many companies the alternative isn't key rotation it just an endless growing set of keys that never expire. To quote Tatu Ylonen the inventor of SSH: > "In analyzing SSH keys for dozens of large enterprises, it has turned out that in many environments 90% of all authorized keys are no longer used. They represent access that was provisioned, but never terminated when the person left or the need for access ceased to exist. Some of the authorized keys are 10-20 years old, and typically about 10% of them grant root access or other privileged access. The vast majority of private user keys found in most enviroments do not have passphrases." Challenges in Managing SSH Keys – and a Call for Solutions https://ylonen.org/papers/ssh-key-challenges.pdf https://ylonen.org/papers/ssh-key-challenges.pdf

udev4096 2y ago

> You no longer need to manage long-lived SSH keys Well, now you are managing CAs. Sure, it's short lived but it's not different than having a policy for rotating your SSH keys

acdha 2y ago
It’s really important to understand why those are different. CAs are organizational and tightly restricted: I don’t use or have access to my CA’s private key but my SSH key is on every client I use. If I leave the company, you have to check every authorized key file on every server to ensure my keys are no longer present. In contrast, the CA doesn’t need to rotate since I never had access to it and since the CA will set an expiration time on each of the keys I do get it’s probably unusable shortly after my departure even if you missed something.

nanis 2y ago

> the SSH certificates issued by the Cloudflare CA include a field called ValidPrinciples Having implemented similar systems before, I was interested to read this post. Then I see this. Now I have to find out if that really is the field, if this was ChatGPT spellcheck, or something else entirely.

blueflow 2y ago
For the others: The correct naming is "principals".
- jgrahamc 2y ago
  Sigh. I'll get that fixed and figure out how that happened.
  - nanis 2y ago
    This was corrected to: > ... SSH certificates issued by the Cloudflare CA include a field called valid_principals which indicates it wasn't just the spelling of `principals`.
    - jgrahamc 2y ago
      It depends... ssh-keygen -L displays the fields as Principals (which are set using the -n parameter) and internally a lot of the OpenSSH code talks about AuthorizedPrincipals...

blueflow 2y ago

Instead of stealing your password/keypair, the baddies will now have to spoof your authentication with cloudflare. If thats just a password, you gained nothing. If you have 2FA set up for that, you could equally use that for SSH directly, using a ssh key on a physical FIDO stick. OpenSSH already has native support for that (ecdsa-sk and ed25519-sk key formats). The gain here is minimal.

c-linkage 2y ago

Welcome to Kerberos[0] over HTTP. [0] https://www.geeksforgeeks.org/kerberos/ https://www.geeksforgeeks.org/kerberos/

nonameiguess 2y ago

Basic summary seems to be: * This has nothing to do with zero-trust. If you already require pubkey auth to every connection made to a server regardless of origin, that's already meeting the definition of zero trust. * What this actually gives you is a solution to the problem of centrally revoking long-lived keys by not having any and instead using certificate auth. Now the CA is the only long-lived key. * This is a reasonable thing large orgs should probably do. There is no reason the CA should be an external third-party like Cloudflare, however. * This also integrates with existing SSO providers so human users can be granted short-lived session certs based on whatever you use to authenticate them to the SSO provider. Also reasonable, also no reason this should be offered as a service from Cloudflare as opposed to something you can self-host like Kerberos. * This also provides ssh command logging by proxying the session and capturing all commands as they get relayed. Arguably not a bad idea in principle, but a log collector like rsyslogd sending to an aggregator accomplishes the same thing in practice, and again, I would think you'd want to self-host a proxy if you choose to go that route, not rent it from Cloudflare. All in all, good things a lot of orgs should do, but they should probably actually do. I get the "well, it's hard" angle, but you're usually looking at large, well-funded orgs when you're talking things like SOC and FedRamp compliance. If you want to be a bank or whatever, yeah, that's hard. It's supposed to be. As I understand it, at least part of the spirit of SOC and FedRamp and the like is your organization has processes, plans, procedures, and personnel in place with the expertise and care to take security seriously, not "we have no idea what any of this means, why it matters, and don't have the time, but we pay a subscription fee to Cloudflare and they say they take care of it."

singhrac 2y ago

I get that HN does not like Cloudflare and does not like the term “Zero Trust”, but geez these comments are repetitive. Can anyone compare to Tailscale SSH? Are they basically offering an (even more) enterprise version of Tailscale’s product line?

andriosr 2y ago

hoopdev here. Zero trust for SSH is just table stakes these days. Real challenge is getting devs to actually adopt better practices without the tooling getting in their way. Found in practice that certs > keys but you need to think beyond just SSH. Most teams have a mix of SSH, K8s, DBs etc. Using separate tools for each just creates more headache. Haven't tried Boundary but Teleport/hoop/Tailscale all handle the mixed protocol issue decently. Main difference is hoop focuses more on protocol-level DLP and automated reviews vs pure network access. Horses for courses though, they're all valid approaches. Key is picking something devs will actually use vs work around. Nothing worse than a "secure" solution that drives people to create workarounds.

dmuth 2y ago

Using CAs and signed certificates in SSH is definitely the way. If anyone wants to play around with that, without the risk of locking themselves out of a server, I built a little "playground" awhile back whihc is a series of Docker containers that can SSH to each other. Give it a try at https://github.com/dmuth/ssh-principal-and-ca-playground https://github.com/dmuth/ssh-principal-and-ca-playground (I haven't touched the project in awhile, so if there are any issues, please open an Issue and I'll gladly look at it!)

WesolyKubeczek 2y ago

I can’t wait for a bug to happen when you authenticate correctly but unexpectedly slide into someone else’s network.