29 ms·
Remote code execution in OpenSSH’s forwarded SSH-agent
- WhackyIdeas 3y agoAm I reading this right - any system with ssh-agent installed by default is vulnerable? The link is short on specifics.
- kam 3y agoIt's exploitable only by a SSH server that you connect to with agent forwarding enabled (i.e. one that you're already trusting with access to your SSH keys).
- jonas-w 3y agoIf I read this correctly it is only when you forward your ssh-agent with the "-A" or "ForwardAgent" option
- tetha 3y agoAs far as I read it, it's about forwarded ssh agents. Basically, if you `ssh -A user@system`, something might be able to execute commands locally. For example, this might turn messy for infrastructures using jump-hosts extensively, if people are used to <ssh -A jumphost> so they can easily <ssh system> afterwards. If you pop the jump host, you could pivot to the workstations with this. At the same time, ssh-agent forwarding makes me queasy from a security perspective even without this. As far as I know, if you <ssh -A> into a system, admins with privileges on the system can gain access to your local ssh-agent already. In the example of the jump host, if you popped the jump host and stuck around for a while, you could probably harvest SSH keys and have some fun later.
- spladug 3y agoFWIW, `ProxyJump` is a safer way to go through a bastion without having to expose your agent to either the bastion or the target host behind it.
- chaosite 3y agoAnd since the person you're replying to was mentioning command-line parameters, it's worth mentioning that this can be done with `ssh -J jumphost user@system`.
- tetha 3y agoUseful, I didn't know that so far. We don't use bastion servers. My only real use case for ssh agent forwarding is if I need some scp / rsync between two remote systems during emergencies and those systems have no trust via SSH keys setup between them. In that very specific case, I don't know a better way than <ssh -A> to the first system and have some <rsync -e ssh> from there to the second system. Still doesn't feel great, even though I know only the people who could steal my keys are on my team.
- spladug 3y agoAh yeah. Not sure on that one. scp does have the `-3` option to copy between two remote hosts via the local host, but that can be significantly slower if the remote hosts are in the same network and local host is not.
- tetha 3y agoExactly. If I need to move a few megabytes around, <scp -3> and a coffee or a few simple tickets is a good way. A year ago or so, I needed 600GB moved between two systems ASAP during an outage that'd turn into a money-bleed at 6am. If I piped that through the VPN and my workstation, I'd probably still be waiting today.
- LinuxBender 3y agoSome time take a look at lftp [1] and its mirror subsystem for this. It can break up a batch of files or even one large file into multiple SFTP streams. Another upside is that it can replicate most rsync behavior in a SFTP-Only Chroot account. Downside is that without a corresponding daemon like rsync on the other end directory enumeration is slow which isn't a problem if one does not have a complex directory structure. Play around with the built in rate limit options total and per thread to keep the network people happy. [1] - https://linux.die.net/man/1/lftp https://linux.die.net/man/1/lftp
- tedunangst 3y agoWell, any system that auto maps the stack executable when loading a library. There are details on the specific requirements in the advisory. https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh-forwarded-ssh-agent.txt https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh...
- more_corn 3y agoI’d you forward your agent to a remote host, that host can mess with you.
- a-dub 3y agowhy publish this before the patches are out?
- tedunangst 3y agoThe patches are out.
- a-dub 3y agowhere? i couldn't find them on the openssh webpage and my distro has not released a new openssh either. edit: nvm. i see the release now. just no release for openbsd itself it seems.
- tedunangst 3y agohttp://www.openssh.com/releasenotes.html#9.3p2 http://www.openssh.com/releasenotes.html#9.3p2
- davidcuddeback 3y ago> just no release for openbsd itself it seems. It was posted to announce@openbsd.org a few hours ago. Binary patches are available through syspatch and source patches on the errata page.
- lxgr 3y agoIt was arguably still not necessary to post the complete exploit chain/proof-of-concept on the same day, since the chaining of four different shared libraries present in Ubuntu probably does not immediately follow from the diff introduced by the fix.
- binkHN 3y agoFull details: https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh-forwarded-ssh-agent.txt https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh...
- aidenn0 3y agoEven without this announcement, friends don't let friends forward their ssh agent. It essentially grants that machine access to your private keys. A RCE vulnerability is strictly worse than key exposure, but you probably shouldn't have been using it anyways.
- starfallg 3y agoYup, and this is explained in detail in the ssh(1) man page. No-one should be using -A. Using a jump host via -J is the way to go.
- Arnavion 3y agoOne use of -A, namely using the ssh server as a jumphost, is covered by -J. The general use of -A, namely doing operations on the ssh server that require keys from the ssh client, is not. If I'm on machine foo and I want to connect to bar.example.org and clone a git repo there from baz.example.org, and baz.example.org requires an identity key that is in foo's ssh agent, then -A is the only option.
- GauntletWizard 3y agossh-askpass is a great tool in these situations. It intercedes and allows you to manually control all key usage attempts.
- devnullbrain 3y agoAh, a skillful execution of Cunningham's Law
- Arnavion 3y agoI'm not sure what relevance ssh-askpass has to what I wrote.
- GauntletWizard 3y ago
- apienx 3y agoAgent forwarding is discouraged by the OpenSSH crew. Yet, it's commonly used because of the convenience it affords. "Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection." https://man.openbsd.org/ssh.1 https://man.openbsd.org/ssh.1 Glad the vulnerability got fixed.
- berkle4455 3y agoWhat's a better solution if you want to be able to SSH across multiple machines? Do you need to always close the current connection to get back to localhost prior to a fresh SSH? e.g. how would I ssh into foo, and then later into bar, or perhaps pull some code from github onto foo that is authenticated by my key? localhost -> foo -> bar localhost -> foo -> github access It seems like the answer is either: a) ssh -A, or b) install my private key on foo.
- totallywrong 3y agoYou can set up an ssh tunnel on localhost to any port on bar or github via foo.
- cassianoleal 3y agoAs others have pointed out already, JumpHost or -J is the preferred way.
- deleted 3y ago[deleted]
- gunapologist99 3y agoI agree! The Github example complicates things, but the correct solution is actually another key: Generate a separate private key on foo and place that one in github, ideally per project as a deploy key.
- lxgr 3y agoPlease don’t call this the "correct" solution. It might work for you; it's not a general rule or widely accepted best practice. Generating a key per host is just a different security model than key forwarding, and arguably a worse one if key forwarding is done defensively (i.e. forward a separate key, with only those permissions you would give to a key present on the connected host itself).
- pritambaral 3y ago> Generate a separate private key on foo ... That is a TERRIBLE idea, if foo can be compromised. Even if you secure the private key with a passphrase, it still needs to be loaded into foo's memory, by a binary resident on foo, which can be used to exfiltrate that private key. If you're confident foo cannot be compromised, then the whole point is moot anyway.
- slt2021 3y agowhat is CVSS/severity of this? Critical? 9+?
- tedunangst 3y agoCVSS with rice: 10/10
- tptacek 3y agoCVSS is meaningless, literally a Ouija board that reflects the intuition of whoever's computing the score. A more reasonable way to look at severity is on a two dimensional scale of lo-hi severity and lo-hi situationality. This is a high-severity, moderately situational bug (it would be highly situational if it required user interaction, and not situational at all if it didn't require control over the remote SSH server).
- NoZebra120vClip 3y agoCVSS are bunk. https://portswigger.net/daily-swig/cvss-system-criticized-for-failure-to-address-real-world-impact https://portswigger.net/daily-swig/cvss-system-criticized-fo...
- shandor 3y agoI can’t parse the exact meaning of the ”victim system” and ”attacker controlled system” in the OpenSSH release page. Does this vulnerability allow the attacker to compromise the original system where the user starts the agent-forwarded connection? Or ”only” compromise machines forward from the jump host? http://www.openssh.com/releasenotes.html#9.3p2 http://www.openssh.com/releasenotes.html#9.3p2
- perlgeek 3y agoIf an attacker can compromise the jump host, they can compromise the system where the user starts the agent-forwarded connection.
- shandor 3y agoOuch, that’s horrible. Time to prompt everyone at $WORK to upgrade…
- Arnavion 3y agoTo be clear, agent-forwarding to a (potentially) malicious ssh server has always been a bad idea. Yes TFA's bug makes it a worse idea and it's absolutely worth it to patch it, but you should not be agent-forwarding to (potentially) malicious ssh servers in the first place.
- gunapologist99 3y agoThat's 100% correct. This is why no one should use agent forwarding with a jumpbox. Only -J. (see https://userify.com/docs/jumpbox https://userify.com/docs/jumpbox ) Another point is that no one should have any real access to the jumpbox and it should be as minimal and stripped-down as possible. It's literally your bastion host, so you've got to keep it as strong as possible.
- sullivanmatt 3y agoThis sounds way worse than it is. To be clear, the "remote" part of the code execution is that an attacker controlling your destination server can cause your client to run an attacker-controlled payload, if the client is forwarding their credentials (`ssh -A`). Most people don't tend to make connections to arbitrary SSH hosts, and certainly they don't do it while forwarding their credentials along. It's a neat attack, and I applaud the Qualys team on their find, but this is not any sort of emergency situation for 99.99% of systems.
- malux85 3y agoYeah if I’m reading the technical analysis right, your conditions that you mention have to be correct and also the attacker must have “poisoned” library files on the targets machine so they can dlopen them, is that right? Pretty unlikely
- sullivanmatt 3y agoThere must be a specific set of libs present on the victim (client), correct. Qualys claims that stock Ubuntu Desktop systems often have these libs, and that they haven't looked into whether other distros tend to. But yes, your point stands. Huge number of preconditions here to fulfill.
- Arnavion 3y agoThe libraries are on the client's machine, not the server's. And they're not "poisoned"; the default distro-provided libs already provide the remote execution capabiity (eclipse-titan, libkf5sonnetui5, libns3-3v5 and systemd-boot packages from Ubuntu 22.04).
- malux85 3y agoAhh I see I thought the attacker also had to have custom malicious libs deployed on the client machine I wasn't sure if standard ones would do, thanks for clarifying that
- e28eta 3y ago
- kneebonian 3y agoCan I just say I know it's really dumb, but I loved that they published the explanation as a simple txt file, instead of setting up some whizbang website for it, or embedding it in their company blog. https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh-forwarded-ssh-agent.txt https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh...
- py4 3y agoUsed to be common for hacking-related docs. Miss those http://phrack.org/issues/49/15.html#article http://phrack.org/issues/49/15.html#article
- tptacek 3y agoThis is the bug of the year. It's well established that if Alice forwards an SSH agent to Bob, Bob can use the SSH agent protocol to make Alice open DLLs, because there's an agent protocol command (SSH_AGENTC_ADD_SMARTCARD_KEY) that OpenSSH implements with dlopen: when you ask the agent to access a smart card, OpenSSH dlopen()'s the library corresponding to the `id` of the device. This is a Jann Horn bug from 2016, and OpenSSH fixed it by whitelisting DLLs to /usr/lib and directories like it. The Qualys bug builds on Horn's bug. When OpenSSH dlopen()'s the library, it then tries to look up a PKCS#11 entry point function, and, when it doesn't find it, it dlclose()'s the library and returns an error. The issue is that most of the libraries in system library paths were never intended to be opened maliciously, and so they do all sorts of stuff in their constructors and destructors (any function marked `__attribute__((constructor))` or `destructor` is called by dlopen and dlclose respectively). In particular, they register callbacks and signal handlers. Most of these libraries are never expected to dlclose at all, so they tend not to be great about cleaning up. Better still, if you randomly load oddball libraries into random programs, some of them crash, generating SIGBUS and SIGSEGV. So you've got a classic UAF situation here: (1) force Alice to load a library that registers a SIGBUS handler; it won't be a PKCS#11 handler so it'll get immediately dlclose()'d, but won't clean up the handler. (2) Load another library, which will take over the program text address the signal handler points to. (3) Finally, load a library that SIGBUS's. If you manage to get a controlled jump swapped into place in step (2), you win. If you're thinking "it's pretty unlikely you're going to be able to line up a controlled jump at exactly the address previously registered as a signal handler", you're right, but there's another quirk of dlclose() they take advantage of: there's an ELF flag, NODELETE, that instructs the linker not to unmap a library when it's unloaded, and a bunch of standard libraries set it, so you can use those libraries to groom the address space. Finally, because some runtimes require executable stacks, there are standard libraries with an ELF flag that instructs the process to make the stack executable. If you load one of these libraries, and you have a controlled jump, you can write shellcode into the stack like it's 1998. To figure out the right sequence of steps, they basically recapitulated the original ROP gadget research idea: they swept all the standard Ubuntu libraries with a fuzzer to find combinations of loads that produced controlled jumps (ie, that died trying to execute stack addresses). A working exploit loads a pattern of "smartcards" that looks like this (all in /usr/lib): syslinux/modules/efi64/gfxboot.c32 (execstack) pulse-15.0+dfsg1/modules/module-remap-sink.so (groom) x86_64-linux-gnu/libgnatcoll_postgres.so.1 (SIGBUS handler) pulse-15.0+dfsg1/modules/module-http-protocol-unix.so (groom) x86_64-linux-gnu/sane/libsane-hp.so.1.0.32 (groom) libreoffice/program/libindex_data.so (groom) x86_64-linux-gnu/gstreamer-1.0/libgstaudiorate.so (groom) libreoffice/program/libscriptframe.so (groom) x86_64-linux-gnu/libisccc-9.16.15-Ubuntu.so (groom) x86_64-linux-gnu/libxkbregistry.so.0.0.0 (groom) debug/.build-id/15/c0bee6bcb06fbf381d0e0e6c52f71e1d1bd694.debug (SIGBUS) The paper goes on to classify like 4 more patterns whereby you can get unexpected control transfers by dlopen() and immediately dlclosing() libraries. The kicker: we noticed that one shared library's constructor function (which can be invoked by a remote attacker via an ssh-agent forwarding) starts a server thread that listens on a TCP port, and we discovered a remotely exploitable vulnerability (a heap-based buffer overflow) in this server's implementation.
- gunapologist99 3y agoNote: you are not vulnerable if you're not doing agent forwarding (-A). Honestly, you probably should never do -A anyway; use -J (proxyjump) instead. https://www.man7.org/linux/man-pages/man1/ssh.1.html https://www.man7.org/linux/man-pages/man1/ssh.1.html https://userify.com/docs/jumpbox https://userify.com/docs/jumpbox
- pcthrowaway 3y agoThere are situations where `-J` won't work. For example, if I'm on a remote server and I want to clone or pull from a private repository, or push to any repository In fact, VS code expects you to forward an agent when developing on a remote instance with the remote-ssh extension. I believe it uses it for syncing repo changes primarily I agree though that you should consider it insecure to forward a connection to a shared host if it has other users who are equally privileged, or more privileged
- gunapologist99 3y ago> There are situations where `-J` won't work. That is true, but actually you should avoid -A in those situations also. > In fact, VS code expects you to forward an agent when developing on a remote instance with the remote-ssh extension. I believe it uses it for syncing repo changes primarily That's the case if you're ever trying to initiate an SSH connection via Git from a remote host. That's because the actual SSH connection isn't being triggered on your desktop or laptop, where -J would do it, but because it's being initiated right on the remote server itself. Let's call the remote server VSCODE (your desktop-in-the-cloud), your laptop/desktop LAPTOP, and the remote repo GITHUB. So, the correct answer there is actually counter-intuitive: Generate a private key on VSCODE (not LAPTOP) for that GITHUB repo. This key will be used only on that originating remote server (VSCODE), never anywhere else. In fact, that repo key will ideally be scoped as tightly as possible in Github/Gitlab/etc. As a general rule of thumb, generate at least one SSH key for each "location" where you'll be logging into another location. Keep your connections point-to-point unless you can use proxyjump.
- 3y ago
- wahern 3y agomusl libc refuses to implement dlclose[1] for precisely the reason that modules too often misbehave when dropped at runtime, and requiring this behavior is very rarely needed if ever. The number of modules that will be loaded is almost always bounded, and keeping a module around in memory is mostly harmless; certainly less harmful on average than trying to unload it. [1] see https://wiki.musl-libc.org/functional-differences-from-glibc.html#Unloading-libraries https://wiki.musl-libc.org/functional-differences-from-glibc...
- 10000truths 3y ago> musl libc refuses to implement dlclose[1] for precisely the reason that modules too often misbehave when dropped at runtime, Which is some rather silly logic, because now musl's libdl is just one more module that misbehaves at runtime by refusing to clean up after itself. Other libraries being sloppy about resource cleanup is no excuse for musl to abdicate its own responsibilities. > and requiring this behavior is very rarely needed if ever. The number of modules that will be loaded is almost always bounded, and keeping a module around in memory is mostly harmless; I take it you've never used an application that hot-reloads plugins? Who cares about memory leaks, right? > certainly less harmful on average than trying to unload it. That's for the application to decide, not musl. If I don't trust a library to behave properly, I will either not dlopen it at all, or I'll dlopen it in an appropriately sandboxed subprocess and interact with it via pipes.
- wahern 3y ago> I take it you've never used an application that hot-reloads plugins? Used and implemented many times. Sometimes this could be a problem, but IME most plugin architectures I've seen don't rely on static constructors or destructors for implicit registration/deregistration of callbacks. Among other reasons, it's a landmine of multi-threading issues and glibc itself has historically had many bugs related to this, albeit mostly a consequence of glibc implementing (until recently) libpthread separate from libc. Also, POSIX does not guarantee that dlclose does anything: "An application writer may use dlclose() to make a statement of intent on the part of the process, but this statement does not create any requirement upon the implementation." > Who cares about memory leaks, right? I can't find the blog post now, but IIRC Solaris made getenv thread-safe by never deallocating any memory allocated for the environ array or its contents. The blog post had a curious and memorable defense which explained how the memory usage was asymptotically bounded, similar to how hash table operations are described as asymptotically O(1) in time. And Solaris cares very much about memory management--unlike Linux or FreeBSD it implements strict memory accounting, so no OOM killer non-sense. Is it ideal? No. But these are legitimate decisions taken in light of lots of experience, and if you want to write truly robust applications one has to pay attention to standards, implementation details, and best practices. IME best practice wrt module systems on Unix and the open source world generally has always been to avoid implicit registration/deregistration of callbacks. I believe the same is true in the Windows world, though I've never written much software for Windows. (Anyone know if FreeLibrary immediately invokes static destructors?) FWIW, I have implemented hacks which relied on static constructors and where hot reloading was more problematic in the absence of static destructors. For example, I once implemented a proof of concept for automatic runtime reloading of SSL certificates and keys in the Splunk server by writing a library interposer (supporting FreeBSD, Linux, macOS, and Solaris) that installed global OpenSSL SSL_CTX callbacks. So I know why these things could be useful. But that example was a PoC hack. And even in situations where dlcose is supported as one might naively expect, you can still easily run into similar gotchas, like a user loading a module twice under different names. You have to choose either to deal with it the hard way, or disclaim support for such niche cases.
- 1letterunixname 3y agoI use a USB security key for gpg-agent. gpg-agent is used as an ssh-agent replacement (also replaces the venerable monkeysphere script). gpg agent forwarding is also needed via StreamLocalBindUnlink for signing built RPMs in a docker container on a remote (local LAN) Docker host. Sometimes rarely, I need to enable "ssh-agent" forwarding where I probably could proxyjump instead. My interactive ssh workflow is sometimes slowed as I use TOTP (via PAM) for 2FA. Noninteractive system accounts get dedicated ssh keys. host keys are constantly scanned and compared to a source of truth. Maybe I'll get around to deploying LDAP with OpenSSH-LPK to move away from flat files. Knee-jerk suggestion to rewrite the world in Rust combined with formal verification. But seriously, being an expert knife juggler is still gambling with the obvious compared to using safer tools in a safer manner. Rust needs the ubiquity of GCC[0] (partially by adding them to LLVM and adding std crates) and more attention paid to bloat (cargo-bloat, etc) before attempting to rewrite the world (apart from special cases). 0. https://gcc.gnu.org/backends.html https://gcc.gnu.org/backends.html
- quotemstr 3y agoWhy in the year of our Lord two thousand and twenty three do we still tolerate loading shared libraries with executable stacks?
- seanhunter 3y agoMan nice job. This is a really cool RCE and well handled.
- mewmew07 3y agoput this at the end of your ~/.ssh/config Host * ForwardAgent no ssh config uses tabs for indentation, make sure you got those right
- exabrial 3y agoPractically, what’s the immediate mitigation as updates are being published? Don’t ssh to unknown addresses from Linux?
- pritambaral 3y agoOn my machine, I just removed the 'ssh-pkcs11-helper' binary, given that I'll definitely have no need for it in the foreseeable future. Then I remembered I don't use 'ssh-agent' as my SSH Agent anyway; I use gpg-agent and I'm pretty confident in its security posture. Then I verified it: gpg-agent simply doesn't support any operations except signing with preloaded keys.
- pritambaral 3y agoThe suggestion floated in many comments here to create extra keypairs on remote hosts for access to other remotes from the former is a terrible idea. Resident private keys on a compromised machine can be exfiltrated. Even if they're passphrase protected, using them requires them to be decrypted on the machine they're resident on. `ssh-agent` is not the only SSH Agent. There exist SSH Agent implementations that are secure. E.g., gpg-agent. Use one of those, with the `ssh-add -c` flag or equivalent, and you get both security and convenience.