23 ms·
RegreSSHion: RCE in OpenSSH's server, on glibc-based Linux systems
- nfriedly 2y agoI stopped exposing SSH to the internet years ago. Now I connect over WireGuard, and then run SSH through that when I need to remotely admin something.
- HeadlessChild 2y agoI guess you could also spin up a OpenBSD server running SSH and use that as a jump host.
- megous 2y agoIn our experiments, it takes ~10,000 tries on average to win this race condition, so ~3-4 hours with 100 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime). Ultimately, it takes ~6-8 hours on average to obtain a remote root shell, because we can only guess the glibc's address correctly half of the time (because of ASLR). MaxStartups default is 10
- Haemm0r 2y agoQuestion regarding this from a non-guru: - Is it correct that this only works for user root if login with password/key for root is allowed? - Is it correct, that this only works if the attacker knows a login name valid for ssh?
- aflukasz 2y agoI believe knowing existing user name or using host-depended value does not matter. The exploit tries to interrupt handlers that are being run due to login grace period timing out - so we are already at a point where authentication workflow has ended without passing all the credentials. Plus, in the "Practice" section, they discuss using user name value as a way to manipulate memory at a certain address, so they want/need to control this value.
- vesinisa 2y agoEven if it means the attack takes 10x as long, it doesn't seem to be limited by bandwidth, only time. Might not take long before the bots appear that try to automatically exploit this on scale.
- ale42 2y agoSuch an amount of connections should anyway trigger all possible logging & IDS systems, right?
- megous 2y agoI doubt most servers use any such thing.
- stefan_ 2y agoIf you don’t value your time, sure? There’s thousands of systems trying to log into publicly accessible SSH servers all the time.
- XorNot 2y agoYeah slow bruteforces are running all over the net all the time. This means there's no reason not to throw this attack into the mix.
- Piskvorrr 2y agoIt should trigger fail2ban, that's for sure. Alerting is useless, with the volume of automated exploits attempted.
- TacticalCoder 2y ago> It should trigger fail2ban, that's for sure. But people here are going to explain that fail2ban is security theater...
- JackSlateur 2y agoDefault is 100: https://github.com/openssh/openssh-portable/blob/master/servconf.c#L413 https://github.com/openssh/openssh-portable/blob/master/serv...
- JeremyNT 2y agoThe config option called MaxStartups accepts a tuple to set 3 associated variables in the code. It wasn't clear to me which value people were referring to.
- jmclnx 2y agoThe default for MaxStartups is 10:30:100 10:30:60 is mentioned in the man for start:rate:full, so I set mine to that value. Thanks for the quote
- djmdjm 2y agoOpenSSH release notes: https://www.openssh.com/txt/release-9.8 https://www.openssh.com/txt/release-9.8 Minimal patches for those can't/don't want to upgrade: https://marc.info/?l=oss-security&m=171982317624594&w=2 https://marc.info/?l=oss-security&m=171982317624594&w=2
- morsch 2y ago> Exploitation on 64-bit systems is believed to be possible but has not been demonstrated at this time.
- djmdjm 2y agoI'm confident that someone will make a workable exploit against 64-bit systems.
- aaronmdjones 2y agoExploits only ever get better. Today's possible is next month's done.
- 0x0 2y agoPatch out for Debian 12; Debian 11 not affected. https://security-tracker.debian.org/tracker/CVE-2024-6387 https://security-tracker.debian.org/tracker/CVE-2024-6387
- nubinetwork 2y agoCan confirm, Pi OS bullseye also has the updated openssh.
- wiredfool 2y agoLooks like Focal (20.04) isn't on an affected version. Jammy (22.04) looks like it is.
- feurio 2y agoMy procrastination pays off ...
- medguru 2y agoAs do theirs ;)
- deleted 2y ago[deleted]
- metadat 2y agoWhat about, uh, 18.04? Edit: 18.04 Bionic is unaffected, the ssh version is 7.6 which is too old.
- creshal 2y agoIf you have extended support: Just update (if it's not so old that it's not even affected in the first place) If you don't have extended support: You're vulnerable to worse, easier to exploit bugs :)
- metadat 2y agoI can confirm this 18.04 machine still gets some important updates like kernel upgrades and patched versions of Apache.
- nubinetwork 2y agoI haven't seen an increase of ssh traffic yet, but the alert only went out a couple hours ago... hopefully distros will ship the patches quickly.
- booi 2y agoi would assume all the distros have patches ready to go awaiting the embargo lift.
- cperciva 2y agoThis is the sort of bug which pre-announcement coordination is designed for. Anyone who doesn't have patches ready was either forgotten (I've seen a few instances of "I thought you were going to tell them!") or isn't on the ball.
- nubinetwork 2y agoGentoo announced it at the same time as qualys, but they're currently trying to backport and bump users to a patched version. https://bugs.gentoo.org/935271 https://bugs.gentoo.org/935271
- nubinetwork 2y agoGentoo has pushed the patched version now.
- cperciva 2y agoPatch out for FreeBSD. Not clear if affected (it has only known to be exploitable with glibc, which we don't use) but best to be safe. https://www.freebsd.org/security/advisories/FreeBSD-SA-24:04.openssh.asc https://www.freebsd.org/security/advisories/FreeBSD-SA-24:04...
- jesprenj 2y ago> Finally, if sshd cannot be updated or recompiled, this signal handler race condition can be fixed by simply setting LoginGraceTime to 0 in the configuration file. This makes sshd vulnerable to a denial of service (the exhaustion of all MaxStartups connections), but it makes it safe from the remote code execution presented in this advisory.
- NelsonMinar 2y ago[flagged]
- letters90 2y ago> In our experiments, it takes ~10,000 tries on average to win this race condition, so ~3-4 hours with 100 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime). Ultimately, it takes ~6-8 hours on average to obtain a remote root shell, because we can only guess the glibc's address correctly half of the time (because of ASLR). Mitigate by using fail2ban? Nice to see that Ubuntu isn't affected at all
- simonjgreen 2y agoFor servers you have control over, as an emergency bandaid, sure. Assumes you are not on an embedded system though like a router.
- letters90 2y agoI didn't consider embedded, probably the biggest target for this.
- ulrikrasmussen 2y agoWhere do you see that Ubuntu isn't affected?
- rs_rs_rs_rs_rs 2y ago>Side note: we discovered that Ubuntu 24.04 does not re-randomize the ASLR of its sshd children (it is randomized only once, at boot time); we tracked this down to the patch below, which turns off sshd's rexec_flag. This is generally a bad idea, but in the particular case of this signal handler race condition, it prevents sshd from being exploitable: the syslog() inside the SIGALRM handler does not call any of the malloc functions, because it is never the very first call to syslog(). No mention on 22.04 yet.
- djmdjm 2y agoUbuntu isn't affected _by this exploit_
- 2y ago
- rfmoz 2y agoFrom the report: > Finally, if sshd cannot be updated or recompiled, this signal handler race condition can be fixed by simply setting LoginGraceTime to 0 in the configuration file. This makes sshd vulnerable to a denial of service (the exhaustion of all MaxStartups connections), but it makes it safe from the remote code execution presented in this advisory. Setting 'LoginGraceTime 0' in sshd_config file seems to mitigate the issue.
- yjftsjthsd-h 2y agoHang on, https://www.man7.org/linux/man-pages/man5/sshd_config.5.html https://www.man7.org/linux/man-pages/man5/sshd_config.5.html says > If the value is 0, there is no time limit. Isn't that worse?
- sodality2 2y agoIt sounds like at the end of the default 600 seconds is when the race condition occurs. Set no limit, and there is no end > In our experiments, it takes ~10,000 tries on average to win this race condition; i.e., with 10 connections (MaxStartups) accepted per 600 seconds (LoginGraceTime), it takes ~1 week on average to obtain a remote root shell.
- pm215 2y agoThe bug is a race condition which is triggered by code which runs when the timeout expires and the SIGALRM handler is run. If there is no time limit, then the SIGALRM handler will never run, and the race doesn't happen. (As the advisory notes, you do then have to deal with the DoS which the timeout setting is intended to avoid, where N clients all connect and then never disconnect, and they aren't timed-out and forcibly disconnected on the server end any more.)
- yjftsjthsd-h 2y agoThanks for the explanation; I'd skimmed a little too fast and assumed that this was the more traditional "how many attempts can we squeeze in each connection" rather than something at the end. I guess this makes the hardening advice about lowering that time limit kind of unfortunate.
- fanf2 2y agoIt’s also worth reading the release notes https://www.openssh.com/releasenotes.html https://www.openssh.com/releasenotes.html This is actually an interesting variant of a signal race bug. The vulnerability report says, “OpenBSD is notably not vulnerable, because its SIGALRM handler calls syslog_r(), an async-signal-safer version of syslog() that was invented by OpenBSD in 2001.” So a signal-safety mitigation encouraged OpenBSD developers to put non-trivial code inside signal handlers, which becomes unsafe when ported to other systems. They would have avoided this bug if they had done one of their refactoring sweeps to minimize the amount of code in signal handlers, according to the usual wisdom and common unix code guidelines.
- djmdjm 2y agoTheo de Raadt made an, I think, cogent observation about this bug and how to prevent similar ones: no signal handler should call any function that isn't a signal-safe syscall. The rationale is that, over time, it's too way easy for any transitive call (where it's not always clear that it can be reached in signal context) to pick up some call that isn't async signal safe.
- fanf2 2y agoExactly, yes :-) Signal handlers have so many hazards it's vital to keep them as simple as possible.
- growse 2y agoI'm not overly familiar with the language and tooling ecosystem, but how trivial is this to detect on a static analysis?
- kccqzy 2y agoQuite easy.
- rwmj 2y agoA rule I try to follow: either set a global variable or write to a self pipe (using the write syscall), and handle the signal in the main loop.
- FiloSottile 2y agoInterestingly, the RCE fix was "smuggled" in public almost a month ago. When PerSourcePenalties are enabled, sshd(8) will monitor the exit status of its child pre-auth session processes. Through the exit status, it can observe situations where the session did not authenticate as expected. These conditions include when the client repeatedly attempted authentication unsucessfully (possibly indicating an attack against one or more accounts, e.g. password guessing), or when client behaviour caused sshd to crash (possibly indicating attempts to exploit sshd). When such a condition is observed, sshd will record a penalty of some duration (e.g. 30 seconds) against the client's address. https://github.com/openssh/openssh-portable/commit/81c1099d22b81ebfd20a334ce986c4f753b0db29 https://github.com/openssh/openssh-portable/commit/81c1099d2... It's not really a reversable patch that gives anything away to attackers: it changes the binary architecture in a way that has the side-effect of removing the specific vulnerability and also mitigates the whole exploit class, if I understand it correctly. Very clever.
- fanf2 2y agoThat's not the RCE fix, this is the RCE fix https://news.ycombinator.com/item?id=40843865 https://news.ycombinator.com/item?id=40843865 That's a previously-announced feature for dealing with junk connections that also happens to mitigate this vulnerability because it makes it harder to win the race. Discussed previously https://news.ycombinator.com/item?id=40610621 https://news.ycombinator.com/item?id=40610621
- djmdjm 2y agoNo, it's a fix. It completely removes the signal race as well as introducing a mitigation for similar future bugs
- FiloSottile 2y agoThe ones you link are the "minimal patches for those can't/don't want to upgrade". The commit I am linking to is taken straight from the advisory. On June 6, 2024, this signal handler race condition was fixed by commit 81c1099 ("Add a facility to sshd(8) to penalise particular problematic client behaviours"), which moved the async-signal-unsafe code from sshd's SIGALRM handler to sshd's listener process, where it can be handled synchronously: https://github.com/openssh/openssh-portable/commit/81c1099d22b81ebfd20a334ce986c4f753b0db29 Because this fix is part of a large commit (81c1099), on top of an even larger defense-in-depth commit (03e3de4, "Start the process of splitting sshd into separate binaries"), it might prove difficult to backport. In that case, the signal handler race condition itself can be fixed by removing or commenting out the async-signal-unsafe code from the sshsigdie() function The cleverness here is that this commit is both "a previously-announced feature for dealing with junk connections", and a mitigation for the exploit class against similar but unknown vulnerabilities, and a patch for the specific vulnerability because it "moved the async-signal-unsafe code from sshd's SIGALRM handler to sshd's listener process, where it can be handled synchronously". The cleverness is that it fixes the vulnerability as part of doing something that makes sense on its own, so you wouldn't know it's the patch even looking at it.
- CJefferson 2y agoThis is a really good find. One thing which (as an independant person, who isn't doing any of the work!) is it often feels like in order to 'win', people are expected to find a full chain which gives them remote access, rather than just finding one issue, and getting it fixed / getting paid for it. It feels to me like finding a single hole should be sufficient -- one memory corruption, one sandbox escape. Maybe at the moment there are just too many little issues, that you need a full end-to-end hack to really convince people to take you seriously, or pay out bounties?
- rlpb 2y agoThere are many wannabe security researchers who find issues that are definitely not exploitable, and then demand CVE numbers and other forms of recognition or even a bounty. For example, there might be an app that crashes when accepting malformed trusted input, but the nature of the app is that it's never intended to and realistically never will be exposed to an adversary. In most people's eyes, these are simply bugs, not security bugs, and while are nice to fix, aren't on the same level. It's not very difficult to find one of these! So there is a need to differentiate between "real" security bugs [like this one] and non-security-impacting bugs, and demonstrating how an issue is exploitable is therefore very important. I don't see the need to demonstrate this going away any time soon, because there will always be no end of non-security-impacting bugs.
- nubinetwork 2y ago> There are many wannabe security researchers who find issues that are definitely not exploitable, and then demand CVE numbers and other forms of recognition or even a bounty I believe this has happened to curl several times recently.
- School-Cotton 2y agoIt happens constantly to any startup with a security@ email address.
- PhilipRoman 2y ago
- INTPenis 2y agoCorrect me if I'm wrong but it seems like sshd on RHEL-based systems is safe because they never call syslog. They run sshd with the -D option already, logging everything to stdout and stderr, as their systemd already catches this output and sends it to journal for logging. So I don't see anywhere they would be calling syslog, unless sshd does it on its own. At most maybe add OPTIONS=-e into /etc/sysconfig/sshd.
- j16sdiz 2y agoNow, how many remote exploit do we have in openbsd?
- cyberpunk 2y agoNo more than before this; openbsd is not vulnerable to this exploit due to a different syslog() implementation.
- ZiiS 2y agoWorth being explicit here. The OpenBSD syslog is not just 'different' enough that it was luckily uneffected. It was intentionally designed to avoid this situation more than 20 years ago.
- fulafel 2y agoAlso there's no publicly known exploit for this one yet even for Linux. The advisory says Qualys put exploit development on hold to coordinate the fix.
- yjftsjthsd-h 2y agoTwo in living memory? If you know something with a better track record do speak up.
- DEADMINCE 2y agoSEL4 and derivatives. For starters. And if you want to simply go by vulnerability counts, as though that meant something, let's throw in MenuetOS and TempleOS.
- yjftsjthsd-h 2y agoOkay, let's say if you know something useful with a better record. TempleOS doesn't have network, so while it's genuinely cool it's not useful to most people. MenuetOS does have network but poor software compatibility. I would actually love to see a seL4 distro but AFAIK it's pretty much only ever used as a hypervisor with a "real" (normal) OS under it, often (usually?) Linux-based. We can certainly consider to what degree OpenBSD is useful with just the base system, but it does include everything out of the box to be a web server with zero extra software added, including sshd in its native environment.
- djernie 2y agoRedHat put an 8.1 score on it: https://access.redhat.com/security/cve/cve-2024-6387 https://access.redhat.com/security/cve/cve-2024-6387
- zshrc 2y agoDoesn’t affect RHEL7 or RHEL8.
- chasil 2y agoOr RHEL9. $ rpm -q openssh openssh-8.7p1-38.0.1.el9.x86_64
- Ianvdl 2y ago> Statement > The flaw affects RHEL9 as the regression was introduced after the OpenSSH version shipped with RHEL8 was published.
- chasil 2y agoHowever, we see the -D option on the listening parent: $ ps ax | grep sshd | head -1 1306 ? Ss 0:01 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups As mentioned elsewhere here, is -D sufficient to avoid exploitation, or is -e necessary as well? $ man sshd | sed -n '/ -[De]/,/^$/p' -D When this option is specified, sshd will not detach and does not become a daemon. This allows easy monitoring of sshd. -e Write debug logs to standard error instead of the system log. RHEL9 is also 64-bit only, and we see from the notice: "we have started to work on an amd64 exploit, which is much harder because of the stronger ASLR." On top of writing the exploit to target 32-bit environments, this also requires a DSA key that implements multiple calls to free(). There is a section on "Rocky Linux 9" near the end of the linked advisory where unsuccessful exploit attempts are discussed.
- Arnavion 2y ago
- nj5rq 2y agoOpenBSD is notably not vulnerable, because its SIGALRM handler calls syslog_r(), an async-signal-safer version of syslog() that was invented by OpenBSD in 2001. Saving the day once again.
- apache101 2y agoTheo and team way ahead of their time like always.
- pjmlp 2y agoNot always, 36C3 - A systematic evaluation of OpenBSD's mitigations https://www.youtube.com/watch?v=3E9ga-CylWQ https://www.youtube.com/watch?v=3E9ga-CylWQ
- zshrc 2y agoNot always, but they make it their goal to be. Code standards are very strict in OpenBSD and security is always a primary thought...
- daneel_w 2y agoWouldn't a good systematic evaluation need (or at least benefit from) a few actual working exploits/PoCs? I keep asking this as a long-time OpenBSD user who is genuinely interested in seeing it done, but so far everyone who has said "it's flawed" also reserved themselves the convenience of not having to prove their point in a practical sense.
- DEADMINCE 2y ago> Wouldn't a good systematic evaluation need (or at least benefit from) a few actual working exploits/PoCs? Sure, see any of the previous exploits for sshd, or any other software shipped in the OpenBSD default install. > I keep asking this as a long-time OpenBSD user who is genuinely interested in seeing it done, but so far everyone who has said "it's flawed" also reserved themselves the convenience of not having to prove their point in a practical sense. The point is they have very little in the way of containing attackers and restricting what they can. Until pledge and unveil, almost all their focus in on eliminating bugs which hey, great, but let's have a little more in case you miss a bug and someone breaks in, eh? An insecure dockerized webserver protected with SELinux is safer than Apache on a default OpenBSD install.
- marcus0x62 2y agoPatch out for Arch Linux https://archlinux.org/packages/core/x86_64/openssh/ https://archlinux.org/packages/core/x86_64/openssh/ edit be sure to manually restart sshd after upgrading; my systems fail during key exchange after package upgrade until restarting the sshd service: % ssh -v 192.168.1.254 OpenSSH_9.8p1, OpenSSL 3.3.1 4 Jun 2024 ... output elided ... debug1: Local version string SSH-2.0-OpenSSH_9.8 kex_exchange_identification: read: Connection reset by peer Connection reset by 192.168.1.254 port 22
- frankjr 2y agoSame here. It's caused by the sshd daemon being split into multiple binaries. In fact, the commit which introduced the change mentions this explicitly: > NB. if you're updating via source, please restart sshd after installing, otherwise you run the risk of locking yourself out. https://github.com/openssh/openssh-portable/commit/03e3de416ed7c34faeb692967737be4a7bbe2eb5 https://github.com/openssh/openssh-portable/commit/03e3de416... Edit: Already reported at https://gitlab.archlinux.org/archlinux/packaging/packages/openssh/-/issues/5 https://gitlab.archlinux.org/archlinux/packaging/packages/op...
- qhwudbebd 2y agoOnce I'd finished upgrading my openssh instances (which are linked against musl not glibc) I thought it'd be interesting to have a poke at musl's syslog(3) and see if it allocates too and so is easily exploitable in the same way. But as far as I can see, it doesn't: https://github.com/bminor/musl/blob/master/src/misc/syslog.c https://github.com/bminor/musl/blob/master/src/misc/syslog.c Everything there is either on stack or in static variables protected from reentrancy by the lock. The {d,sn,vsn}printf() calls there don't allocate in musl, although they might in glibc. Have I missed anything here?
- singron 2y agoIf you are right about the allocations, then I think the worst it can do is deadlock since the locks aren't recursive. Deadlock in sigalrm could still lead to a DOS since that might prevent it from cleaning up connections.
- mananaysiempre 2y agoHeretical opinion: signal handler activations should count as separate threads for the purposes of recursive locking.
- bhawks 2y agoHow would be done without introducing deadlock?
- mananaysiempre 2y agoYou’d get a deadlock, absolutely. But I’m fine with that: if the thread wants to access some state protected by a mutex, then while holding it (effectively) spawns a signal handler activation and waits for it to complete, and the signal handler tries to accept some state protected by the same mutex, then the program has just deadlocked (mutex → signal handler → mutex) and deserves to hang (or die, as this is a very simple situation as far as deadlock detection goes). That’s in any case better than corrupted state.
- yjftsjthsd-h 2y ago> Exploitation on non-glibc systems is conceivable but has not been examined. ( https://www.openssh.com/txt/release-9.8 https://www.openssh.com/txt/release-9.8 ) Darn - here I was hoping Alpine was properly immune, but it sounds more like "nobody's checked if it works on musl" at this point.
- _ikke_ 2y ago> OpenSSH sshd on musl-based systems is not vulnerable to RCE via CVE-2024-6387 (regreSSHion). https://fosstodon.org/@musl/112711796005712271 https://fosstodon.org/@musl/112711796005712271
- TacticalCoder 2y agoAnd who was notoriously not exploitable? The ones hiding sshd behind port knocks. And fail2ban: would work too. And a restrictive firewall: would help too. I don't use port-knocking but I really just don't get all those saying: "It's security theater". We had not one but two major OpenSSH "near fiasco" (this RCE and the xz lib thing) that were both rendered unusable for attackers by using port knocking. To me port-knocking is not "security theater": it adds one layer of defense. It's defense-in-depth. Not theater. And the port-knocking sequence doesn't have to be always the same: it can, say, change every 30 seconds, using TOTP style secret sequence generation. How many exploits rendered cold dead in their tracks by port-knocking shall we need before people stop saying port-knocking is security theater? Other measures do also help... Like restrictive firewalling rules, which many criticize as "it only helps keep the logs smaller": no, they don't just help keep the logs smaller. I'm whitelisting the three ISP's IP blocks anyone can reasonably be needing to SSH from: now the attacker needs not only the zero-day, but it also need to know he needs to be on one of those three ISPs' IPs. The argument that consists in saying: "sshd is unexploitable, so nothing else must be done to protect the server" is... Dead.
- _joel 2y agoThose not notoriously exploitable were those using gated ssh access only via known IPs or connecting via tailnets/vpn.
- growse 2y agoWhat benefits does port knocking give over and above a simple VPN? They're both additional layers of authentication, except a VPN seems much more rigorous and brings potentially other benefits. In a world where tailscale etc. have made quality VPNs trivial to implement, why would I both with port knocking?
- jgalt212 2y agoVPNs drop your bandwidth speeds by 50% on average. And if tailscale has to use a relay server, instead of a direct connection, bandwidth will drop by 70-80%.
- deleted 2y ago[deleted]
- MaximilianEmel 2y agoWas this abused in the wild?
- matthewcroughan 2y agoFor my own setup, I'm looking into Path Aware Networking (PAN) architectures like SCION to avoid exposing paths to my sshd, without having to set up a VPN or port knocking. https://scion-architecture.net https://scion-architecture.net
- pgraf 2y agoGenuinely curious, how would you block an attacker from getting to your SSH port without knowing the path you will connect from (which is the case for remote access) at configuration time? I don‘t see how Path-Aware Networking would replace a VPN solution
- matthewcroughan 2y agoThe SCION Book goes over a lot of potential solutions that are possible because of the architecture, but my favorite is hidden paths. https://scion.docs.anapaya.net/en/latest/hidden-paths.html https://scion.docs.anapaya.net/en/latest/hidden-paths.html > Hidden path communication enables the hiding of specific path segments, i.e. certain path segments are only available for authorized ASes. In the common case, path segments are publicly available to any network entity. They are fetched from the control service and used to construct forwarding paths.
- ttul 2y agoTLDR: this vulnerability does appear to allow an attacker to potentially gain remote root access on vulnerable Linux systems running OpenSSH, with some important caveats: 1. It affects OpenSSH versions 8.5p1 to 9.7p1 on glibc-based Linux systems. 2. The exploit is not 100% reliable - it requires winning a race condition. 3. On a modern system (Debian 12.5.0 from 2024), the researchers estimate it takes: - ~3-4 hours on average to win the race condition - ~6-8 hours on average to obtain a remote root shell (due to ASLR) 4. It requires certain conditions: - The system must be using glibc (not other libc implementations) - 100 simultaneous SSH connections must be allowed (MaxStartups setting) - LoginGraceTime must be set to a non-zero value (default is 120 seconds) 5. The researchers demonstrated working exploits on i386 systems. They believe it's likely exploitable on amd64 systems as well, but hadn't completed that work yet. 6. It's been patched in OpenSSH 9.8p1 released in June 2024.
- jonaslejon 2y agoOpenSSH 9.8p1 was released July 1, 2024 according to https://www.openssh.com/releasenotes.html#9.8p1 https://www.openssh.com/releasenotes.html#9.8p1
- hsbauauvhabzb 2y agoI’m not sure how many Linux users would know if they’re using glibc or another variation. Is there a list?
- arjvik 2y agoIf you don't know, you're likely running glibc. Distros that use musl do so intentionally (alpine, etc.)
- arjvik 2y agoWhy is it that the ASLR only adds 1 bit of randomness (doubling the time it takes to win the attack)?
- InfiniteLoup 2y ago> 4. It requires certain conditions: - The system must be using glibc (not other libc implementations) - 100 simultaneous SSH connections must be allowed (MaxStartups setting) - LoginGraceTime must be set to a non-zero value (default is 120 seconds) Stupid question, perhaps, but if those two lines inside the sshd_config are commented out with '#', does this mean that grace period and max. sessions are technically unlimited and therefore potentially vulnerable? Found my own answer: If the values are commented out, it means that the default values are being used. If the file hasn't been modified the default values are those you see inside the config file.
- lostmsu 2y agoOut of curiosity, does Windows have anything as disruptive as signals? I assume it is also not vulnerable, because SSH server there do not use glibc.
- relaxing 2y agoYes, Windows has signals. No Windows, doesn’t have glibc. But don’t worry, msvcrt has its own vulnerabilities.
- acatton 2y agoYearly reminder to run your ssh server behind spiped.[1] [2] [3] [1] https://www.tarsnap.com/spiped.html https://www.tarsnap.com/spiped.html [2] https://news.ycombinator.com/item?id=29483092 https://news.ycombinator.com/item?id=29483092 [3] https://news.ycombinator.com/item?id=28538750 https://news.ycombinator.com/item?id=28538750
- gruez 2y agoWhat's the advantage of this relatively obscure tool compared to something standard like wireguard or stunnel?
- acatton 2y ago* The tool is not obscure, it's packaged in most distributions.[1][2][3] It was written and maintained by Colin Percival, aka "the tarnsnap guy" or "the guy who invented scrypt". He is the security officer for FreeBSD. * spiped can be used transparently by just putting a "ProxyCommand" in your ssh_config. This means you can connect to a server just by using "ssh", normally. (as opposed to wireguard where you need to always be on your VPN, otherwise connnect to your VPN manually before running ssh) * As opposed to wireguard which runs in the kernel, spiped can easily be set-up to run as a user, and be fully hardened by using the correct systemd .service configuration [4] * The protocol is much more lightweight than TLS (used by stunnel), it's just AES, padded to 1024 bytes with a 32 bit checksum. [5] * The private key is much easier to set up than stunnel's TLS certificate, "dd if=/dev/urandom count=4 bs=1k of=key" and you're good to go. [1] https://packages.debian.org/bookworm/spiped https://packages.debian.org/bookworm/spiped [2] https://www.freshports.org/sysutils/spiped/ https://www.freshports.org/sysutils/spiped/ [3] https://archlinux.org/packages/extra/x86_64/spiped/ https://archlinux.org/packages/extra/x86_64/spiped/ [4] https://ruderich.org/simon/notes/systemd-service-hardening https://ruderich.org/simon/notes/systemd-service-hardening [5] https://github.com/Tarsnap/spiped/blob/master/DESIGN.md https://github.com/Tarsnap/spiped/blob/master/DESIGN.md
- SparkyMcUnicorn 2y agoWireguard can also run in userspace (e.g. boringtun[0], wireguard-go[1], Tailscale). [0] https://github.com/cloudflare/boringtun https://github.com/cloudflare/boringtun [1] https://git.zx2c4.com/wireguard-go/about/ https://git.zx2c4.com/wireguard-go/about/
- poikroequ 2y agoAfter the xz backdoor a few months ago, I decided to turn off SSH everywhere I don't need it, either by disabling it or uninstalling it entirely. While SSH is quite secure, it's too lucrative a target, so it will always pose a risk.
- lupusreal 2y agoI now only bind services to wireguard interfaces. The bet is that a compromise in both the service and wireguard at the same time is unlikely (and I have relatively high confidence in wireguard.)
- devsda 2y agoI'm confident in making ssh changes while logged in via ssh. Compared to ssh, wireguard configs feel too easy to mess up and risk getting locked out if its the only way of accessing the device.
- pja 2y agoUse tailscale instead? It’s user friendly enough that you’re unlikely to mess it up.
- someplaceguy 2y ago> The bet is that a compromise in both the service and wireguard at the same time is unlikely An RCE in wireguard would be enough -- no need to compromise both.
- remram 2y agoI was going to say something like this, but in practice wireguard is very very tiny. It doesn't have pluggable authentication, or passwords, or user transitions, or forked subprocesses, or systemd integrations. Using it or another simple secure transport in front of SSH is probably a good idea.
- 2y ago
- gavinhoward 2y agoAs someone who does unspeakable, but safe, things in signal handlers, I can confirm that it is easy to stray off the path of async-signal-safety.
- guerby 2y agoI agree and I'm surprised OpenSSH developpers did not remove the use of SIGALRM and replace it by select/poll timer and explicitly managed future event list. Likely more portable and safe by default from this class of bugs that has bitten ssh code more than one time now... Defensive programming tells us to minize code in signal handlers and the safest is to avoid using the signal at all when possible :).
- NelsonMinar 2y agoOne interesting comment in the OpenSSH release notes > Successful exploitation has been demonstrated on 32-bit Linux/glibc systems with ASLR. Under lab conditions, the attack requires on average 6-8 hours of continuous connections up to the maximum the server will accept. Exploitation on 64-bit systems is believed to be possible but has not been demonstrated at this time. It's likely that these attacks will be improved upon. https://www.openssh.com/releasenotes.html https://www.openssh.com/releasenotes.html
- deleted 2y ago[deleted]
- sneak 2y agoWhy are we all still running an ssh server written in an unsafe language in 2024?
- deleted 2y ago[deleted]
- bauruine 2y agoBecause nobody has written an sshd in a memory safe language with the same track record of safety as OpenSSH. I personally wouldn't trust a new sshd for a few years at least.
- fullspectrumdev 2y agoThere’s a Rust library that implements most of the protocol, but I’ve not found a “drop in replacement” using said library yet. Might actually make for a fun side project to build a SSH server using that library and see how well it performs.
- SoftTalker 2y agoDoes Rust have some invulnerability to race conditions?
- pornel 2y agoIt does have invulnerability to data races. However, that guarantee applies only to data types and code in Rust. The dangerous interaction between signals and other functions is outside of what Rust can help with.
- xmodem 2y agoThere are several crates available which implement the dangerous parts of signal handling safely for you.
- maxmalkav 2y agoQuoting some ska tune in a SSH vulnerability report really caught me off ward, but I loved it.
- betaby 2y agoIn some setups I decided to have jumphost via HAproxy ssl as described there https://www.haproxy.com/blog/route-ssh-connections-with-haproxy https://www.haproxy.com/blog/route-ssh-connections-with-hapr... so no ssh directly exposed at all.
- aflukasz 2y agoSo this is effectively like ProxyJump, just with the jump node exposed over SSL and backed by HAProxy binary instead of OpenSSH? What benefits do you see? I mean, you still expose some binary that implements authentication and authorization using cryptography. I think that even RBAC scenarios described in the link above should be achievable with OpenSSH, right?
- ementally 2y agohttps://dustri.org/b/notes-on-regresshion-on-musl.html https://dustri.org/b/notes-on-regresshion-on-musl.html
- jamilbk 2y agoFrom the diff introducing the bug [1], the issue according to the analysis is that the function was refactored from this: void sigdie(const char *fmt,...) { #ifdef DO_LOG_SAFE_IN_SIGHAND va_list args; va_start(args, fmt); do_log(SYSLOG_LEVEL_FATAL, fmt, args); va_end(args); #endif _exit(1); } to this: void sshsigdie(const char *file, const char *func, int line, const char *fmt, ...) { va_list args; va_start(args, fmt); sshlogv(file, func, line, 0, SYSLOG_LEVEL_FATAL, fmt, args); va_end(args); _exit(1); } which lacks the #ifdef. What could have prevented this? More eyes on the pull request? It's wild that software nearly the entire world relies on for secure access is maintained by seemingly just two people [2]. [1] https://github.com/openssh/openssh-portable/commit/752250caabda3dd24635503c4cd689b32a650794 https://github.com/openssh/openssh-portable/commit/752250caa... [2] https://github.com/openssh/openssh-portable/graphs/contributors https://github.com/openssh/openssh-portable/graphs/contribut...
- ghostpepper 2y ago> It's wild that software nearly the entire world relies on for secure access is maintained by seemingly just two people obligatory xkcd https://xkcd.com/2347/ https://xkcd.com/2347/
- unilynx 2y agoIt's always easy with hindsight to tell how to prevent something. In this case, a comment might have helped why the #ifdef was needed, eg void CloseAllFromTheHardWay(int firstfd) //Code here must be async-signal-safe! Locks may be in indeterminate state { struct rlimit lim; getrlimit(RLIMIT_NOFILE,&lim); for (int fd=(lim.rlim_cur == RLIM_INFINITY ? 1024 : lim.rlim_cur);fd>=firstfd;--fd) close(fd); } Although to be honest, getrlimit isn't actually on the list here: https://man7.org/linux/man-pages/man7/signal-safety.7.html https://man7.org/linux/man-pages/man7/signal-safety.7.html But I hope that removing the comment or modifying code with a comment about async-signal-safe might have been noticed in review. The code you quoted only has the mention SAFE_IN_SIGHAND to suggest that this code might need to be async-signal-safe
- loeg 2y ago
- sharpshadow 2y agoHow refreshing to read a pure txt on the phone. It displays text better than a dozen websites.
- thenickdude 2y agoThere's a purported PoC exploit that delivers shellcode available on GitHub, but I saw someone comment the link here, and then their comment disappeared on the next refresh.
- Wowfunhappy 2y agoThey note that OpenBSD is not vulnerable. Is macOS also (probably?) safe then?
- rrix2 2y agoSo what are the odds we ever see a viable x86_64 exploit?
- Sparkyte 2y agoPeople still use SSH these days? I kid, but really you probably shouldn't on Production. You should be exporting your logs and everything else. The host or VM bootstrapped golden images with everything as needed. It is okay to start that way and figure out your enternals but that isn't for Production. Production is a locked down closed environment. Recomment from another Hacker News post.
- remram 2y agoYes I too have production systems running no software at all.
- Uptrenda 2y agoAnyone else here just totally crap bricks when they see news like this? Like, I wake up and instantly think all my servers are going to be owned and freak out. Though its usually never that bad, sometimes it is. Looks like in this case my debian servers were fine though. edit: maybe i should add an iptable rule to only allow ssh from my IP.
- drsanthosh1997 2y agohow to install openssh 9.8p1 in ubuntu 22.04.4 LTS
- ransom1538 2y agoIf you are on GCP and don't have time to patch, GCP recommends turning off your port 22 for now. https://cloud.google.com/compute/docs/security-bulletins https://cloud.google.com/compute/docs/security-bulletins 1. Find things that are 0.0.0.0 port 22, example, https://gist.github.com/james-ransom/97e1c8596e28b9f759bac79a34dd92ac https://gist.github.com/james-ransom/97e1c8596e28b9f759bac79... 2. Force them to the local network, gcloud compute firewall-rules update default-allow-ssh --source-ranges=10.0.0.0/8 --project=$i;
- madacol 2y agoTLDR: these are the safe versions 4.4p1 <= OpenSSH < 8.5p1 AND >= 9.8p1 --- - OpenSSH < 4.4p1 is vulnerable to this signal handler race condition, if not backport-patched against CVE-2006-5051, or not patched against CVE-2008-4109, which was an incorrect fix for CVE-2006-5051; - 4.4p1 <= OpenSSH < 8.5p1 is not vulnerable to this signal handler race condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" that was added to sigdie() by the patch for CVE-2006-5051 transformed this unsafe function into a safe _exit(1) call); - 8.5p1 <= OpenSSH < 9.8p1 is vulnerable again to this signal handler race condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" was accidentally removed from sigdie()).
- programmer86 2y agoI have a Ubuntu 22.10 system with ssh using socket activation. Does this bug still have an impact? I've read that Ubuntu 24.4 is safe because of socket activation. Can any expert here comment?