12 ms·

OpenSSH Backdoors

257 points by benhawkes 2y ago

throwaway984393 2y ago
[dead]
DonHopkins 2y ago
> The "many eyes" theory of open source security isn't popular right now, but it certainly seems like bigger targets have smaller margins for error. https://news.ycombinator.com/item?id=20383529 https://news.ycombinator.com/item?id=20383529 DonHopkins on July 8, 2019 | parent | context | favorite | on: Contributor Agreements Considered Harmful And then there's Linus's Law, which he made up, then tried to blame on Linus. "Given enough eyeballs, all bugs are shallow." -Eric S Raymond "My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt https://en.wikipedia.org/wiki/Linus%27s_Law https://en.wikipedia.org/wiki/Linus%27s_Law >In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate.[4] While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs".[5][6] >Although detection of even deliberately inserted flaws[7][8] can be attributed to Raymond's claim, the persistence of the Heartbleed security bug in a critical piece of code for two years has been considered as a refutation of Raymond's dictum.[9][10][11][12] Larry Seltzer suspects that the availability of source code may cause some developers and researchers to perform less extensive tests than they would with closed source software, making it easier for bugs to remain.[12] In 2015, the Linux Foundation's executive director Jim Zemlin argued that the complexity of modern software has increased to such levels that specific resource allocation is desirable to improve its security. Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking".[11] Large scale experiments or peer-reviewed surveys to test how well the mantra holds in practice have not been performed. The little experience Raymond DOES have auditing code has been a total fiasco and embarrassing failure, since his understanding of the code was incompetent and deeply tainted by his preconceived political ideology and conspiracy theories about global warming, which was his only motivation for auditing the code in the first place. His sole quest was to discredit the scientists who warned about global warming. The code he found and highlighted was actually COMMENTED OUT, and he never addressed the fact that the scientists were vindicated. http://rationalwiki.org/wiki/Eric_S._Raymond http://rationalwiki.org/wiki/Eric_S._Raymond >During the Climategate fiasco, Raymond's ability to read other peoples' source code (or at least his honesty about it) was called into question when he was caught quote-mining analysis software written by the CRU researchers, presenting a commented-out section of source code used for analyzing counterfactuals as evidence of deliberate data manipulation. When confronted with the fact that scientists as a general rule are scrupulously honest, Raymond claimed it was a case of an "error cascade," a concept that makes sense in computer science and other places where all data goes through a single potential failure point, but in areas where outside data and multiple lines of evidence are used for verification, doesn't entirely make sense. (He was curiously silent when all the researchers involved were exonerated of scientific misconduct.) More context: https://news.ycombinator.com/item?id=20382640 https://news.ycombinator.com/item?id=20382640
- toast0 2y ago
  > Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking". This makes a lot of sense, because for the most part, you only go looking for bugs when you've run into a problem. Looking for bugs you haven't run into is a lot harder (especially in complex software like OpenSSL), you might get lucky and someone sees a bug while looking for something else, but mostly things go unlooked at until they cause a problem that attracts attention. Even when you pay for a professional audit, things can be missed; but you'll likely get better results for security with organized and focused reviews than by hoping your user base finds everything.
  - cbsmith 2y ago
    Large open source projects are regularly subjected to security audits. I think the reality is that closed source software is vulnerable to the same attack, the only difference is fewer eyes to see it and more likely a profit motive will keep those eyes directed in other ways.
- poikroequ 2y ago
  It's not a complete fallacy. In events like this, after the news hits, there are a flurry of eyeballs looking, at least for a little while. The heartbleed bug got people to look at the openssl code and realize what a mess that code is. Inspectre and meltdown has led to the discovery of many more CPU vulnerabilities. After Chatgpt hit the market, there has been lots of new research on AI security, such as into prompt injection attacks.
davidfiala 2y ago
Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated resources and a financial stake in preventing backdoors?
- chgs 2y ago
  Better. Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees. Now imagine why a nation state would want to get staff working in such a company. Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.
  - wannacboatmovie 2y ago
    Your argument is a model that does no vetting of contributors whatsoever, which resulted in the catastrophe that is the topic of discussion, is better than a hypothetical company which is full of compromised developers that have free reign to commit to the source tree with no oversight? That sounds extremely contrived.
    - adolph 2y ago
      If you are positing that government infiltration of companies is hypothetical and not a real threat, here is an example of compromised corporate staff: https://en.wikipedia.org/wiki/Saudi_infiltration_of_Twitter https://en.wikipedia.org/wiki/Saudi_infiltration_of_Twitter
    - chgs 2y ago
      This wasn’t a contributor to OpenSSH, it was a deep level supply chain attack - something that closed source commercial companies are not immune to. Given how much closed source companies love BSD/apache/etc licenses where they can simply use these low level libraries and charge for stuff on the top I’m not sure how they would be immune from such an attack. The risk from this was highlighted in xkcd back in 2020 https://xkcd.com/2347/ https://xkcd.com/2347/
Vecr 2y ago
Was there ever a writeup of exactly how the XZ exploit worked? I mean exactly, I get the general overview and even quite a few of the specifics, but last time I checked no one had credibly figured out exactly how all the obfuscated components went together.
- 4llan 2y ago
  Gynvael Coldwind made a great analysis about it: https://gynvael.coldwind.pl/?lang=en&id=782 https://gynvael.coldwind.pl/?lang=en&id=782 https://news.ycombinator.com/item?id=39878681 https://news.ycombinator.com/item?id=39878681 xz/liblzma: Bash-stage Obfuscation Explained
  - mananaysiempre 2y ago
    That is, as it says in the title, about the Bash-stage obfuscation. That’s fun but it’d also be interesting to know what capabilities the exploit payload actually provided to the attacker. Last I looked into that a month or so ago there were at least two separate endpoints already discovered, and the investigation was still in progress.
    - dangsux 2y ago
      [dead]
- kva-gad-fly 2y ago
  https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 ?
  - Vecr 2y ago
    Yeah, what's posted by you and other users so far is stuff I know, build scripts, injection, obfuscation. I'm more looking for a careful reverse engineering of the actual payload.
    - 2y ago
rwmj 2y ago
> However, it's interesting to note that in both 2002 and 2024 we got a backdoor rather than a bugdoor. As far as we know. Related, there was a pretty interesting backdoor-by-bug attempt on the Linux kernel (at least, one that we know of) back in 2003: https://lwn.net/Articles/57135/ https://lwn.net/Articles/57135/ The Linux "bug" was unsophisticated by modern standards, but you could imagine a modern equivalent that's harder to spot: Make the "bug" happen across several lines of code, especially if some of those lines are part of existing code (so don't appear in the patch being reviewed). Ensure the compiler doesn't warn about it. Make the set of triggering events very unlikely unless you know the attack. It would be very surprising to me if three letter agencies hadn't done or attempted this.
- mmsc 2y ago
  And in 2010, a similar backdoor appeared in UnrealICRD: https://lwn.net/Articles/392201/ https://lwn.net/Articles/392201/. Also in proftpd: https://www.aldeid.com/wiki/Exploits/proftpd-1.3.3c-backdoor https://www.aldeid.com/wiki/Exploits/proftpd-1.3.3c-backdoor. Both were done by ac1db1tch3z who the author of OP's post, Ben Hawkes, got a shoutout from for another local privilege escalation vulnerability from over a decade ago :-). Anyways, in response to the backdoor in unrealircd, Core Security came up with a "hiding backdoors in plain sight" challenge: https://seclists.org/fulldisclosure/2010/Jul/66 https://seclists.org/fulldisclosure/2010/Jul/66 "Bugdoors" are not new, and I'm sure some have been patched without anybody realizing they were introduced maliciously.
  - baby 2y ago
    And there was the socat backdoor
  - cqqxo4zV46cp 2y ago
    Oh man! I’d forgotten all about the UnrealIRCd backdoor! I was running an IRC network at the time. What a blast from the past.
- fouronnes3 2y ago
  To think that there's a safe somewhere in a TLA basement with a "how to get root anywhere" tutorial.
egberts1 2y ago
Ya think that the tests subditectory would contain sufficient "integrity" test cases to ensure that it doesn't fail, but alas ... nooooooo.
- tedunangst 2y ago
  What does this mean?
- saagarjha 2y ago
  What exactly would you test here?
  - egberts1 2y ago
    Permutation, value range, logic testing ... CODE COVERAGE!
    - saagarjha 2y ago
      None of those would protect against backdoors.
jmakov 2y ago
At this point what makes us think all major contributors are not on a payroll of one or the other state agency? The attack surface of the whole SW supply chain is huge.
- deleted 2y ago
  [deleted]
- Sesse__ 2y ago
  All major, and not a single one of them has leaked it?
  - Jerrrrrrry 2y ago
    Besides the literal constant torrent of evidence?
    - meepmorp 2y ago
      Since it's a literal constant torrent, it should be easy to pick out some more specific things for the rest of us, please.
      - Jerrrrrrry 2y ago
        "Facebook on Monday joined a lawsuit pressing the Obama administration to allow it to disclose more details of its forced cooperation). Google and Microsoft filed suit in June." Not to mention Twitter files. These are just the companies with enough legal gravitas and social gravity to even mention the canary dying.
- tedunangst 2y ago
  Sounds like a good social experiment. Work your way up to major contributor for a project, see how long until you're approached by the MIB.
mmh0000 2y ago
Whenever these stories come up, I like to remind people about the Underhanded C Code Contest[1] and the IOCCC[2] TL;DR: Clever C programmers can hide very evil shit in plain sight. [1] https://www.underhanded-c.org/ https://www.underhanded-c.org/ [2] https://www.ioccc.org/ https://www.ioccc.org/
- dkga 2y ago
  Wow, I was definitely not aware of this.
- cortesoft 2y ago
  What happened to the underhanded c contest? Seems the last one was in 2016?
  - thelastparadise 2y ago
    Maybe because absolutely no one trusts any C code anymore.
  - ruk_booze 2y ago
    How sad to see that this is no more. I helped out reviewing the winning contribution[1] by Linus Åkesson, for what I now know was the final contest. Linus, aka ”LFT”, is a remarkable and truly talented person in so many ways. If you haven’t heard of him before, I suggest you check out all of his projects. [1] https://www.linusakesson.net/programming/underhanded/2015.php https://www.linusakesson.net/programming/underhanded/2015.ph...
daghamm 2y ago
Even further back, someone claimed that a three letter agency had paid some developer to introduce backdoor to openbsd (or possibly openssh). Theo did not belive this and publicly disputed these claim and even revealed the name of the whistle-blower. But I have always felt the story rang true and Theo sound not have been so dismissing. Can't find the story, but it should be on the mailing lists somewhere
- jmclnx 2y ago
  I remember this, it was the FBI. OpenBSD people did a huge audit and nothing was found. That was also like 20 years ago. Also, other articles stated that never happened. Plus, the "backdoor" in OpenSSH was a Linux only thing possibly related to systemd. It never affected OpenBSD. That is because of Linux people patching OpenSSH with "dependency hell". I believe systemd people is doing something about these dependency chains.
  - Arch-TK 2y ago
    The "thing to do" about the dependencies is not to have them in the first place. Distributions where patching OpenSSH to add a libsystemd dependency instead of adding 15 lines of code.
  - fijiaarone 2y ago
    So you’re saying more than 1 person was paid to put the back door in.
    - bigiain 2y ago
      And at least one of them was intentionally whistleblown to create an OpenBSD witchhunt wasting both OpenBSD developers time and distraction all the other *BSD and Linux security/devs, while the _real_ target slid under the radar...
- daneel_w 2y ago
  It was talk about a backdoor in OpenBSD's IPsec stack. The software was audited and nothing was found. The person who stepped forward (after being named) claimed on Twitter to have been formerly involved with the FBI and a supposed project of theirs looking into the feasibility of infiltrating the OpenBSD developer sphere in order to plant a backdoor, but that the project never reached planning. Edit: the discussions and auditing of the IPsec stack happened around 2011 if memory serves me right. The supposed backdoor "happened" a decade earlier. The "agent" was named Greg Perry.
TacticalCoder 2y ago
There's another venue for backdoors in most Linux distros. It's insanely complex by design (so that an endless supply of backdoors can be implanted), touches absolutely everything, was required for the XZ backdoor to work, etc. But it you mention its name, you get downvoted.
mfreeman451 2y ago
[dead]
bawolff 2y ago
Must be the only time checksums were actually useful.
udev4096 2y ago
I wonder if every major open source project should adopt the way SQLite operates. No outside contributions at all
- imhoguy 2y ago
  What is the meaning of "outside"? What if SQLite founder decides to step down. In case of `xz` a fresh insider was a malicious actor who was supposed to help keep up the maintenance. I think open-community projects like Linux Kernel have benefits as they won't die with founder interest waning, still there is more eyes to find issues very early.
gorgoiler 2y ago
In terms of its level of severity (and all round insanity) attacking OpenSSH with a backdoor is as if someone had hoax-mailed a packet labelled “Anthrax” to every tech business in the world. Brazenly stupid and pointlessly broad: the motivation could just as easily have been to cause mass societal disruption (“terrorism”) instead of a targeted attack that just happened to sweep up everyone else in its arms.
hi-v-rocknroll 2y ago
My wish is that FOSS software test engineering approaches and tools evolve to appropriately model and formally-verify code behavior in the spirit of what the seL4 project did and maybe further. Similarly, system behavior should also be formally-specified and provable. Until we get there, it's one removed from whipping up untested code ad-hoc and then YOLO'ing to hope it all works out. It's going to take a lot more work and a new way of defining constraints, relationships, and nonvariants but it's unavoidable to prove that code behaves as intended. PS: Neither "Just rewrite everything in X where X = Rust", "just use fuzzing", or "just use MISRA coding standards" doesn't get us there. Holistic improvements help, but not with the fundamental deficiency above.
- weinzierl 2y ago
  The important word is "just", when X facilitates formal verification by a considerable margin.
- MaxBarraclough 2y ago
  That would be a great solution, but full formal verification is a very high bar. The more realistic answer is to use safer languages than C for this sort of critical work. Rust and Ada spring to mind. They could even expose a C ABI/API. From a quick search, it looks like this has already been achieved for TLS, but it sees little real usage. [0] There are also Rust implementations of SSH (Russh and Thrussh), but I don't think they expose a C ABI/API. I'm surprised this rather obvious solution doesn't seem to even receive serious consideration. I'd be surprised if performance was seriously impaired. This blog post [1] found Rustls to be faster than OpenSSL. I couldn't find a similar performance evaluation for Russh or Thrussh. [0] https://github.com/rustls/rustls-ffi https://github.com/rustls/rustls-ffi [1] https://bencher.dev/learn/case-study/rustls/ https://bencher.dev/learn/case-study/rustls/
  - hi-v-rocknroll 2y ago
    Well, the important things that run the world demand rigor. The "it's a hobby" defense rings hollow when decades of slip-shod processes have repeated led to a class of failures that repeated dozens and dozens of times.
codethief 2y ago
> Regardless, I'm not convinced we can defend against this with the way we're currently thinking about operating system design. Amen! On that note: Why is it so so difficult to set up a (rootless) container/sandbox correctly? (I mean, look at what runc does – shit's incredibly complex!) And why is it next to impossible to nest containers without privileges and arcane knowledge of how some of the underlying kernel syscalls work? Even those 250 lines of code to landlock `make` that the author mentions sound awful to me. I don't want to have to set up a sandbox for every single application by hand, let alone set up rules for all things that a malicious application could possibly exploit. Instead, I want security-by-default! Have every application run in a tight sandbox by default and let the application specify what permissions it needs, so that I only need to review those and can grant them as I like. Meanwhile, deny access to everything else! Clearly, we are not (Linux is not) ready for this yet – we lack both a good UI for all of this permission handling and an agreed-upon contract that all application developers can follow. In fact, for the vast majority of applications we don't even really know (have documented) what permissions / access to kernel syscalls they would need, so it'd be incredibly hard to switch to a principle of least privilege-based approach over night. But man, one can always dream…
- bogota 2y ago
  Sounds like IAM for the OS and we all know that just leads to wild card permissions everywhere because the developer doesn’t know what they need
  - Gibbon1 2y ago
    The solution to that is linkers need to generate those permissions and create a manifest section in the elf file so the OS can handle it transparently.
sylware 2y ago
Backdoor, bugdoor or or "convenient" bug... you can spot in the source code... Now, have a look at machine code which is spit out by compilers... and there, right there... no bug or backdoor in the source code though, but yet... how would you call that? compiler-injected-door? To fight this you just need to trust and audit those small pieces of s..oftware which are gcc/clang(llvm).

257 points by benhawkes 2y ago

[dead]

> The "many eyes" theory of open source security isn't popular right now, but it certainly seems like bigger targets have smaller margins for error. https://news.ycombinator.com/item?id=20383529 https://news.ycombinator.com/item?id=20383529 DonHopkins on July 8, 2019 | parent | context | favorite | on: Contributor Agreements Considered Harmful And then there's Linus's Law, which he made up, then tried to blame on Linus. "Given enough eyeballs, all bugs are shallow." -Eric S Raymond "My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt https://en.wikipedia.org/wiki/Linus%27s_Law https://en.wikipedia.org/wiki/Linus%27s_Law >In Facts and Fallacies about Software Engineering, Robert Glass refers to the law as a "mantra" of the open source movement, but calls it a fallacy due to the lack of supporting evidence and because research has indicated that the rate at which additional bugs are uncovered does not scale linearly with the number of reviewers; rather, there is a small maximum number of useful reviewers, between two and four, and additional reviewers above this number uncover bugs at a much lower rate.[4] While closed-source practitioners also promote stringent, independent code analysis during a software project's development, they focus on in-depth review by a few and not primarily the number of "eyeballs".[5][6] >Although detection of even deliberately inserted flaws[7][8] can be attributed to Raymond's claim, the persistence of the Heartbleed security bug in a critical piece of code for two years has been considered as a refutation of Raymond's dictum.[9][10][11][12] Larry Seltzer suspects that the availability of source code may cause some developers and researchers to perform less extensive tests than they would with closed source software, making it easier for bugs to remain.[12] In 2015, the Linux Foundation's executive director Jim Zemlin argued that the complexity of modern software has increased to such levels that specific resource allocation is desirable to improve its security. Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking".[11] Large scale experiments or peer-reviewed surveys to test how well the mantra holds in practice have not been performed. The little experience Raymond DOES have auditing code has been a total fiasco and embarrassing failure, since his understanding of the code was incompetent and deeply tainted by his preconceived political ideology and conspiracy theories about global warming, which was his only motivation for auditing the code in the first place. His sole quest was to discredit the scientists who warned about global warming. The code he found and highlighted was actually COMMENTED OUT, and he never addressed the fact that the scientists were vindicated. http://rationalwiki.org/wiki/Eric_S._Raymond http://rationalwiki.org/wiki/Eric_S._Raymond >During the Climategate fiasco, Raymond's ability to read other peoples' source code (or at least his honesty about it) was called into question when he was caught quote-mining analysis software written by the CRU researchers, presenting a commented-out section of source code used for analyzing counterfactuals as evidence of deliberate data manipulation. When confronted with the fact that scientists as a general rule are scrupulously honest, Raymond claimed it was a case of an "error cascade," a concept that makes sense in computer science and other places where all data goes through a single potential failure point, but in areas where outside data and multiple lines of evidence are used for verification, doesn't entirely make sense. (He was curiously silent when all the researchers involved were exonerated of scientific misconduct.) More context: https://news.ycombinator.com/item?id=20382640 https://news.ycombinator.com/item?id=20382640

toast0 2y ago
> Regarding some of 2014's largest global open source software vulnerabilities, he says, "In these cases, the eyeballs weren't really looking". This makes a lot of sense, because for the most part, you only go looking for bugs when you've run into a problem. Looking for bugs you haven't run into is a lot harder (especially in complex software like OpenSSL), you might get lucky and someone sees a bug while looking for something else, but mostly things go unlooked at until they cause a problem that attracts attention. Even when you pay for a professional audit, things can be missed; but you'll likely get better results for security with organized and focused reviews than by hoping your user base finds everything.
- cbsmith 2y ago
  Large open source projects are regularly subjected to security audits. I think the reality is that closed source software is vulnerable to the same attack, the only difference is fewer eyes to see it and more likely a profit motive will keep those eyes directed in other ways.
poikroequ 2y ago
It's not a complete fallacy. In events like this, after the news hits, there are a flurry of eyeballs looking, at least for a little while. The heartbleed bug got people to look at the openssl code and realize what a mess that code is. Inspectre and meltdown has led to the discovery of many more CPU vulnerabilities. After Chatgpt hit the market, there has been lots of new research on AI security, such as into prompt injection attacks.

davidfiala 2y ago

Even with the best intentions, can a volunteer-driven project like OpenSSH truly guarantee the same level of security as a commercial solution with dedicated resources and a financial stake in preventing backdoors?

chgs 2y ago
Better. Imagine a closed source company with cost pressures employing a random developer who can commit code, perhaps without any peer review, but certainly limited peer review from harried employees. Now imagine why a nation state would want to get staff working in such a company. Now if companies like Microsoft or Amazon or Google want to pay people to work on these open source projects that’s a different thing, and a great thing for them to do given how much they rely on the code.
- wannacboatmovie 2y ago
  Your argument is a model that does no vetting of contributors whatsoever, which resulted in the catastrophe that is the topic of discussion, is better than a hypothetical company which is full of compromised developers that have free reign to commit to the source tree with no oversight? That sounds extremely contrived.
  - adolph 2y ago
    If you are positing that government infiltration of companies is hypothetical and not a real threat, here is an example of compromised corporate staff: https://en.wikipedia.org/wiki/Saudi_infiltration_of_Twitter https://en.wikipedia.org/wiki/Saudi_infiltration_of_Twitter
  - chgs 2y ago
    This wasn’t a contributor to OpenSSH, it was a deep level supply chain attack - something that closed source commercial companies are not immune to. Given how much closed source companies love BSD/apache/etc licenses where they can simply use these low level libraries and charge for stuff on the top I’m not sure how they would be immune from such an attack. The risk from this was highlighted in xkcd back in 2020 https://xkcd.com/2347/ https://xkcd.com/2347/

Vecr 2y ago

Was there ever a writeup of exactly how the XZ exploit worked? I mean exactly, I get the general overview and even quite a few of the specifics, but last time I checked no one had credibly figured out exactly how all the obfuscated components went together.

4llan 2y ago
Gynvael Coldwind made a great analysis about it: https://gynvael.coldwind.pl/?lang=en&id=782 https://gynvael.coldwind.pl/?lang=en&id=782 https://news.ycombinator.com/item?id=39878681 https://news.ycombinator.com/item?id=39878681 xz/liblzma: Bash-stage Obfuscation Explained
- mananaysiempre 2y ago
  That is, as it says in the title, about the Bash-stage obfuscation. That’s fun but it’d also be interesting to know what capabilities the exploit payload actually provided to the attacker. Last I looked into that a month or so ago there were at least two separate endpoints already discovered, and the investigation was still in progress.
  - dangsux 2y ago
    [dead]
kva-gad-fly 2y ago
https://www.openwall.com/lists/oss-security/2024/03/29/4 https://www.openwall.com/lists/oss-security/2024/03/29/4 ?
- Vecr 2y ago
  Yeah, what's posted by you and other users so far is stuff I know, build scripts, injection, obfuscation. I'm more looking for a careful reverse engineering of the actual payload.
  - 2y ago

rwmj 2y ago

> However, it's interesting to note that in both 2002 and 2024 we got a backdoor rather than a bugdoor. As far as we know. Related, there was a pretty interesting backdoor-by-bug attempt on the Linux kernel (at least, one that we know of) back in 2003: https://lwn.net/Articles/57135/ https://lwn.net/Articles/57135/ The Linux "bug" was unsophisticated by modern standards, but you could imagine a modern equivalent that's harder to spot: Make the "bug" happen across several lines of code, especially if some of those lines are part of existing code (so don't appear in the patch being reviewed). Ensure the compiler doesn't warn about it. Make the set of triggering events very unlikely unless you know the attack. It would be very surprising to me if three letter agencies hadn't done or attempted this.

mmsc 2y ago
And in 2010, a similar backdoor appeared in UnrealICRD: https://lwn.net/Articles/392201/ https://lwn.net/Articles/392201/. Also in proftpd: https://www.aldeid.com/wiki/Exploits/proftpd-1.3.3c-backdoor https://www.aldeid.com/wiki/Exploits/proftpd-1.3.3c-backdoor. Both were done by ac1db1tch3z who the author of OP's post, Ben Hawkes, got a shoutout from for another local privilege escalation vulnerability from over a decade ago :-). Anyways, in response to the backdoor in unrealircd, Core Security came up with a "hiding backdoors in plain sight" challenge: https://seclists.org/fulldisclosure/2010/Jul/66 https://seclists.org/fulldisclosure/2010/Jul/66 "Bugdoors" are not new, and I'm sure some have been patched without anybody realizing they were introduced maliciously.
- baby 2y ago
  And there was the socat backdoor
- cqqxo4zV46cp 2y ago
  Oh man! I’d forgotten all about the UnrealIRCd backdoor! I was running an IRC network at the time. What a blast from the past.
fouronnes3 2y ago
To think that there's a safe somewhere in a TLA basement with a "how to get root anywhere" tutorial.

egberts1 2y ago

Ya think that the tests subditectory would contain sufficient "integrity" test cases to ensure that it doesn't fail, but alas ... nooooooo.

tedunangst 2y ago
What does this mean?
saagarjha 2y ago
What exactly would you test here?
- egberts1 2y ago
  Permutation, value range, logic testing ... CODE COVERAGE!
  - saagarjha 2y ago
    None of those would protect against backdoors.

jmakov 2y ago

At this point what makes us think all major contributors are not on a payroll of one or the other state agency? The attack surface of the whole SW supply chain is huge.

deleted 2y ago
[deleted]
Sesse__ 2y ago
All major, and not a single one of them has leaked it?
- Jerrrrrrry 2y ago
  Besides the literal constant torrent of evidence?
  - meepmorp 2y ago
    Since it's a literal constant torrent, it should be easy to pick out some more specific things for the rest of us, please.
    - Jerrrrrrry 2y ago
      "Facebook on Monday joined a lawsuit pressing the Obama administration to allow it to disclose more details of its forced cooperation). Google and Microsoft filed suit in June." Not to mention Twitter files. These are just the companies with enough legal gravitas and social gravity to even mention the canary dying.
tedunangst 2y ago
Sounds like a good social experiment. Work your way up to major contributor for a project, see how long until you're approached by the MIB.

mmh0000 2y ago

Whenever these stories come up, I like to remind people about the Underhanded C Code Contest[1] and the IOCCC[2] TL;DR: Clever C programmers can hide very evil shit in plain sight. [1] https://www.underhanded-c.org/ https://www.underhanded-c.org/ [2] https://www.ioccc.org/ https://www.ioccc.org/

dkga 2y ago
Wow, I was definitely not aware of this.
cortesoft 2y ago
What happened to the underhanded c contest? Seems the last one was in 2016?
- thelastparadise 2y ago
  Maybe because absolutely no one trusts any C code anymore.
- ruk_booze 2y ago
  How sad to see that this is no more. I helped out reviewing the winning contribution[1] by Linus Åkesson, for what I now know was the final contest. Linus, aka ”LFT”, is a remarkable and truly talented person in so many ways. If you haven’t heard of him before, I suggest you check out all of his projects. [1] https://www.linusakesson.net/programming/underhanded/2015.php https://www.linusakesson.net/programming/underhanded/2015.ph...

daghamm 2y ago

Even further back, someone claimed that a three letter agency had paid some developer to introduce backdoor to openbsd (or possibly openssh). Theo did not belive this and publicly disputed these claim and even revealed the name of the whistle-blower. But I have always felt the story rang true and Theo sound not have been so dismissing. Can't find the story, but it should be on the mailing lists somewhere

jmclnx 2y ago
I remember this, it was the FBI. OpenBSD people did a huge audit and nothing was found. That was also like 20 years ago. Also, other articles stated that never happened. Plus, the "backdoor" in OpenSSH was a Linux only thing possibly related to systemd. It never affected OpenBSD. That is because of Linux people patching OpenSSH with "dependency hell". I believe systemd people is doing something about these dependency chains.
- Arch-TK 2y ago
  The "thing to do" about the dependencies is not to have them in the first place. Distributions where patching OpenSSH to add a libsystemd dependency instead of adding 15 lines of code.
- fijiaarone 2y ago
  So you’re saying more than 1 person was paid to put the back door in.
  - bigiain 2y ago
    And at least one of them was intentionally whistleblown to create an OpenBSD witchhunt wasting both OpenBSD developers time and distraction all the other *BSD and Linux security/devs, while the _real_ target slid under the radar...
daneel_w 2y ago
It was talk about a backdoor in OpenBSD's IPsec stack. The software was audited and nothing was found. The person who stepped forward (after being named) claimed on Twitter to have been formerly involved with the FBI and a supposed project of theirs looking into the feasibility of infiltrating the OpenBSD developer sphere in order to plant a backdoor, but that the project never reached planning. Edit: the discussions and auditing of the IPsec stack happened around 2011 if memory serves me right. The supposed backdoor "happened" a decade earlier. The "agent" was named Greg Perry.

TacticalCoder 2y ago

There's another venue for backdoors in most Linux distros. It's insanely complex by design (so that an endless supply of backdoors can be implanted), touches absolutely everything, was required for the XZ backdoor to work, etc. But it you mention its name, you get downvoted.

mfreeman451 2y ago

[dead]

bawolff 2y ago

Must be the only time checksums were actually useful.

udev4096 2y ago

I wonder if every major open source project should adopt the way SQLite operates. No outside contributions at all

imhoguy 2y ago
What is the meaning of "outside"? What if SQLite founder decides to step down. In case of `xz` a fresh insider was a malicious actor who was supposed to help keep up the maintenance. I think open-community projects like Linux Kernel have benefits as they won't die with founder interest waning, still there is more eyes to find issues very early.

gorgoiler 2y ago

In terms of its level of severity (and all round insanity) attacking OpenSSH with a backdoor is as if someone had hoax-mailed a packet labelled “Anthrax” to every tech business in the world. Brazenly stupid and pointlessly broad: the motivation could just as easily have been to cause mass societal disruption (“terrorism”) instead of a targeted attack that just happened to sweep up everyone else in its arms.

hi-v-rocknroll 2y ago

My wish is that FOSS software test engineering approaches and tools evolve to appropriately model and formally-verify code behavior in the spirit of what the seL4 project did and maybe further. Similarly, system behavior should also be formally-specified and provable. Until we get there, it's one removed from whipping up untested code ad-hoc and then YOLO'ing to hope it all works out. It's going to take a lot more work and a new way of defining constraints, relationships, and nonvariants but it's unavoidable to prove that code behaves as intended. PS: Neither "Just rewrite everything in X where X = Rust", "just use fuzzing", or "just use MISRA coding standards" doesn't get us there. Holistic improvements help, but not with the fundamental deficiency above.

weinzierl 2y ago
The important word is "just", when X facilitates formal verification by a considerable margin.
MaxBarraclough 2y ago
That would be a great solution, but full formal verification is a very high bar. The more realistic answer is to use safer languages than C for this sort of critical work. Rust and Ada spring to mind. They could even expose a C ABI/API. From a quick search, it looks like this has already been achieved for TLS, but it sees little real usage. [0] There are also Rust implementations of SSH (Russh and Thrussh), but I don't think they expose a C ABI/API. I'm surprised this rather obvious solution doesn't seem to even receive serious consideration. I'd be surprised if performance was seriously impaired. This blog post [1] found Rustls to be faster than OpenSSL. I couldn't find a similar performance evaluation for Russh or Thrussh. [0] https://github.com/rustls/rustls-ffi https://github.com/rustls/rustls-ffi [1] https://bencher.dev/learn/case-study/rustls/ https://bencher.dev/learn/case-study/rustls/
- hi-v-rocknroll 2y ago
  Well, the important things that run the world demand rigor. The "it's a hobby" defense rings hollow when decades of slip-shod processes have repeated led to a class of failures that repeated dozens and dozens of times.

codethief 2y ago

> Regardless, I'm not convinced we can defend against this with the way we're currently thinking about operating system design. Amen! On that note: Why is it so so difficult to set up a (rootless) container/sandbox correctly? (I mean, look at what runc does – shit's incredibly complex!) And why is it next to impossible to nest containers without privileges and arcane knowledge of how some of the underlying kernel syscalls work? Even those 250 lines of code to landlock `make` that the author mentions sound awful to me. I don't want to have to set up a sandbox for every single application by hand, let alone set up rules for all things that a malicious application could possibly exploit. Instead, I want security-by-default! Have every application run in a tight sandbox by default and let the application specify what permissions it needs, so that I only need to review those and can grant them as I like. Meanwhile, deny access to everything else! Clearly, we are not (Linux is not) ready for this yet – we lack both a good UI for all of this permission handling and an agreed-upon contract that all application developers can follow. In fact, for the vast majority of applications we don't even really know (have documented) what permissions / access to kernel syscalls they would need, so it'd be incredibly hard to switch to a principle of least privilege-based approach over night. But man, one can always dream…

bogota 2y ago
Sounds like IAM for the OS and we all know that just leads to wild card permissions everywhere because the developer doesn’t know what they need
- Gibbon1 2y ago
  The solution to that is linkers need to generate those permissions and create a manifest section in the elf file so the OS can handle it transparently.

sylware 2y ago

Backdoor, bugdoor or or "convenient" bug... you can spot in the source code... Now, have a look at machine code which is spit out by compilers... and there, right there... no bug or backdoor in the source code though, but yet... how would you call that? compiler-injected-door? To fight this you just need to trust and audit those small pieces of s..oftware which are gcc/clang(llvm).