4 ms·

>Which raises the concerning question of how much more sleeper maintainers there are. Given how easy the infiltration is and how extremely hard to detect it is

>Which raises the concerning question of how much more sleeper maintainers there are. Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. For an intelligence operation it is also extremely cheap, you just need a few knowledgeable developers spending some time each week on the project. The upside being a backdoor into a significant portion of infrastructure, the downside being wasted time. I do not think it is unlikely that in many important open source software projects there are one or two people assign to keep an eye on things. They don't even need to be malicious, just being somewhat trusted contributors is enough. I would be extremely surprised if the NSA hasn't a couple of guys who keep watch on the Linux Kernel.

moritonal 3y ago
The irony is that this would make a oddly effective way of having paid open source devs who for the most part just honestly improve projects. With the massive downside they undermine it at a critical moment.
- rightbyte 3y ago
  Ye then maybe thanks to them finally we'll have The year of the air gapped Linux desktop. I dunno what to do if e.g. Debian gets compromized, as in, I can't trust the collective of maintainers. I assume any Windows machine is backdoored. Trivially proven by forced auto updates. Maybe air gapping some home computer for sensitive data might be a good idea.
- pants2 3y ago
  The ideal situation is having multiple intelligence agencies all working on one project and spotting each others' backdoors, so at the end of the day we just have a really secure and well-maintained project.
david_draco 3y ago
> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. I read it exactly the other way around: the infiltration took years and detecting it was the default with a fuzzer, which had to be disabled for the exploit to succeed. It speaks to the hardening and that hardening should be required more. And of course the precarious roles of maintainers, which have been discussed elsewhere.
- dullcrisp 3y ago
  The exploit could be detected with a fuzzer perhaps. The infiltration seems like it was something that would be very easy for a funded intelligence agency to do, and would be nigh undetectable if they had been subtly introducing bugs rather than shipping a sophisticated backdoor to every Linux distro. You have to assume if this was an intelligence agency they didn’t burn their only agent like this.
  - HankB99 3y ago
    > they didn’t burn their only agent like this. I think I'd characterize that as "their only identity." What's the chance that some number (>1) of actual agents were sharing this identity? If that's the case, I'd extrapolate that to multiple identities. In fact the social engineering to gain the trust of the original maintainer likely involved several identities. I suspect that detectability of intentionally injected bugs would be very low.
- Thorrez 3y ago
  >a fuzzer, which had to be disabled for the exploit to succeed. According to this comment, the fuzzer wouldn't have detected it. It wasn't necessary to disable the fuzzer: >https://news.ycombinator.com/item?id=39911249 https://news.ycombinator.com/item?id=39911249
- geggo98 3y ago
  You are right, it took quite some time. On the other hand, it looks like the legitimate part of contributing to xz was only a part time job for the attacker. The rest of the time, they either worked on the exploits, or in other things, like infiltrating other projects using a different handle. Basically I can imagine the attackers being a well organized group, using work sharing and pipelining. Some members of the group would be preparing exploits, some would infiltrate projects and some would make sure not to get caught. And since infiltrating takes time, they would make sure to have multiple projects in the pipeline, seine in the early contributor stage, some in the social pressure stage, and some in the exploiting stage.
jannes 3y ago
At this point I wouldn’t be surprised if the NSA also has a couple of Microsoft employees on their payroll.
cactusfrog 3y ago
I believe LLMs could be useful here as a component of pre-commit hook.
Gormo 3y ago
> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. This particular case seems to be an example of the exact opposite. It took "Jia Tan" two years to conduct all of the social engineering necessary to get into a position to introduce the backdoor, then upon doing so, was caught almost immediately, with the initial discovery not even coming from a dedicated security researcher, but from a sysadmin who kept digging when he saw unusual performance issues. And the threat actor here deliberately went after the weakest link in the chain. Assuming that the goal was to compromise sshd, the antagonist evidently found it too much of a challenge to attempt to infiltrate OpenSSH itself, so targeted a small-team compression project that only some deployments even link to, and still got caught extremely rapidly.
- constantcrying 3y ago
  I don't think you are looking at it the right way. Two years matter if you are a hobbyist or your goal is to compromise the system for some individual gain. For a nation state actor two years is nothing. Likely the entire attack didn't cost more than a couple of thousand hours of developer time. I would guess it was easily cheaper than 100k in financial terms, that is extremely efficient for an intelligence operation with the upside being access to a large amount of servers. The method by which he got caught also depended largely upon random chance, some major "if's" needed to happen, the performance reduction was an unfortunate side effect from the perspective of the attacker, really a minor mistake exposed him. Even if he were exposed a couple months later the damage would have been enormous, if that version had become a stable part of any major distro for the next few years hundreds of thousands of machines would have been vulnerable. There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.
  - Gormo 3y ago
    > For a nation state actor two years is nothing. Perhaps it is, perhaps not. If they are playing a long game, two years might be a straightforward investment. But the point is that it still took two years either way -- the fact that something is only feasible for big institutional actors implies that it is not in itself "easy". And the kinds of organizations that have the motivation and resources to engage in this kind of long-game infiltration by definition have the resources to influence, infiltrate, and manipulate in a variety of ways. It is absolutely not clear here whether having formal organizations involved in directing the development of xz would have made it easier or harder for whomever was behind this attack. > The method by which he got caught also depended largely upon random chance, Why do you presume it's random chance, and that it wouldn't be better represented by more complex probability models? On the surface, it seems like the chance of overall detection would be reasonably represented as an equivalent of an MTBF calculation, with the number of skilled users capable of detecting the vulnerability through normal use as an input. (Not to say that we have enough data to calculate this probability, or that this necessarily even happens enough to have a statistically significant sample -- it might all be "black swans" -- but there's certainly non-random stochastic inputs into this.) > There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro. There's no guarantee for any of this. All rules can be gamed, all incentives can be manipulated, all organizations can be compromised. There is no perfect solution to security, and no guarantee that any prescriptive solutions based on generalizing from the specifics of one incident would not create more opportunities for attackers rather than fewer.
sherburt3 3y ago
[dead]

moritonal 3y ago

The irony is that this would make a oddly effective way of having paid open source devs who for the most part just honestly improve projects. With the massive downside they undermine it at a critical moment.

rightbyte 3y ago
Ye then maybe thanks to them finally we'll have The year of the air gapped Linux desktop. I dunno what to do if e.g. Debian gets compromized, as in, I can't trust the collective of maintainers. I assume any Windows machine is backdoored. Trivially proven by forced auto updates. Maybe air gapping some home computer for sensitive data might be a good idea.
pants2 3y ago
The ideal situation is having multiple intelligence agencies all working on one project and spotting each others' backdoors, so at the end of the day we just have a really secure and well-maintained project.

david_draco 3y ago

> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. I read it exactly the other way around: the infiltration took years and detecting it was the default with a fuzzer, which had to be disabled for the exploit to succeed. It speaks to the hardening and that hardening should be required more. And of course the precarious roles of maintainers, which have been discussed elsewhere.

dullcrisp 3y ago
The exploit could be detected with a fuzzer perhaps. The infiltration seems like it was something that would be very easy for a funded intelligence agency to do, and would be nigh undetectable if they had been subtly introducing bugs rather than shipping a sophisticated backdoor to every Linux distro. You have to assume if this was an intelligence agency they didn’t burn their only agent like this.
- HankB99 3y ago
  > they didn’t burn their only agent like this. I think I'd characterize that as "their only identity." What's the chance that some number (>1) of actual agents were sharing this identity? If that's the case, I'd extrapolate that to multiple identities. In fact the social engineering to gain the trust of the original maintainer likely involved several identities. I suspect that detectability of intentionally injected bugs would be very low.
Thorrez 3y ago
>a fuzzer, which had to be disabled for the exploit to succeed. According to this comment, the fuzzer wouldn't have detected it. It wasn't necessary to disable the fuzzer: >https://news.ycombinator.com/item?id=39911249 https://news.ycombinator.com/item?id=39911249
geggo98 3y ago
You are right, it took quite some time. On the other hand, it looks like the legitimate part of contributing to xz was only a part time job for the attacker. The rest of the time, they either worked on the exploits, or in other things, like infiltrating other projects using a different handle. Basically I can imagine the attackers being a well organized group, using work sharing and pipelining. Some members of the group would be preparing exploits, some would infiltrate projects and some would make sure not to get caught. And since infiltrating takes time, they would make sure to have multiple projects in the pipeline, seine in the early contributor stage, some in the social pressure stage, and some in the exploiting stage.

jannes 3y ago

At this point I wouldn’t be surprised if the NSA also has a couple of Microsoft employees on their payroll.

cactusfrog 3y ago

I believe LLMs could be useful here as a component of pre-commit hook.

Gormo 3y ago

> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. This particular case seems to be an example of the exact opposite. It took "Jia Tan" two years to conduct all of the social engineering necessary to get into a position to introduce the backdoor, then upon doing so, was caught almost immediately, with the initial discovery not even coming from a dedicated security researcher, but from a sysadmin who kept digging when he saw unusual performance issues. And the threat actor here deliberately went after the weakest link in the chain. Assuming that the goal was to compromise sshd, the antagonist evidently found it too much of a challenge to attempt to infiltrate OpenSSH itself, so targeted a small-team compression project that only some deployments even link to, and still got caught extremely rapidly.

constantcrying 3y ago
I don't think you are looking at it the right way. Two years matter if you are a hobbyist or your goal is to compromise the system for some individual gain. For a nation state actor two years is nothing. Likely the entire attack didn't cost more than a couple of thousand hours of developer time. I would guess it was easily cheaper than 100k in financial terms, that is extremely efficient for an intelligence operation with the upside being access to a large amount of servers. The method by which he got caught also depended largely upon random chance, some major "if's" needed to happen, the performance reduction was an unfortunate side effect from the perspective of the attacker, really a minor mistake exposed him. Even if he were exposed a couple months later the damage would have been enormous, if that version had become a stable part of any major distro for the next few years hundreds of thousands of machines would have been vulnerable. There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.
- Gormo 3y ago
  > For a nation state actor two years is nothing. Perhaps it is, perhaps not. If they are playing a long game, two years might be a straightforward investment. But the point is that it still took two years either way -- the fact that something is only feasible for big institutional actors implies that it is not in itself "easy". And the kinds of organizations that have the motivation and resources to engage in this kind of long-game infiltration by definition have the resources to influence, infiltrate, and manipulate in a variety of ways. It is absolutely not clear here whether having formal organizations involved in directing the development of xz would have made it easier or harder for whomever was behind this attack. > The method by which he got caught also depended largely upon random chance, Why do you presume it's random chance, and that it wouldn't be better represented by more complex probability models? On the surface, it seems like the chance of overall detection would be reasonably represented as an equivalent of an MTBF calculation, with the number of skilled users capable of detecting the vulnerability through normal use as an input. (Not to say that we have enough data to calculate this probability, or that this necessarily even happens enough to have a statistically significant sample -- it might all be "black swans" -- but there's certainly non-random stochastic inputs into this.) > There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro. There's no guarantee for any of this. All rules can be gamed, all incentives can be manipulated, all organizations can be compromised. There is no perfect solution to security, and no guarantee that any prescriptive solutions based on generalizing from the specifics of one incident would not create more opportunities for attackers rather than fewer.

sherburt3 3y ago

[dead]