3 ms·

> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. This particular case seems to be an example of the exact opposite.

by Gormo 3y ago

> Given how easy the infiltration is and how extremely hard to detect it is, likely a lot. This particular case seems to be an example of the exact opposite. It took "Jia Tan" two years to conduct all of the social engineering necessary to get into a position to introduce the backdoor, then upon doing so, was caught almost immediately, with the initial discovery not even coming from a dedicated security researcher, but from a sysadmin who kept digging when he saw unusual performance issues. And the threat actor here deliberately went after the weakest link in the chain. Assuming that the goal was to compromise sshd, the antagonist evidently found it too much of a challenge to attempt to infiltrate OpenSSH itself, so targeted a small-team compression project that only some deployments even link to, and still got caught extremely rapidly.

constantcrying 3y ago
I don't think you are looking at it the right way. Two years matter if you are a hobbyist or your goal is to compromise the system for some individual gain. For a nation state actor two years is nothing. Likely the entire attack didn't cost more than a couple of thousand hours of developer time. I would guess it was easily cheaper than 100k in financial terms, that is extremely efficient for an intelligence operation with the upside being access to a large amount of servers. The method by which he got caught also depended largely upon random chance, some major "if's" needed to happen, the performance reduction was an unfortunate side effect from the perspective of the attacker, really a minor mistake exposed him. Even if he were exposed a couple months later the damage would have been enormous, if that version had become a stable part of any major distro for the next few years hundreds of thousands of machines would have been vulnerable. There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.
- Gormo 3y ago
  > For a nation state actor two years is nothing. Perhaps it is, perhaps not. If they are playing a long game, two years might be a straightforward investment. But the point is that it still took two years either way -- the fact that something is only feasible for big institutional actors implies that it is not in itself "easy". And the kinds of organizations that have the motivation and resources to engage in this kind of long-game infiltration by definition have the resources to influence, infiltrate, and manipulate in a variety of ways. It is absolutely not clear here whether having formal organizations involved in directing the development of xz would have made it easier or harder for whomever was behind this attack. > The method by which he got caught also depended largely upon random chance, Why do you presume it's random chance, and that it wouldn't be better represented by more complex probability models? On the surface, it seems like the chance of overall detection would be reasonably represented as an equivalent of an MTBF calculation, with the number of skilled users capable of detecting the vulnerability through normal use as an input. (Not to say that we have enough data to calculate this probability, or that this necessarily even happens enough to have a statistically significant sample -- it might all be "black swans" -- but there's certainly non-random stochastic inputs into this.) > There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro. There's no guarantee for any of this. All rules can be gamed, all incentives can be manipulated, all organizations can be compromised. There is no perfect solution to security, and no guarantee that any prescriptive solutions based on generalizing from the specifics of one incident would not create more opportunities for attackers rather than fewer.
  - constantcrying 3y ago
    I don't disagree, I am not even sure what exactly you are arguing against. I completely agree with your comment about the detection being the result of iterative low probability chances of detection. My point was that, even if that is the case, there was a major chance that it would have gone into many stable distos or would have been exploited.
    - Gormo 3y ago
      > I don't disagree, I am not even sure what exactly you are arguing against. I am arguing that this incident demonstrated the resilience of the FOSS model to even extremely strategic, long-term compromise attempts. Some antagonist invests years worth of time, effort, and money to introduce this backdoor, and the openness of the entire process allowed a single engineer to unravel the whole thing almost immediately after the backdoor was introduced, and every distro immediately sprung into action and neutralized it entirely. All of the people complaining about the vulnerability of small projects, the pseudonimity of contributors, the need for more institutional involvement, etc. are getting things exactly backwards. Imagine if this were a closed-source project funded and managed by an opaque institution, and the attacker used a different set of social engineering tactics to backdoor the code from within the org. Suppose then that Andreas Freund noticed the exact same behavior and set out to investigate it. How far would he have gotten? This incident validates the "many eyes" concept and is a point in favor of the FOSS model.
kevans91 3y ago
fwiw, characterizing Andres as a sysadmin isn't really the whole picture; he's a postgres developer that conducts benchmarking operations with some frequency (and he's quite good at what he does)... he's perhaps naturally a bit more sensitive to things like the cumulative effect of 500ms or so over a number of sshd invocations.
- Gormo 3y ago
  You're right -- I went back and changed "sysadmin" to "engineer". Either way, though, he was not a dedicated security researcher, and managed to unravel this entire thing upon noticing an anomaly in the course of his regular work.

by Gormo 3y ago

constantcrying 3y ago

I don't think you are looking at it the right way. Two years matter if you are a hobbyist or your goal is to compromise the system for some individual gain. For a nation state actor two years is nothing. Likely the entire attack didn't cost more than a couple of thousand hours of developer time. I would guess it was easily cheaper than 100k in financial terms, that is extremely efficient for an intelligence operation with the upside being access to a large amount of servers. The method by which he got caught also depended largely upon random chance, some major "if's" needed to happen, the performance reduction was an unfortunate side effect from the perspective of the attacker, really a minor mistake exposed him. Even if he were exposed a couple months later the damage would have been enormous, if that version had become a stable part of any major distro for the next few years hundreds of thousands of machines would have been vulnerable. There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro.

Gormo 3y ago
> For a nation state actor two years is nothing. Perhaps it is, perhaps not. If they are playing a long game, two years might be a straightforward investment. But the point is that it still took two years either way -- the fact that something is only feasible for big institutional actors implies that it is not in itself "easy". And the kinds of organizations that have the motivation and resources to engage in this kind of long-game infiltration by definition have the resources to influence, infiltrate, and manipulate in a variety of ways. It is absolutely not clear here whether having formal organizations involved in directing the development of xz would have made it easier or harder for whomever was behind this attack. > The method by which he got caught also depended largely upon random chance, Why do you presume it's random chance, and that it wouldn't be better represented by more complex probability models? On the surface, it seems like the chance of overall detection would be reasonably represented as an equivalent of an MTBF calculation, with the number of skilled users capable of detecting the vulnerability through normal use as an input. (Not to say that we have enough data to calculate this probability, or that this necessarily even happens enough to have a statistically significant sample -- it might all be "black swans" -- but there's certainly non-random stochastic inputs into this.) > There is absolutely no reason to assume that if another attack of this quality happens it will not find it's way into some stable distro. There's no guarantee for any of this. All rules can be gamed, all incentives can be manipulated, all organizations can be compromised. There is no perfect solution to security, and no guarantee that any prescriptive solutions based on generalizing from the specifics of one incident would not create more opportunities for attackers rather than fewer.
- constantcrying 3y ago
  I don't disagree, I am not even sure what exactly you are arguing against. I completely agree with your comment about the detection being the result of iterative low probability chances of detection. My point was that, even if that is the case, there was a major chance that it would have gone into many stable distos or would have been exploited.
  - Gormo 3y ago
    > I don't disagree, I am not even sure what exactly you are arguing against. I am arguing that this incident demonstrated the resilience of the FOSS model to even extremely strategic, long-term compromise attempts. Some antagonist invests years worth of time, effort, and money to introduce this backdoor, and the openness of the entire process allowed a single engineer to unravel the whole thing almost immediately after the backdoor was introduced, and every distro immediately sprung into action and neutralized it entirely. All of the people complaining about the vulnerability of small projects, the pseudonimity of contributors, the need for more institutional involvement, etc. are getting things exactly backwards. Imagine if this were a closed-source project funded and managed by an opaque institution, and the attacker used a different set of social engineering tactics to backdoor the code from within the org. Suppose then that Andreas Freund noticed the exact same behavior and set out to investigate it. How far would he have gotten? This incident validates the "many eyes" concept and is a point in favor of the FOSS model.

kevans91 3y ago

fwiw, characterizing Andres as a sysadmin isn't really the whole picture; he's a postgres developer that conducts benchmarking operations with some frequency (and he's quite good at what he does)... he's perhaps naturally a bit more sensitive to things like the cumulative effect of 500ms or so over a number of sshd invocations.

Gormo 3y ago
You're right -- I went back and changed "sysadmin" to "engineer". Either way, though, he was not a dedicated security researcher, and managed to unravel this entire thing upon noticing an anomaly in the course of his regular work.