7 ms·

Imagine how frustrating it has to be for the attacker to meticulously plan and execute this and get foiled so late in the game, and so publicly

Alifatisk 3y ago
Must be punching the air right now
- cellis 3y ago
  Or...falling back on less noticed contingency plans...
  - Ekaros 3y ago
    My pet theory is that this was just one project they have been running for years. They are likely doing many more at same time. Slowly inserting parts in various projects and getting their contributors inside the projects.
    - 0cf8612b2e1e 3y ago
      That seems like a safe bet. If you are planning a multi year operation, it would be silly to do it all under a single account. Best to minimize the blast radius if any single exploit gets discovered.
    - guyomes 3y ago
      Some of this activity was actually documented through the alleged data breach of a hacking company [1]. [1]: https://en.wikipedia.org/wiki/Hacking_Team https://en.wikipedia.org/wiki/Hacking_Team
    - SAI_Peregrinus 3y ago
      If it's an intelligence agency exploit, this is nearly certain. Getting agents hired as employees of foreign companies to provide intelligence is an ancient practice. Getting agents to be open source maintainers is a continuation of the same thing.
- noobermin 3y ago
  Or, they must be strapped into a chair having teeth being pulled out by whoever directed them.
  - int_19h 3y ago
    Why would they be? You don't get far with this kind of approach to valuable skilled human resources.
- TechDebtDevin 3y ago
  Likely a team of people at a three letter agency.
  - morkalork 3y ago
    I wonder if they have OKRs too.
  - TheBlight 3y ago
    Everyone keeps saying this but it seems unlikely to me that they'd do this for a relatively short window of opportunity and leave their methods for all to see.
    - avidiax 3y ago
      You are judging this by the outcome, as though it were pre-ordained, and also assuming that this is the only method this agency has. It is much more likely that this backdoor would have gone unnoticed for months or years. The access this backdoor provides would be used only once per system, to install other APT (advanced persistent threats), probably layers of them. Use a typical software RAT or rootkit as the first layer. If that is discovered, fallback to the private keys you stole, or the social engineer the company directory you copied. If that fails, rely on the firmware rootkit that only runs if it's timer hasn't been reset in 6 months. Failing that, re-use this backdoor if it's still available.
      - TheBlight 3y ago
        It was found in a few weeks so why is it more likely it wouldn't have been noticed for months/years with more people running the backdoored version of the code?
        ufo 3y ago
        We were lucky that the backdoor called attention to itself, because it impacted the performance off ssh and introduced valgrind warnings.
pixl97 3y ago
So who's started scanning open source lib test cases for more stuff like this?
- beefnugs 3y ago
  Like everything else in this world, its utterly and completely unfixably broken: There is no way you can have complex huge dependencies properly. The best most paranoid thing is to have multiple networking layers hoping they have to exploit multiple things at once, and whitelist only networking for exactly what you are expecting to happen. (which is completely incompatible with the idea of ssl, we would need a new type of firewall that sits between ALL applications before encryption, like a firewall between applications and the crypto library itself, which breaks a bunch of other things people want to do)
  - int_19h 3y ago
    You can have complex huge dependencies, provided that they can be audited properly. Which means that, for starters, they need to be broken into smaller chunks for which the behavior can be fully specified. They need to be written in languages that are, inasmuch as it is possible, formally verifiable against the spec, even at considerable expense of performance. And where that is not possible, the languages still need to be the kind where any behavior that is not straightforward sticks out like a sore thumb. Then once you do all that, of course, you have one particular version that has been audited and can be used as a dependency. Which means that "ship early, ship often" and "move fast and break things" are completely off the table. In short, it means doing more or less the exact opposite of the trends in the industry for the past 50 years or so. Starting with rewriting everything that we already have. And it will be insanely expensive, of course, because the sheer amount of man-hours needed on all this are massive. Nor can you get away with cheap disposable labor for most of it, since writing specs and formal proofs and doing audits all requires a very skilled labor force.
- slaymaker1907 3y ago
  I think it would help to try and secure signing infrastructure as much as possible. First of all, try to have 3 devs for any given project (this is the most difficult step). Make sure logging into or changing any of the signing stuff requires at least 2 of the devs, one to initiate the JIT and one to monitor. Additionally, take supply chain steps like requiring independent approval for PRs and requiring the signing process to only work with automated builds. Don't allow for signing bits that were built on a dev machine. Finally, I think it would help to start implementing signing executables and scripts which get checked at runtime. Windows obviously does executable signing, but it largely doesn't sign stuff like Python scripts. JS in the browser is kind of signed given that the whole site is signed via https. It's not perfect, but it would help in preventing one program from modifying another for very sensitive contexts like web servers.
zoeysmithe 3y ago
This actor, and others like them, may have dozens or hundreds of these things out there. We don't know. This was only found accidentally, not via code review.

Alifatisk 3y ago

Must be punching the air right now

cellis 3y ago
Or...falling back on less noticed contingency plans...
- Ekaros 3y ago
  My pet theory is that this was just one project they have been running for years. They are likely doing many more at same time. Slowly inserting parts in various projects and getting their contributors inside the projects.
  - 0cf8612b2e1e 3y ago
    That seems like a safe bet. If you are planning a multi year operation, it would be silly to do it all under a single account. Best to minimize the blast radius if any single exploit gets discovered.
  - guyomes 3y ago
    Some of this activity was actually documented through the alleged data breach of a hacking company [1]. [1]: https://en.wikipedia.org/wiki/Hacking_Team https://en.wikipedia.org/wiki/Hacking_Team
  - SAI_Peregrinus 3y ago
    If it's an intelligence agency exploit, this is nearly certain. Getting agents hired as employees of foreign companies to provide intelligence is an ancient practice. Getting agents to be open source maintainers is a continuation of the same thing.
noobermin 3y ago
Or, they must be strapped into a chair having teeth being pulled out by whoever directed them.
- int_19h 3y ago
  Why would they be? You don't get far with this kind of approach to valuable skilled human resources.
TechDebtDevin 3y ago
Likely a team of people at a three letter agency.
- morkalork 3y ago
  I wonder if they have OKRs too.
- TheBlight 3y ago
  Everyone keeps saying this but it seems unlikely to me that they'd do this for a relatively short window of opportunity and leave their methods for all to see.
  - avidiax 3y ago
    You are judging this by the outcome, as though it were pre-ordained, and also assuming that this is the only method this agency has. It is much more likely that this backdoor would have gone unnoticed for months or years. The access this backdoor provides would be used only once per system, to install other APT (advanced persistent threats), probably layers of them. Use a typical software RAT or rootkit as the first layer. If that is discovered, fallback to the private keys you stole, or the social engineer the company directory you copied. If that fails, rely on the firmware rootkit that only runs if it's timer hasn't been reset in 6 months. Failing that, re-use this backdoor if it's still available.
    - TheBlight 3y ago
      It was found in a few weeks so why is it more likely it wouldn't have been noticed for months/years with more people running the backdoored version of the code?
      - ufo 3y ago
        We were lucky that the backdoor called attention to itself, because it impacted the performance off ssh and introduced valgrind warnings.

pixl97 3y ago

So who's started scanning open source lib test cases for more stuff like this?

beefnugs 3y ago
Like everything else in this world, its utterly and completely unfixably broken: There is no way you can have complex huge dependencies properly. The best most paranoid thing is to have multiple networking layers hoping they have to exploit multiple things at once, and whitelist only networking for exactly what you are expecting to happen. (which is completely incompatible with the idea of ssl, we would need a new type of firewall that sits between ALL applications before encryption, like a firewall between applications and the crypto library itself, which breaks a bunch of other things people want to do)
- int_19h 3y ago
  You can have complex huge dependencies, provided that they can be audited properly. Which means that, for starters, they need to be broken into smaller chunks for which the behavior can be fully specified. They need to be written in languages that are, inasmuch as it is possible, formally verifiable against the spec, even at considerable expense of performance. And where that is not possible, the languages still need to be the kind where any behavior that is not straightforward sticks out like a sore thumb. Then once you do all that, of course, you have one particular version that has been audited and can be used as a dependency. Which means that "ship early, ship often" and "move fast and break things" are completely off the table. In short, it means doing more or less the exact opposite of the trends in the industry for the past 50 years or so. Starting with rewriting everything that we already have. And it will be insanely expensive, of course, because the sheer amount of man-hours needed on all this are massive. Nor can you get away with cheap disposable labor for most of it, since writing specs and formal proofs and doing audits all requires a very skilled labor force.
slaymaker1907 3y ago
I think it would help to try and secure signing infrastructure as much as possible. First of all, try to have 3 devs for any given project (this is the most difficult step). Make sure logging into or changing any of the signing stuff requires at least 2 of the devs, one to initiate the JIT and one to monitor. Additionally, take supply chain steps like requiring independent approval for PRs and requiring the signing process to only work with automated builds. Don't allow for signing bits that were built on a dev machine. Finally, I think it would help to start implementing signing executables and scripts which get checked at runtime. Windows obviously does executable signing, but it largely doesn't sign stuff like Python scripts. JS in the browser is kind of signed given that the whole site is signed via https. It's not perfect, but it would help in preventing one program from modifying another for very sensitive contexts like web servers.

zoeysmithe 3y ago

This actor, and others like them, may have dozens or hundreds of these things out there. We don't know. This was only found accidentally, not via code review.