5 ms·

At least for some comic relief I'd like to imagine Jia's boss slapping him and saying something like "you idiot, we worked on this for so many years and you cou

by rdtsc 3y ago

At least for some comic relief I'd like to imagine Jia's boss slapping him and saying something like "you idiot, we worked on this for so many years and you couldn't have checked for any perf issues?" But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show.

chris_wot 3y ago
Surely this is something the FBI should be involved with? Or some authority?
- colinsane 3y ago
  sure. what makes you think they aren't?
  - sitkack 3y ago
    Why would the FBI investigate the NSA? We have zero idea who the actors involved are.
    - colinsane 3y ago
      my comment allows for the NSA to be involved in this and for the FBI to not be investigating them.
    - pxc 3y ago
      It's not actually unusual for three-letter US agencies to be at odds with one another. But one possible reason is if the FBI is convinced that something the NSA is doing is illegal. They may not always be inclined to tolerate that.
      - sitkack 3y ago
        You would have to agree that it was possible for the government to break the law. And what the repercussions are when that happens.
      - denton-scratch 3y ago
        > It's not actually unusual for three-letter US agencies to be at odds with one another. I'd noticed that; this seems to have been the case for a long time. You'd think that having state security agencies at war with one-another would be a disaster, but perhaps it's a feature: a sort of social "layered security". At any rate, it seems much better than having a bunch of state security agencies that all sing from the same songsheet.
        int_19h 3y ago
        It's a bog standard practice, actually, even if you look very far back to the ancient world. Having a single agency responsible for security of yourself and what you own is a bad idea because no matter how much you try ensure the loyalty of people in it, it's prone to, at the minimum, suppressing its own failures and magnifying its successes to make itself look better than it actually is, giving you a false sense of security. It is also the natural point from which to orchestrate a coup, which is something that can be used by your adversaries, but even without their involvement people working there eventually realize that they hold all the keys to the kingdom and there's little risk in them just taking over. So rulers in all ages tended to create multiple different security apparatuses for themselves and their states, and often actively encouraged rivalries between them, even if that makes them less efficient.
- t0mas88 3y ago
  Probably the FBI for the public part of it, but if this wasn't a US owned operation you can be sure the CIA/NSA/military will do their own investigation.
djao 3y ago
Maybe they didn't have time to test? They could have been scrambling to make it into timed releases such as Ubuntu 24.04 or Fedora 40.
- raggi 3y ago
  There is one possible time pressure involved, which is that libsystemd dropped the liblzma dependency
  - XorNot 3y ago
    Absolutely no intelligence agency would look at a successful compromise where they have a highly positioned agent in an organization like this, and burn them trying to rush an under-developed exploit in that would then become not useful almost immediately (because the liblzma dependency would be dropped next distro upgrade cycle). If you had a human-asset with decision making authority and trust in place, then as funded organization with regular working hours, you'd simply can the project and start prototyping new potential uses.
    - refulgentis 3y ago
      No true Scotsman
    - nequo 3y ago
      Might a time-sensitive high-priority goal override such reasoning? For example, the US presidential election is coming up. Making it into Ubuntu LTS could be worth the risk if valuable government targets are running that.
      - kijin 3y ago
        Jia Tan tried to get his backdoored XZ into Ubuntu 24.04 just before the freeze, so that makes sense. Now is about the right time to get it into Fedora if he wants to backdoor RHEL 10, too. But I don't think valuable government targets are in any hurry to upgrade. I wouldn't expect widespread adoption of 24.04, even in the private sector, until well after the U.S. election. By the next election, though, everyone will be running it. Edit: According to another comment [1], there would only have been a short window of vulnerability during which this attack would have worked, due to changes in systemd. This might have increased pressure on the attacker to act quickly. [1] https://news.ycombinator.com/item?id=39881465 https://news.ycombinator.com/item?id=39881465
vbezhenar 3y ago
> But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show. Think about backdoors that are already present and will never be found out.
josephg 3y ago
This was the backdoor we found. We found the backdoor with performance issues. Whats more likely - that this is the only backdoor like this in linux, or that there are more out there and this is the one we happened to find? I really hope someone is out there testing for all of this stuff in linux: - Look for system() calls in compiled binaries and check all of them - Look for uses of IFUNC - specifically when a library uses IFUNC to replace other functions in the resulting executable - Make a list of all the binaries / libraries which don't landlock. Grep the sourcecode of all those projects and make sure none of them expect to be using landlock.
- saagarjha 3y ago
  All of this was obfuscated. None of this will be detectable with current static analysis techniques.
  - josephg 3y ago
    IFUNC and landlock could be debugged pretty easily at runtime, just by adding some instrumentation.
    - saagarjha 3y ago
      Yes but that requires you to know that someone will use it beforehand.
      - josephg 3y ago
        Now that we know IFUNC can me misused like this, it would be pretty silly if we allow other, future exploits to use the same trick.
        saagarjha 3y ago
        ifunc was only used because it’s an obscure feature that is little-used and provides a way to convert a backdoor into easy execution. There are many others and it would be silly to try to catch them all.

by rdtsc 3y ago

chris_wot 3y ago

Surely this is something the FBI should be involved with? Or some authority?

colinsane 3y ago
sure. what makes you think they aren't?
- sitkack 3y ago
  Why would the FBI investigate the NSA? We have zero idea who the actors involved are.
  - colinsane 3y ago
    my comment allows for the NSA to be involved in this and for the FBI to not be investigating them.
  - pxc 3y ago
    It's not actually unusual for three-letter US agencies to be at odds with one another. But one possible reason is if the FBI is convinced that something the NSA is doing is illegal. They may not always be inclined to tolerate that.
    - sitkack 3y ago
      You would have to agree that it was possible for the government to break the law. And what the repercussions are when that happens.
    - denton-scratch 3y ago
      > It's not actually unusual for three-letter US agencies to be at odds with one another. I'd noticed that; this seems to have been the case for a long time. You'd think that having state security agencies at war with one-another would be a disaster, but perhaps it's a feature: a sort of social "layered security". At any rate, it seems much better than having a bunch of state security agencies that all sing from the same songsheet.
      - int_19h 3y ago
        It's a bog standard practice, actually, even if you look very far back to the ancient world. Having a single agency responsible for security of yourself and what you own is a bad idea because no matter how much you try ensure the loyalty of people in it, it's prone to, at the minimum, suppressing its own failures and magnifying its successes to make itself look better than it actually is, giving you a false sense of security. It is also the natural point from which to orchestrate a coup, which is something that can be used by your adversaries, but even without their involvement people working there eventually realize that they hold all the keys to the kingdom and there's little risk in them just taking over. So rulers in all ages tended to create multiple different security apparatuses for themselves and their states, and often actively encouraged rivalries between them, even if that makes them less efficient.
t0mas88 3y ago
Probably the FBI for the public part of it, but if this wasn't a US owned operation you can be sure the CIA/NSA/military will do their own investigation.

djao 3y ago

Maybe they didn't have time to test? They could have been scrambling to make it into timed releases such as Ubuntu 24.04 or Fedora 40.

raggi 3y ago
There is one possible time pressure involved, which is that libsystemd dropped the liblzma dependency
- XorNot 3y ago
  Absolutely no intelligence agency would look at a successful compromise where they have a highly positioned agent in an organization like this, and burn them trying to rush an under-developed exploit in that would then become not useful almost immediately (because the liblzma dependency would be dropped next distro upgrade cycle). If you had a human-asset with decision making authority and trust in place, then as funded organization with regular working hours, you'd simply can the project and start prototyping new potential uses.
  - refulgentis 3y ago
    No true Scotsman
  - nequo 3y ago
    Might a time-sensitive high-priority goal override such reasoning? For example, the US presidential election is coming up. Making it into Ubuntu LTS could be worth the risk if valuable government targets are running that.
    - kijin 3y ago
      Jia Tan tried to get his backdoored XZ into Ubuntu 24.04 just before the freeze, so that makes sense. Now is about the right time to get it into Fedora if he wants to backdoor RHEL 10, too. But I don't think valuable government targets are in any hurry to upgrade. I wouldn't expect widespread adoption of 24.04, even in the private sector, until well after the U.S. election. By the next election, though, everyone will be running it. Edit: According to another comment [1], there would only have been a short window of vulnerability during which this attack would have worked, due to changes in systemd. This might have increased pressure on the attacker to act quickly. [1] https://news.ycombinator.com/item?id=39881465 https://news.ycombinator.com/item?id=39881465

vbezhenar 3y ago

> But seriously, we could have found ourselves with this in all stable repos: RHEL, Debian, Ubuntu, IoT devices 5 years from now and it would have been a much larger shit show. Think about backdoors that are already present and will never be found out.

josephg 3y ago

This was the backdoor we found. We found the backdoor with performance issues. Whats more likely - that this is the only backdoor like this in linux, or that there are more out there and this is the one we happened to find? I really hope someone is out there testing for all of this stuff in linux: - Look for system() calls in compiled binaries and check all of them - Look for uses of IFUNC - specifically when a library uses IFUNC to replace other functions in the resulting executable - Make a list of all the binaries / libraries which don't landlock. Grep the sourcecode of all those projects and make sure none of them expect to be using landlock.

saagarjha 3y ago
All of this was obfuscated. None of this will be detectable with current static analysis techniques.
- josephg 3y ago
  IFUNC and landlock could be debugged pretty easily at runtime, just by adding some instrumentation.
  - saagarjha 3y ago
    Yes but that requires you to know that someone will use it beforehand.
    - josephg 3y ago
      Now that we know IFUNC can me misused like this, it would be pretty silly if we allow other, future exploits to use the same trick.
      - saagarjha 3y ago
        ifunc was only used because it’s an obscure feature that is little-used and provides a way to convert a backdoor into easy execution. There are many others and it would be silly to try to catch them all.