6 ms·

As a developer, this amazes me, and it just shows what — to me — feels like a top-tier attack method, is probably only entry-to-mid level complexity for the fol

As a developer, this amazes me, and it just shows what — to me — feels like a top-tier attack method, is probably only entry-to-mid level complexity for the folks working at that stage. Some of the things I see posted here on HN are well above this level, so I'd assume for the right kind of money (or other incentives), this is only the beginning of what's possible. And, if you think of ALL the packages and ALL the many millions of libraries on GitHub, this vector is SO EFFECTIVE, there will be hundreds of cases like it uncovered in the next few months, I am certain of it. I worry about all the con/pro-sumer hardware makers, from Philips Hue to Alexas, from the SumUps to the camera makers, from Netgear to TP-Link. All their products are packed with open-source libraries. And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors.

awesomeMilou 3y ago
> And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors. This rationale baffles me, it feels that the dependency-hell circlejerk crowd is working on making OSS maintainers look even more bad with this scenario. Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install. If you are affected by this mess, I'm sorry to say, but it's your fault. If you are worried about developers of software you use for free, as in free beer, going rogue, either put in incentives for them to not do that (i.e. pay them) or fork the project and implement your own security measures on top of what's already there. If you're worried that you could encounter exploits from dependencies in commercial software you use, you should negotiate a contract that includes compensation from damages from supply chain attacks. If you're unwilling to do that, sorry mate, you're just unprofessional. Inb4: Yes, I am really trying to say that you should check the supply chain of even your most basic dependencies such as SSH.
- finaard 3y ago
  Unfortunately that's "industry standard" nowadays. I lost count how often I had that discussion over the last two decades. Just look at stuff like pip, npm or pretty much any "modern" package manager in use by developers - they're all pretty much designed to pull in a shitload of arbitrary unaudited and in some causes unauditable dependencies. And nobody wants to listen. That's why I prefer to work in heavily regulated areas nowadays - that way I can shorten that discussion with "yeah, but regulatory requirements don't let us do that, sorry" The absolute basic should be having a local archive of dependencies which at least received a basic sanity check, and updates or additions to that should review changes being added. CI gets access to that cache, but by itself does not have network access to make sure no random crap gets pulled into the build. You'd be surprised how many popular build systems can't do that at all, or only with a lot of workarounds.
  - pknopf 3y ago
    Package managers that use git are less prone to this kinda of attack (goland, rust).
    - steveklabnik 3y ago
      Cargo does not “use git.”
      - AnyTimeTraveler 3y ago
        What they meant was probably, that you have the option to rely entirely on using git repositories for your dependencies or even just paths to other projects on your disk. You can also setup your own dependency registry and only work with that.
        steveklabnik 3y ago
        That is possible but it’s not the default and isn’t as nice as the default is.
      - 3836293648 3y ago
- Dunedan 3y ago
  > Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install. That sounds like you assume RedHat would've caught the vulnerability in xz-utils, before shipping it in the next release of RHEL. I'm not so sure about that, as there is only so much you can do in terms of supply chain analysis and such a sophisticated vulnerability can be pretty hard to spot. Also mind that it only got discovered by accident after all.
  - tcmart14 3y ago
    I don't know if RedHat would have caught it. But the benefit of Red Hat is, they would be the one to fall on the sword. Your product is built on RHEL. This happens. You get to shift blame to RHEL and RedHat would eat it. The positive is, after the dust has settled Red Hat could choose to sort of adopt the compromised piece (invest engineering effort and take it over) or take some stewardship (keeping an eye on it and maybe give a hand to whoever is maintaining it after).
    - cqqxo4zV46cp 3y ago
      Sorry, but what? What does “shifting the blame” mean here? Ever heard of “it’s not my fault but it’s my problem?”. It really sounds like you’re speaking from the perspective of a hypothetical employee looking to not get PIPd or whatever. GP is talking about something quite different, and you’ve run off taking some sort of great personal offence to someone dare implying that there are downsides to open source, not even that it’s worse overall, but that there are downsides. Chill.
      - awesomeMilou 3y ago
        They're right though, the benefit of paying absurd amounts to your linux vendor is raking in certs that you can use with your insurance provider to cover your a** in case something like this happens. That's the sole reason of certs after all. Though I'd like to figure out if RedHat really is going to eat it if push comes to shove.
- rlpb 3y ago
  I think it's more than an individual or an organisation. The industry as a whole has favoured not caring about dependencies or where they come from in order to increase velocity. Concerns about supply chain (before we even had the term) were dismissed as "unlikely and we won't be blamed because everyone's doing it"). The organisations that did have some measures were complained about loudly, and they diluted their requirements over time in order to avoid stagnation. Example: Debian used to have a "key must be signed by three other Debian developers" requirement. They had to relax the requirement in part because, from the perspective of the wider ecosystem, nobody else had these onerous requirements and so they seemed unreasonable (although Covid was the final straw). If we'd had an ecosystem-wide culture of "know your upstream maintainer", then this kind of expectation as a condition of maintainership would be normal, we'd have much better tooling to do it, and such requirements would not have seemed onerous to anyone. It's like there's an Overton Window of what is acceptable, that has perhaps shifted too far in favour of velocity and at the cost of security, and this kind of incident is needed to get people to sit up and take notice. This incident provides the ecosystem as a whole the opportunity to consider slowing down in order to improve supply chain security. There's no silver bullet, but there are a variety of measures available to mitigate, such as trying to know the real world identity of maintainers, more cautious code review, banning practices such as binary blobs in source trees, better tooling to roll back, etc. All of these require slowing down velocity in some way. Change can only realistically happen by shifting the Overton Window across the ecosystem as a whole, with everyone accepting the hit to velocity. I think that an individual or organisation within the ecosystem isn't really in a position to stray too far from this Overton Window without becoming ineffective, because of the way that ecosystem elements all depend on each other. > If you're unwilling to do that, sorry mate, you're just unprofessional. There are no professionals doing what you suggest today, because if they did, they'd be out-competed on price immediately. It's too expensive and customers do not care.
deleted 3y ago
[deleted]
bipson 3y ago
That's why I don't see e.g. TP-Link basing their router firmware on OpenWRT as a win, and why I want the "vanilla" upstream project (or something that tracks upstream by design) running on my devices. Applies to all of my devices btw. I don't like Android having to use an old kernel, I didn't like MacOS running some ancient Darwin/BSD thing, etc. The required effort for backporting worries me. Don't get me wrong, I'm not saying OSS has no vulns.
- doubled112 3y ago
  More orgs directly contributing to upstream is best in my eyes too. I'm not against forking, but there are usually real benefits to running the latest version of the most popular one. One opposite of this I've seen is Mikrotik's RouterOS. I'm under the understanding that they usually reimplement software and protocols rather than depending on an upstream. I'd imagine that is what leads to issues such as missing UDP support in OpenVPN for 10 years, and I'm not sure it gives me the warmest fuzzy feeling about security. Pros and cons, I suppose. More secure because it's not the same target as everybody else. Less secure because there are fewer users and eyes looking at this thing.
  - philipwhiuk 3y ago
    > there are usually real benefits to running the latest version of the most popular one. Using the absolute latest version is acting as a beta tester for everyone else and this is not the first case where it means you get absolutely hosed.
0xdeadbeefbabe 3y ago
[dead]
- diego_sandoval 3y ago
  > this isn't that easy to pull off, How much does it cost in money? 400k? 600k? 1M? That's peanuts for any three letter agency from the US, UK, China, etc.
fieldcny 3y ago
Any moderately well run shop will have a mechanism to get updates when a dependency of theirs has a security issues, depending on the line of business it may actually be required by a regulator or certification body (eg PCI etc) We should probably be more afraid of the backdoors you can’t see in proprietary that would almost never be found.
begueradj 3y ago
>, is probably only entry-to-mid level complexity for the folks working at that stage. On the contrary: the developers and maintainers who are more informed than us described it as highly sophisticated attack. I also read early InfoSec (information security) articles which were able to only describe a part of the code, not the whole strategy behind the attack because, again, the attack and code are sophisticated. You can also read early InfoSec articles which describe the attack in different ways simply because it was not that simple to understand. Then I read articles saying something like this: "Finally it seems it's an RCE attack". Of course, now that even a scanner is developed to detect that vulnerability on your server, we can all claim: "Oh that was a so simple and stupid attack, how come no one detected it much earlier ?!"

awesomeMilou 3y ago

> And I am 100% certain that most of their dev teams do not spend time scanning these for obscure injection vectors. This rationale baffles me, it feels that the dependency-hell circlejerk crowd is working on making OSS maintainers look even more bad with this scenario. Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install. If you are affected by this mess, I'm sorry to say, but it's your fault. If you are worried about developers of software you use for free, as in free beer, going rogue, either put in incentives for them to not do that (i.e. pay them) or fork the project and implement your own security measures on top of what's already there. If you're worried that you could encounter exploits from dependencies in commercial software you use, you should negotiate a contract that includes compensation from damages from supply chain attacks. If you're unwilling to do that, sorry mate, you're just unprofessional. Inb4: Yes, I am really trying to say that you should check the supply chain of even your most basic dependencies such as SSH.

finaard 3y ago
Unfortunately that's "industry standard" nowadays. I lost count how often I had that discussion over the last two decades. Just look at stuff like pip, npm or pretty much any "modern" package manager in use by developers - they're all pretty much designed to pull in a shitload of arbitrary unaudited and in some causes unauditable dependencies. And nobody wants to listen. That's why I prefer to work in heavily regulated areas nowadays - that way I can shorten that discussion with "yeah, but regulatory requirements don't let us do that, sorry" The absolute basic should be having a local archive of dependencies which at least received a basic sanity check, and updates or additions to that should review changes being added. CI gets access to that cache, but by itself does not have network access to make sure no random crap gets pulled into the build. You'd be surprised how many popular build systems can't do that at all, or only with a lot of workarounds.
- pknopf 3y ago
  Package managers that use git are less prone to this kinda of attack (goland, rust).
  - steveklabnik 3y ago
    Cargo does not “use git.”
    - AnyTimeTraveler 3y ago
      What they meant was probably, that you have the option to rely entirely on using git repositories for your dependencies or even just paths to other projects on your disk. You can also setup your own dependency registry and only work with that.
      - steveklabnik 3y ago
        That is possible but it’s not the default and isn’t as nice as the default is.
    - 3836293648 3y ago
Dunedan 3y ago
> Any given commercial operation that claims any credibility for itself does supply chain analysis before adopting a dependency. This is, among other things why ordinarily you'd pay RedHat to maintain a stable Linux Release for you and why projects such as FreeBSD severely limit the software they ship in the default install. That sounds like you assume RedHat would've caught the vulnerability in xz-utils, before shipping it in the next release of RHEL. I'm not so sure about that, as there is only so much you can do in terms of supply chain analysis and such a sophisticated vulnerability can be pretty hard to spot. Also mind that it only got discovered by accident after all.
- tcmart14 3y ago
  I don't know if RedHat would have caught it. But the benefit of Red Hat is, they would be the one to fall on the sword. Your product is built on RHEL. This happens. You get to shift blame to RHEL and RedHat would eat it. The positive is, after the dust has settled Red Hat could choose to sort of adopt the compromised piece (invest engineering effort and take it over) or take some stewardship (keeping an eye on it and maybe give a hand to whoever is maintaining it after).
  - cqqxo4zV46cp 3y ago
    Sorry, but what? What does “shifting the blame” mean here? Ever heard of “it’s not my fault but it’s my problem?”. It really sounds like you’re speaking from the perspective of a hypothetical employee looking to not get PIPd or whatever. GP is talking about something quite different, and you’ve run off taking some sort of great personal offence to someone dare implying that there are downsides to open source, not even that it’s worse overall, but that there are downsides. Chill.
    - awesomeMilou 3y ago
      They're right though, the benefit of paying absurd amounts to your linux vendor is raking in certs that you can use with your insurance provider to cover your a** in case something like this happens. That's the sole reason of certs after all. Though I'd like to figure out if RedHat really is going to eat it if push comes to shove.
rlpb 3y ago
I think it's more than an individual or an organisation. The industry as a whole has favoured not caring about dependencies or where they come from in order to increase velocity. Concerns about supply chain (before we even had the term) were dismissed as "unlikely and we won't be blamed because everyone's doing it"). The organisations that did have some measures were complained about loudly, and they diluted their requirements over time in order to avoid stagnation. Example: Debian used to have a "key must be signed by three other Debian developers" requirement. They had to relax the requirement in part because, from the perspective of the wider ecosystem, nobody else had these onerous requirements and so they seemed unreasonable (although Covid was the final straw). If we'd had an ecosystem-wide culture of "know your upstream maintainer", then this kind of expectation as a condition of maintainership would be normal, we'd have much better tooling to do it, and such requirements would not have seemed onerous to anyone. It's like there's an Overton Window of what is acceptable, that has perhaps shifted too far in favour of velocity and at the cost of security, and this kind of incident is needed to get people to sit up and take notice. This incident provides the ecosystem as a whole the opportunity to consider slowing down in order to improve supply chain security. There's no silver bullet, but there are a variety of measures available to mitigate, such as trying to know the real world identity of maintainers, more cautious code review, banning practices such as binary blobs in source trees, better tooling to roll back, etc. All of these require slowing down velocity in some way. Change can only realistically happen by shifting the Overton Window across the ecosystem as a whole, with everyone accepting the hit to velocity. I think that an individual or organisation within the ecosystem isn't really in a position to stray too far from this Overton Window without becoming ineffective, because of the way that ecosystem elements all depend on each other. > If you're unwilling to do that, sorry mate, you're just unprofessional. There are no professionals doing what you suggest today, because if they did, they'd be out-competed on price immediately. It's too expensive and customers do not care.

deleted 3y ago

[deleted]

bipson 3y ago

That's why I don't see e.g. TP-Link basing their router firmware on OpenWRT as a win, and why I want the "vanilla" upstream project (or something that tracks upstream by design) running on my devices. Applies to all of my devices btw. I don't like Android having to use an old kernel, I didn't like MacOS running some ancient Darwin/BSD thing, etc. The required effort for backporting worries me. Don't get me wrong, I'm not saying OSS has no vulns.

doubled112 3y ago
More orgs directly contributing to upstream is best in my eyes too. I'm not against forking, but there are usually real benefits to running the latest version of the most popular one. One opposite of this I've seen is Mikrotik's RouterOS. I'm under the understanding that they usually reimplement software and protocols rather than depending on an upstream. I'd imagine that is what leads to issues such as missing UDP support in OpenVPN for 10 years, and I'm not sure it gives me the warmest fuzzy feeling about security. Pros and cons, I suppose. More secure because it's not the same target as everybody else. Less secure because there are fewer users and eyes looking at this thing.
- philipwhiuk 3y ago
  > there are usually real benefits to running the latest version of the most popular one. Using the absolute latest version is acting as a beta tester for everyone else and this is not the first case where it means you get absolutely hosed.

0xdeadbeefbabe 3y ago

[dead]

diego_sandoval 3y ago
> this isn't that easy to pull off, How much does it cost in money? 400k? 600k? 1M? That's peanuts for any three letter agency from the US, UK, China, etc.

fieldcny 3y ago

Any moderately well run shop will have a mechanism to get updates when a dependency of theirs has a security issues, depending on the line of business it may actually be required by a regulator or certification body (eg PCI etc) We should probably be more afraid of the backdoors you can’t see in proprietary that would almost never be found.

begueradj 3y ago

>, is probably only entry-to-mid level complexity for the folks working at that stage. On the contrary: the developers and maintainers who are more informed than us described it as highly sophisticated attack. I also read early InfoSec (information security) articles which were able to only describe a part of the code, not the whole strategy behind the attack because, again, the attack and code are sophisticated. You can also read early InfoSec articles which describe the attack in different ways simply because it was not that simple to understand. Then I read articles saying something like this: "Finally it seems it's an RCE attack". Of course, now that even a scanner is developed to detect that vulnerability on your server, we can all claim: "Oh that was a so simple and stupid attack, how come no one detected it much earlier ?!"