3 ms·

> The problem with xz wasn't a small community; it was no community. A single malicious actor got control of the project, and there was little oversight from an

by zeven7 3y ago

> The problem with xz wasn't a small community; it was no community. A single malicious actor got control of the project, and there was little oversight from anyone else. So because of this a lot of other highly used software was importing and depending on unreviewed code. It's scary to think how common this is. The attack surface seems unmanageable. There need to be tighter policies around what dependencies are included, ensuring that they meet some kind of standard.

colejohnson66 3y ago
As always, there's a relevant XKCD: #2347.[0] It's frightening how much of modern infrastructure depends on vulnerable systems. [0]: https://xkcd.com/2347/ https://xkcd.com/2347/
eacapeisfutuile 3y ago
Does that mean we don’t look into changes in dependencies at all when bumping them?
imiric 3y ago
> There need to be tighter policies around what dependencies are included, ensuring that they meet some kind of standard. This is why it's a good practice to minimize the amount of dependencies, and add dependencies only when absolutely required. Taking this a step further, doing a cursory review of each dependency, seeing the transitive dependencies it introduces, are also beneficial. Of course, it's impractical to do this for the entire dependency tree, and at some point we have to trust that the projects we depend on follow this same methodology, but having a lax attitude about dependency management is part of the problem that caused the xz situation. One thing that I think would improve this are "maintenance scores". A service that would scan projects on GitHub and elsewhere, and assign a score to each project that indicates how well maintained it is. It would take into account the number of contributors in the past N months, development activity, community size and interaction, etc. Projects could showcase this in a badge in their READMEs, and it could be integrated in package managers and IDEs that could warn users if adding a dependency that has a low maintenance score. Hopefully this would disuade people to use poorly maintained projects, and encourage them to use better maintained ones, or avoid the dependency altogether. It would also encourage maintainers to improve their score, and there would be higher visibility of projects that are struggling, but have a high user base, as potentially more vulnerable to this type of attack. And then we can work towards figuring out how to provide the help and resources they need to improve. Does such a service/concept exist already? I think GitHub should introduce something like this, since they have all the data to power it.
- braiamp 3y ago
  Your idea sounds nice in theory, but it's absolutely not worth the amount of effort. To put it in perspective, think about xz case, and how the amount of contributions would have prevented the release artifact (tar file) from being modified? Because other people would have used the tar file? Why? The only ones that use tarfiles are the ones that would be redistributing the code, they will not audit it. The ones that could audit it would look at the version system repository, not at the tar files. In other words, your solution wouldn't even be effective at potentially discovering this issue. The only thing that would effectively do this, is that people stop trusting build artifacts and instead use direct from public repositories packaging. You could figure out if someone maliciously modified the release artifact by comparing it against the tagged version, but at that point, why not just shallow clone the entire thing and be done.
- anonymous-panda 3y ago
  That’s not an effective idea for the same reason that lines of code is not a good measure of productivity. It’s an easy measure to automate but it’s purely performative as it doesn’t score the qualitative value of any of the maintenance work. At best it encourages you to use only popular projects which is its own danger (software monoculture is cheaper to attack) without actually resolving the danger - this attack is reasonably sophisticated and underhanded that could be slipped through almost any code review. One real issue is that xz’s build system is so complicated that it’s possible to slip things in which is an indication that the traditional autoconf Linux build mechanism needs to be retired and banned from distros. But even that’s not enough because an attack only needs to succeed once. The advice to minimize your dependencies is an impractical one in a lot of cases (clearly) and not in your full control as you may acquire a surprising dependency due to transitiveness. And updating your dependencies is a best practice which in this case actually introduces the problem. We need to focus on real ways to improve the supply chain. eg having repeatable idempotent builds with signed chain of trusts that are backed by real identities that can be prosecuted and burned. For example, it would be pretty effective counter incentive for talent if we could permanently ban this person from ever working on lots of projects. That’s typically how humans deal with members of a community who misbehave and we don’t have a good digital equivalent for software development. Of course that’s also dangerous as blackball environments tend to become weaponized.
cjbprime 3y ago
Even if you mandated two code reviewers per merge, the attacker can just have three fake personas backed by the same single human and use them to author and approve malware.
- Deathcrow 3y ago
  Also, in a more optimistic scenario without sockpuppets, it's unlikely that malicious and underhanded contributions will be caught by anyone that isn't a security researcher.

by zeven7 3y ago

colejohnson66 3y ago

As always, there's a relevant XKCD: #2347.[0] It's frightening how much of modern infrastructure depends on vulnerable systems. [0]: https://xkcd.com/2347/ https://xkcd.com/2347/

eacapeisfutuile 3y ago

Does that mean we don’t look into changes in dependencies at all when bumping them?

imiric 3y ago

> There need to be tighter policies around what dependencies are included, ensuring that they meet some kind of standard. This is why it's a good practice to minimize the amount of dependencies, and add dependencies only when absolutely required. Taking this a step further, doing a cursory review of each dependency, seeing the transitive dependencies it introduces, are also beneficial. Of course, it's impractical to do this for the entire dependency tree, and at some point we have to trust that the projects we depend on follow this same methodology, but having a lax attitude about dependency management is part of the problem that caused the xz situation. One thing that I think would improve this are "maintenance scores". A service that would scan projects on GitHub and elsewhere, and assign a score to each project that indicates how well maintained it is. It would take into account the number of contributors in the past N months, development activity, community size and interaction, etc. Projects could showcase this in a badge in their READMEs, and it could be integrated in package managers and IDEs that could warn users if adding a dependency that has a low maintenance score. Hopefully this would disuade people to use poorly maintained projects, and encourage them to use better maintained ones, or avoid the dependency altogether. It would also encourage maintainers to improve their score, and there would be higher visibility of projects that are struggling, but have a high user base, as potentially more vulnerable to this type of attack. And then we can work towards figuring out how to provide the help and resources they need to improve. Does such a service/concept exist already? I think GitHub should introduce something like this, since they have all the data to power it.

braiamp 3y ago
Your idea sounds nice in theory, but it's absolutely not worth the amount of effort. To put it in perspective, think about xz case, and how the amount of contributions would have prevented the release artifact (tar file) from being modified? Because other people would have used the tar file? Why? The only ones that use tarfiles are the ones that would be redistributing the code, they will not audit it. The ones that could audit it would look at the version system repository, not at the tar files. In other words, your solution wouldn't even be effective at potentially discovering this issue. The only thing that would effectively do this, is that people stop trusting build artifacts and instead use direct from public repositories packaging. You could figure out if someone maliciously modified the release artifact by comparing it against the tagged version, but at that point, why not just shallow clone the entire thing and be done.
anonymous-panda 3y ago
That’s not an effective idea for the same reason that lines of code is not a good measure of productivity. It’s an easy measure to automate but it’s purely performative as it doesn’t score the qualitative value of any of the maintenance work. At best it encourages you to use only popular projects which is its own danger (software monoculture is cheaper to attack) without actually resolving the danger - this attack is reasonably sophisticated and underhanded that could be slipped through almost any code review. One real issue is that xz’s build system is so complicated that it’s possible to slip things in which is an indication that the traditional autoconf Linux build mechanism needs to be retired and banned from distros. But even that’s not enough because an attack only needs to succeed once. The advice to minimize your dependencies is an impractical one in a lot of cases (clearly) and not in your full control as you may acquire a surprising dependency due to transitiveness. And updating your dependencies is a best practice which in this case actually introduces the problem. We need to focus on real ways to improve the supply chain. eg having repeatable idempotent builds with signed chain of trusts that are backed by real identities that can be prosecuted and burned. For example, it would be pretty effective counter incentive for talent if we could permanently ban this person from ever working on lots of projects. That’s typically how humans deal with members of a community who misbehave and we don’t have a good digital equivalent for software development. Of course that’s also dangerous as blackball environments tend to become weaponized.

cjbprime 3y ago

Even if you mandated two code reviewers per merge, the attacker can just have three fake personas backed by the same single human and use them to author and approve malware.

Deathcrow 3y ago
Also, in a more optimistic scenario without sockpuppets, it's unlikely that malicious and underhanded contributions will be caught by anyone that isn't a security researcher.