4 ms·

This seems very difficult to defend against. What is a project with a single burnt-out committer to do?

rsc 3y ago
lcamtuf's two posts argue that this may simply not be an open-source maintainer's job to defend against. ("The maintainers of libcolorpicker.so can’t be the only thing that stands between your critical infrastructure and Russian or Chinese intelligence services. Spies are stopped by spies.") That doesn't mean we shouldn't try to help burnt out committers, but the problem seems very hard. As lcamtuf also says, many things don't need maintenance, and just paying people doesn't address what happens when they just don't want to do it anymore. In an alternate universe with different leadership, an organization like the FSF might use donated funds to pay a maintenance staff and an open-source maintainer might be able to lean on them. Of course, that still doesn't address the problem of Jia Tan getting a job with this organization. https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-the-easy https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-t...
- uluyol 3y ago
  There are efforts from industry to try to secure open source, e.g., https://cloud.google.com/security/products/assured-open-source-software https://cloud.google.com/security/products/assured-open-sour... I suspect some variant of this will grow so that some companies, MS/GitHub for example, audit large body of code and vet it for everyone else.
- intunderflow 3y ago
  Why the assumption this is a Russian or Chinese intelligence service? Western governments aren't above this sort of conduct: https://www.mail-archive.com/cryptography@metzdowd.com/msg12325.html https://www.mail-archive.com/cryptography@metzdowd.com/msg12...
  - cb321 3y ago
    Why are people assuming it's any intelligence service/state actor? With cryptocurrency valuations, it would seem like remote rooting gajillions of machines would be highly incentivized for a private person/collective. Not to mention other financial incentives. Our digital infrastructure secures enormous value much of which can be pilfered anonymously. I admit, the op has a "professional/polished vibe" to me as well, but we seem to know very little except for what work time/zones were preferred by the possibly collective/possibly singular human(s) behind the Jia Tan identity. Does anyone have slick linguistic tools to assess if the writer is a single author? Maybe an opportunity to show off.. It's sort of how they caught Ted Kaczynski. It also absolutely makes sense to think of all the state actors (I agree including as you say the US/UK) as part of the ongoing threat model. If the KGB/Ministry of State Security/NSA/MI6 were not doing this before then they surely might in the future. Maybe with more gusto/funding now! They all seem to have an "information dominance at all costs" mentality, at least as agency collectives, whatever individuals inside think.
    - maerF0x0 3y ago
      people often assume state actors are the pinnacle of sophistication, and especially long games. (Also notably Chinese culture is very attuned / trained for long games, relative to American impatience). This was a sophisticated attack, therefore presumption.
      - cb321 3y ago
        Fair enough - I agree that sophistication inspires the presumption, but it's still just that (not that you said otherwise - just emphasizing). Anyway, I've yet to hear of anything in the xz work beyond the ability of 1-3 skilled black hats with a lot of post-COVID time on their hands. The NSA ensuring the Intel ME/AMT could be turned off seems another level of sophistication entirely, for example { a "higher pinnacle" in your nice phrasing :-) }. In terms of sheer numbers, my impression is that the vast majority of attacks blocked at almost every sophistication level are more "criminals" than "states". Admittedly, that may partly be states just acquiring ability to act surgically rather than launching big compromise initiatives like botnets (or otherwise states going undetected). I'm sure it's hard to know. Maybe we're just in a sad era of increasing attack sophistication measured in "Kilo-Sophistic-meters"? The AntiVirus industry has been having a boom lately. It's probably already been mentioned many times, but besides economic & geopolitical incentives, maybe the attacker(s) was a hard core `systemd` or IBM/RedHat hater or one of the people who supposedly issued death threats to Lennart Poettering now at Microsoft or even an open source hater burnt out or wanting to burn the world down like The Joker in Batman. In light of that, Russ' Setting The Stage Prelude could perhaps profitably add the introduction of that lib dependency into `systemd` and also various distros defaulting to `systemd`. Anyway, premature conclusions are to be cautioned against over & over. That's all I was trying to do. { And I'm not claiming you were concluding anything - you seem pretty open-minded about it all. I was always just amplifying parent comments. I absolutely agree long games "feel" more State Actor - part of what I meant by "vibe", but to quote the detective in V For Vendetta - "it's just a feeling". To Old People, 2 years famously doesn't seem as long as to a 25- or 15-year old. ;-) It's actually short on the 5+ year Linux distro maintenance time scales. }
meowface 3y ago
As a burnt out creator of open source projects with thousands of GitHub stars who's received numerous questionable maintainership requests: either very carefully vet people or let it stagnate. I chose the latter and just waited until others forked it, in large part because I didn't want to be responsible for someone hijacking the project to spread malware. If I had ever received a request from a well-known figure with a longstanding reputation, who's appeared in-person at talks and has verifiable employment, I might've been more receptive. But all the requests I got were from random internet identities that easily could've been fabricated, and in any case had no previous reputation. "Jia Tan" and their other sockpuppets very likely are not their real identities.
NarcissistDev 3y ago
The same as what the ant colonies do when one of their own gets infected by the Ophiocordyceps unilateralis fungus.
coldpie 3y ago
It's not a perfect solution, but I think Big Companies could have a role to play here. Some kind of tenure/patronage system, where for every $1B a Big Company makes in profit, they employ one critical open source maintainer with a decent salary (like $200k or something). The job would only have two requirements: 1) don't make everyone so mad that everyone on the Internet asks for you to be fired, and 2) name a suitable replacement when you're ready to move on. The replacement would become an employee at Big Company, which means Big Company would need to do whatever vetting they normally do (background checks, real address to send paychecks and taxes, etc). In this scenario, Jia Tan would not be a suitable replacement, since they don't friggin' exist. Yes, there's problems with this approach. When money gets involved, incentives can get distorted. It limits the pool of acceptable maintainers to those employable by a Big Company. But I think these are solvable problems, especially if there's a strong culture of maintainer independence. It provides a real improvement over the current situation of putting so much pressure on individuals doing good work for no benefit.
- jodrellblank 3y ago
  > "It's not a perfect solution" Is it a solution at all? Say Oracle offer Lasse Collin $200k for maintaining xz but he doesn't want to work for Oracle so he refuses, then what? Amazon offer Lasse $200k but they require fulltime work on other open source packages which he isn't experienced with or interested in, so he refuses, then what? Google employ someone else for $200k but they can't force Lasse Collin to hand over commit rights to xz to Google, or force him to work with a fulltime Google employee pestering with many constant changes trying to justify their job, and they can't force Debian to accept a new Google fork of xz, so then what? And NetFlix, Microsoft, Facebook, Uber, they can't all employ an xz maintainer, xz doesn't need that many people, but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with, how would they catch this kind of subtle multi-pronged long-term attack on some low-attention, slow moving, project? Google already employ a very capable security team who find issues in all kinds of projects and publicise them, they didn't find this one. Is it likely this attack could have made its way into ChromeOS and Android if it wasn't noticed now, or would Google have noticed it? > "1) don't make everyone so mad that everyone on the Internet asks for you to be fired" So it's a sinecure position, doing nothing is the safest thing? > "and 2) name a suitable replacement when you're ready to move on" How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on some of the boring details like the build system and tests and didn't seem to be doing it for the hype of saying "I improved compression by 10%" for their resume? Are they needing to be skilled in hiring and recruitment now?
clnhlzmn 3y ago
This is not Lasse Collin’s responsibility. What is a burnt out committer supposed to do? Absolutely nothing would be fine. Doing exactly what Lasse Collin did and turn over partial control of the project to an apparently helpful contributor with apparent community support is also perfectly reasonable.
caoilte 3y ago
get the project taken over by a foundation eg the Apache Foundation.
2devnull 3y ago
Check the GitHub profile of anybody that commits. Is there a photo of the person? Can you see a commit history and repos that help validate who they seem to be. In this instance, noticing the people emailing to pressure you have fake looking names that start with adjacent letters and the same domain name. Be more paranoid.
- pixl97 3y ago
  > Is there a photo of the person? Does that even matter these days? Especially if we're talking nation state level stuff convincing histories are not hard to create to deflect casual observers. >Be more paranoid. Most people in OSS just want to write some code to do something, not defend the world against evil.
  - 2devnull 3y ago
    “Does it even matter?” Yes, it would have prevented this attack. It isn’t totally sufficient but it’s quick and easy and would have prevented this attack. “Most people don’t want …” I get it. I think the issue is that pushing junk code from malicious contributors into your project causes more hassle in the long run. If you just want to code and make stuff work, you should probably be careful who you pull from. It’s not just for the benefit of others, it’s first and foremost to protect the code base and the time and sanity of other contributors.
    - pixl97 3y ago
      "Sorry, we had to kill open source software because bad people exist" -Microsoft laughing all the way to the bank. The more paranoid walls you put up the more actual contributors getting into the movement say "eh, screw this, who wants to code anyway". This isn't just a problems with OSS, this is a fundamental issue the internet as a whole is experiencing and no one has good answers that don't have terrible trade offs of their own.
- pabs3 3y ago
  Only non-paranoid people have a photo of themselves online.
INTPenis 3y ago
That's a great question and instinctively I'd say better to halt development than cave to pressure.
Vicinity9635 3y ago
[dead]
alickz 3y ago
tell people to fork your project and fuck off
ta1243 3y ago
This is the problem - we have record numbers of computer programmers, having increased exponentially from the 90s, however very few of them want to work just for fun on some library buried deep in the system, most are just out for a quick buck. Look at the number of people who make meaningful commits to ffmpeg vs the number of people that wrap it in some UI and somehow persuade thousands of people that their solution is amazing, and ffmpeg is a sexy project.

rsc 3y ago

lcamtuf's two posts argue that this may simply not be an open-source maintainer's job to defend against. ("The maintainers of libcolorpicker.so can’t be the only thing that stands between your critical infrastructure and Russian or Chinese intelligence services. Spies are stopped by spies.") That doesn't mean we shouldn't try to help burnt out committers, but the problem seems very hard. As lcamtuf also says, many things don't need maintenance, and just paying people doesn't address what happens when they just don't want to do it anymore. In an alternate universe with different leadership, an organization like the FSF might use donated funds to pay a maintenance staff and an open-source maintainer might be able to lean on them. Of course, that still doesn't address the problem of Jia Tan getting a job with this organization. https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-ba... https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-the-easy https://lcamtuf.substack.com/p/oss-backdoors-the-allure-of-t...

uluyol 3y ago
There are efforts from industry to try to secure open source, e.g., https://cloud.google.com/security/products/assured-open-source-software https://cloud.google.com/security/products/assured-open-sour... I suspect some variant of this will grow so that some companies, MS/GitHub for example, audit large body of code and vet it for everyone else.
intunderflow 3y ago
Why the assumption this is a Russian or Chinese intelligence service? Western governments aren't above this sort of conduct: https://www.mail-archive.com/cryptography@metzdowd.com/msg12325.html https://www.mail-archive.com/cryptography@metzdowd.com/msg12...
- cb321 3y ago
  Why are people assuming it's any intelligence service/state actor? With cryptocurrency valuations, it would seem like remote rooting gajillions of machines would be highly incentivized for a private person/collective. Not to mention other financial incentives. Our digital infrastructure secures enormous value much of which can be pilfered anonymously. I admit, the op has a "professional/polished vibe" to me as well, but we seem to know very little except for what work time/zones were preferred by the possibly collective/possibly singular human(s) behind the Jia Tan identity. Does anyone have slick linguistic tools to assess if the writer is a single author? Maybe an opportunity to show off.. It's sort of how they caught Ted Kaczynski. It also absolutely makes sense to think of all the state actors (I agree including as you say the US/UK) as part of the ongoing threat model. If the KGB/Ministry of State Security/NSA/MI6 were not doing this before then they surely might in the future. Maybe with more gusto/funding now! They all seem to have an "information dominance at all costs" mentality, at least as agency collectives, whatever individuals inside think.
  - maerF0x0 3y ago
    people often assume state actors are the pinnacle of sophistication, and especially long games. (Also notably Chinese culture is very attuned / trained for long games, relative to American impatience). This was a sophisticated attack, therefore presumption.
    - cb321 3y ago
      Fair enough - I agree that sophistication inspires the presumption, but it's still just that (not that you said otherwise - just emphasizing). Anyway, I've yet to hear of anything in the xz work beyond the ability of 1-3 skilled black hats with a lot of post-COVID time on their hands. The NSA ensuring the Intel ME/AMT could be turned off seems another level of sophistication entirely, for example { a "higher pinnacle" in your nice phrasing :-) }. In terms of sheer numbers, my impression is that the vast majority of attacks blocked at almost every sophistication level are more "criminals" than "states". Admittedly, that may partly be states just acquiring ability to act surgically rather than launching big compromise initiatives like botnets (or otherwise states going undetected). I'm sure it's hard to know. Maybe we're just in a sad era of increasing attack sophistication measured in "Kilo-Sophistic-meters"? The AntiVirus industry has been having a boom lately. It's probably already been mentioned many times, but besides economic & geopolitical incentives, maybe the attacker(s) was a hard core `systemd` or IBM/RedHat hater or one of the people who supposedly issued death threats to Lennart Poettering now at Microsoft or even an open source hater burnt out or wanting to burn the world down like The Joker in Batman. In light of that, Russ' Setting The Stage Prelude could perhaps profitably add the introduction of that lib dependency into `systemd` and also various distros defaulting to `systemd`. Anyway, premature conclusions are to be cautioned against over & over. That's all I was trying to do. { And I'm not claiming you were concluding anything - you seem pretty open-minded about it all. I was always just amplifying parent comments. I absolutely agree long games "feel" more State Actor - part of what I meant by "vibe", but to quote the detective in V For Vendetta - "it's just a feeling". To Old People, 2 years famously doesn't seem as long as to a 25- or 15-year old. ;-) It's actually short on the 5+ year Linux distro maintenance time scales. }

meowface 3y ago

As a burnt out creator of open source projects with thousands of GitHub stars who's received numerous questionable maintainership requests: either very carefully vet people or let it stagnate. I chose the latter and just waited until others forked it, in large part because I didn't want to be responsible for someone hijacking the project to spread malware. If I had ever received a request from a well-known figure with a longstanding reputation, who's appeared in-person at talks and has verifiable employment, I might've been more receptive. But all the requests I got were from random internet identities that easily could've been fabricated, and in any case had no previous reputation. "Jia Tan" and their other sockpuppets very likely are not their real identities.

NarcissistDev 3y ago

The same as what the ant colonies do when one of their own gets infected by the Ophiocordyceps unilateralis fungus.

coldpie 3y ago

It's not a perfect solution, but I think Big Companies could have a role to play here. Some kind of tenure/patronage system, where for every $1B a Big Company makes in profit, they employ one critical open source maintainer with a decent salary (like $200k or something). The job would only have two requirements: 1) don't make everyone so mad that everyone on the Internet asks for you to be fired, and 2) name a suitable replacement when you're ready to move on. The replacement would become an employee at Big Company, which means Big Company would need to do whatever vetting they normally do (background checks, real address to send paychecks and taxes, etc). In this scenario, Jia Tan would not be a suitable replacement, since they don't friggin' exist. Yes, there's problems with this approach. When money gets involved, incentives can get distorted. It limits the pool of acceptable maintainers to those employable by a Big Company. But I think these are solvable problems, especially if there's a strong culture of maintainer independence. It provides a real improvement over the current situation of putting so much pressure on individuals doing good work for no benefit.

jodrellblank 3y ago
> "It's not a perfect solution" Is it a solution at all? Say Oracle offer Lasse Collin $200k for maintaining xz but he doesn't want to work for Oracle so he refuses, then what? Amazon offer Lasse $200k but they require fulltime work on other open source packages which he isn't experienced with or interested in, so he refuses, then what? Google employ someone else for $200k but they can't force Lasse Collin to hand over commit rights to xz to Google, or force him to work with a fulltime Google employee pestering with many constant changes trying to justify their job, and they can't force Debian to accept a new Google fork of xz, so then what? And NetFlix, Microsoft, Facebook, Uber, they can't all employ an xz maintainer, xz doesn't need that many people, but if they just employ 'open source maintainers' scattering their attention over all kinds of random projects they have no prior experience with, how would they catch this kind of subtle multi-pronged long-term attack on some low-attention, slow moving, project? Google already employ a very capable security team who find issues in all kinds of projects and publicise them, they didn't find this one. Is it likely this attack could have made its way into ChromeOS and Android if it wasn't noticed now, or would Google have noticed it? > "1) don't make everyone so mad that everyone on the Internet asks for you to be fired" So it's a sinecure position, doing nothing is the safest thing? > "and 2) name a suitable replacement when you're ready to move on" How could Lasse Collin have named a more suitable replacement than someone who seemed technically capable, interested in the project, and motivated to work on some of the boring details like the build system and tests and didn't seem to be doing it for the hype of saying "I improved compression by 10%" for their resume? Are they needing to be skilled in hiring and recruitment now?

clnhlzmn 3y ago

This is not Lasse Collin’s responsibility. What is a burnt out committer supposed to do? Absolutely nothing would be fine. Doing exactly what Lasse Collin did and turn over partial control of the project to an apparently helpful contributor with apparent community support is also perfectly reasonable.

caoilte 3y ago

get the project taken over by a foundation eg the Apache Foundation.

2devnull 3y ago

Check the GitHub profile of anybody that commits. Is there a photo of the person? Can you see a commit history and repos that help validate who they seem to be. In this instance, noticing the people emailing to pressure you have fake looking names that start with adjacent letters and the same domain name. Be more paranoid.

pixl97 3y ago
> Is there a photo of the person? Does that even matter these days? Especially if we're talking nation state level stuff convincing histories are not hard to create to deflect casual observers. >Be more paranoid. Most people in OSS just want to write some code to do something, not defend the world against evil.
- 2devnull 3y ago
  “Does it even matter?” Yes, it would have prevented this attack. It isn’t totally sufficient but it’s quick and easy and would have prevented this attack. “Most people don’t want …” I get it. I think the issue is that pushing junk code from malicious contributors into your project causes more hassle in the long run. If you just want to code and make stuff work, you should probably be careful who you pull from. It’s not just for the benefit of others, it’s first and foremost to protect the code base and the time and sanity of other contributors.
  - pixl97 3y ago
    "Sorry, we had to kill open source software because bad people exist" -Microsoft laughing all the way to the bank. The more paranoid walls you put up the more actual contributors getting into the movement say "eh, screw this, who wants to code anyway". This isn't just a problems with OSS, this is a fundamental issue the internet as a whole is experiencing and no one has good answers that don't have terrible trade offs of their own.
pabs3 3y ago
Only non-paranoid people have a photo of themselves online.

INTPenis 3y ago

That's a great question and instinctively I'd say better to halt development than cave to pressure.

Vicinity9635 3y ago

[dead]

alickz 3y ago

tell people to fork your project and fuck off

ta1243 3y ago

This is the problem - we have record numbers of computer programmers, having increased exponentially from the 90s, however very few of them want to work just for fun on some library buried deep in the system, most are just out for a quick buck. Look at the number of people who make meaningful commits to ffmpeg vs the number of people that wrap it in some UI and somehow persuade thousands of people that their solution is amazing, and ffmpeg is a sexy project.