14 ms·

Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.

ametrau 3y ago
Probably a state actor. You can look far into the future when you’re working for the party.
- DrewRWx 3y ago
  And that long term perspective could be used constructively instead!
- calvinmorrison 3y ago
  Which like, also wouldn't be totally weird if I found out that the xz or whatever library maintainer worked for the DoE as a researcher? I kind of expect governments to be funding this stuff.
  - CanaryLayout 3y ago
    From what I read on masto, the original maint had personal life breakdown, etc. Their interest in staying as primary maint is gone. This is a very strong argument for FOSS to pick up the good habit of ditching/un-mainlining projects where they are sitting around for state actors to volunteer injecting commits to, and dep-stripping active projects from this cruft. Who wants to maintain on a shitty compression format? Someone who is dephunting, it turns out. Okay so your pirate-torrent person needs liblzma.so Offer it in the scary/oldware section of the package library that you need to hunt down the instructions to turn on. Let the users see that it's marked as obsolete, enterprises will see that it should go on the banlist.
    - soraminazuki 3y ago
      Um, what? This incident is turning into such a big deal because xz is deeply ingrained as a core dependency in the software ecosystem. It's not an obscure tool for "pirates."
    - Phenylacetyl 3y ago
      [dead]
    - Bulat_Ziganshin 3y ago
      Collin worked on XZ and its predecessor ~15 years. It seems that he did that for free, at least in recent times. Anyone will lose motivation to work for free over this period of time. At the same time, XZ became a cornerstone of major Linxus distributions, being systemd dependency and loaded, in particular, as part of sshd. What could go wrong? In hindsight, the commercial idea of Red Hat, utilizing the free work of thousands of developers working "just for fun", turned out to be not so brilliant.
      - wannacboatmovie 3y ago
        On the contrary, this is a good example for why 'vulnerable' OSS projects that have become critical components, for which the original developer has abandoned or lost interest, should be turned over to an entity like RedHat who can assign a paid developer. It's important to do this before some cloak and dagger rando steps out of the shadows to offer friendly help, who oh by the way happens to be a cryptography and compression expert. A lot of comments in this thread seem to be missing the forest for the trees: this was a multiyear long operation that targeted a vulnerable developer of a heavily-used project. This was not the work of some lone wolf. The amount of expertise needed and the amount of research and coordination needed to execute this required hundreds of man-hours. The culprits likely had a project manager.... Someone had to stalk out OSS developers to find out who was vulnerable (the xz maintainer had publicly disclosed burnout/mental health issues); then the elaborate trap was set. The few usernames visible on GitHub are like pulling a stubborn weed that pops up in the yard... until you start pulling on it you don't realize the extensive reality lying beneath the surface. The implied goal here was to add a backdoor into production Debian and Red Hat EL. Something that would take years to execute. This was NOT the work of one person.
hangonhn 3y ago
I imagine it might be easier to just compromise a weakly protected account than to actual put in a 2 years long effort with real contributions. If we mandated MFA for all contributors who contribute to these really important projects then we can know with greater certainty if it was really a long con vs. a recently compromised account.
- __s 3y ago
  This seems like a great way to invest in supporting open source projects in meantime if these projects are being used by these actors. Just have to maintain an internal fork without the backdoors Maybe someone can disrupt the open source funding problem by brokering exploit bounties /s
- the8472 3y ago
  github already mandates MFA for members of important projects
  - rvense 3y ago
    Doesn't it mandate it for everyone? I don't use it anymore and haven't logged in since forever, but I think I got a series of e-mails that it was being made mandatory.
    - ryukoposting 3y ago
      It will soon. I think I have to sort it out before April 4. My passwords are already >20 random characters, so I wasn't going to do it until they told me to.
      - pabs3 3y ago
        If you are using pass to store those, check out pass-otp and browserpass, since GitHub still allows TOTP for MFA. pass-otp is based on oathtool, so you can do it more manually too if you don't use pass.
jnxx 3y ago
And, Joey Hess has counted at least 750 commits to xz from that handle. https://hachyderm.io/@joeyh/112180715824680521 https://hachyderm.io/@joeyh/112180715824680521 This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.
- ebfe1 3y ago
  clickhouse has pretty good github_events dataset on playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/ https://ghe.clickhouse.tech/ Example of what this user JiaT75 did so far: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSdKaWFUNzUnIE9SREVSIEJZIGZpbGVfdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT... pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0ZWRfYXQsIGV2ZW50X3R5cGUsIGFjdG9yX2xvZ2luLCByZXBvX25hbWUsdGl0bGUsYWN0aW9uLCBzdWJzdHJpbmcoYm9keSwgMSwgMjAwKSBhcyBib2R5X3RydW5jYXRlZCAgRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGZpbGVfdGltZSA+IG5vdygpIC0gSU5URVJWQUwgNjAgREFZIEFORCBldmVudF90eXBlPSdQdWxsUmVxdWVzdEV2ZW50JyBBTkQgdGl0bGUgSUxJS0UgJyU1LjYlJyBBTkQgdGl0bGUgSUxJS0UgJyV4eiUnIEFORCBib2R5IE5PVCBJTElLRSAnJVNueWslJyBhbmQgYm9keSBub3QgaWxpa2UgJyVDVkUtMjAyNC0zMDk0JScgYW5kIHRpdGxlIG5vdCBpbGlrZSAnJWRvd25ncmFkZSUnIE9SREVSIEJZIGZpbGVfdGltZSBERVNDIExJTUlUIDEwMA== https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...
  - palijer 3y ago
    Yeah. It would be interesting to see who adopted to the compromised versions and how quickly, compared to how quickly they normally adopt new versions (not bots pulling upgrades, but how quickly maintainers approve and merge them) If there were a bunch of people who adopted it abnormally fast compared to usual, might point to there being more "bad actors" in this operation (said at the risk of sounding paranoid if this turns out to be a state run thing)
- waynesonfire 3y ago
  750 commits... is xz able to send e-mails yet?
moritonal 3y ago
Warning, drunk brain talking. But a LLM driven email based "collaborator" could play a very long gMw adding basic features to a code made whilst earning trust backed by a generated online presence. My money is on a resurgance in the Web of Trust.
- w4ffl35 3y ago
  Clearly a human is even better at it.
- llmblockchain 3y ago
  State level actor? China?
  - logankeenan 3y ago
    You're likely being downvoted because the Github profile looking like east Asian isn't evidence of where the attacker/attackers are from. Nation states will go to long lengths to disguise their identity. Using broken Russian English when they are not Russian, putting comments in the code of another language, and all sorts of other things to create misdirection.
    - llmblockchain 3y ago
      That's certainly true-- at the very least it "seems" like Asian, but it could very well be from any nation. If they were patient enough to work up to this point they would likely not be dumb enough to leak such information.
    - account42 3y ago
      Otoh with these kind of kneejerk "antiracist" reactions the best way to disguise a chinese operative becomes to prentend you're chinese.
- jnxx 3y ago
  The web of trust is a really nice idea, but it works badly against that kind of attacks. Just consider that in the real world, most living people (all eight billions) are linked by only six degrees of separation. It really works, for code and for trusted social relations (like "I lend you 100 bucks and you pay me them back when you get your salary") mostly when you know the code author in person. This is also not a new insight. In the beginning of the naughties, there was a web site named kuro5hin.org, which experiemented with user ratings and trust networks. It turned out impossible to prevent take-overs.
lucasRW 3y ago
More likely that the account of that dev was breawched, dont you think ?
dang 3y ago
(I detached this subthread from https://news.ycombinator.com/item?id=39866275 https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)

ametrau 3y ago

Probably a state actor. You can look far into the future when you’re working for the party.

DrewRWx 3y ago
And that long term perspective could be used constructively instead!
calvinmorrison 3y ago
Which like, also wouldn't be totally weird if I found out that the xz or whatever library maintainer worked for the DoE as a researcher? I kind of expect governments to be funding this stuff.
- CanaryLayout 3y ago
  From what I read on masto, the original maint had personal life breakdown, etc. Their interest in staying as primary maint is gone. This is a very strong argument for FOSS to pick up the good habit of ditching/un-mainlining projects where they are sitting around for state actors to volunteer injecting commits to, and dep-stripping active projects from this cruft. Who wants to maintain on a shitty compression format? Someone who is dephunting, it turns out. Okay so your pirate-torrent person needs liblzma.so Offer it in the scary/oldware section of the package library that you need to hunt down the instructions to turn on. Let the users see that it's marked as obsolete, enterprises will see that it should go on the banlist.
  - soraminazuki 3y ago
    Um, what? This incident is turning into such a big deal because xz is deeply ingrained as a core dependency in the software ecosystem. It's not an obscure tool for "pirates."
  - Phenylacetyl 3y ago
    [dead]
  - Bulat_Ziganshin 3y ago
    Collin worked on XZ and its predecessor ~15 years. It seems that he did that for free, at least in recent times. Anyone will lose motivation to work for free over this period of time. At the same time, XZ became a cornerstone of major Linxus distributions, being systemd dependency and loaded, in particular, as part of sshd. What could go wrong? In hindsight, the commercial idea of Red Hat, utilizing the free work of thousands of developers working "just for fun", turned out to be not so brilliant.
    - wannacboatmovie 3y ago
      On the contrary, this is a good example for why 'vulnerable' OSS projects that have become critical components, for which the original developer has abandoned or lost interest, should be turned over to an entity like RedHat who can assign a paid developer. It's important to do this before some cloak and dagger rando steps out of the shadows to offer friendly help, who oh by the way happens to be a cryptography and compression expert. A lot of comments in this thread seem to be missing the forest for the trees: this was a multiyear long operation that targeted a vulnerable developer of a heavily-used project. This was not the work of some lone wolf. The amount of expertise needed and the amount of research and coordination needed to execute this required hundreds of man-hours. The culprits likely had a project manager.... Someone had to stalk out OSS developers to find out who was vulnerable (the xz maintainer had publicly disclosed burnout/mental health issues); then the elaborate trap was set. The few usernames visible on GitHub are like pulling a stubborn weed that pops up in the yard... until you start pulling on it you don't realize the extensive reality lying beneath the surface. The implied goal here was to add a backdoor into production Debian and Red Hat EL. Something that would take years to execute. This was NOT the work of one person.

hangonhn 3y ago

I imagine it might be easier to just compromise a weakly protected account than to actual put in a 2 years long effort with real contributions. If we mandated MFA for all contributors who contribute to these really important projects then we can know with greater certainty if it was really a long con vs. a recently compromised account.

__s 3y ago
This seems like a great way to invest in supporting open source projects in meantime if these projects are being used by these actors. Just have to maintain an internal fork without the backdoors Maybe someone can disrupt the open source funding problem by brokering exploit bounties /s
the8472 3y ago
github already mandates MFA for members of important projects
- rvense 3y ago
  Doesn't it mandate it for everyone? I don't use it anymore and haven't logged in since forever, but I think I got a series of e-mails that it was being made mandatory.
  - ryukoposting 3y ago
    It will soon. I think I have to sort it out before April 4. My passwords are already >20 random characters, so I wasn't going to do it until they told me to.
    - pabs3 3y ago
      If you are using pass to store those, check out pass-otp and browserpass, since GitHub still allows TOTP for MFA. pass-otp is based on oathtool, so you can do it more manually too if you don't use pass.

jnxx 3y ago

And, Joey Hess has counted at least 750 commits to xz from that handle. https://hachyderm.io/@joeyh/112180715824680521 https://hachyderm.io/@joeyh/112180715824680521 This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

ebfe1 3y ago
clickhouse has pretty good github_events dataset on playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/ https://ghe.clickhouse.tech/ Example of what this user JiaT75 did so far: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSdKaWFUNzUnIE9SREVSIEJZIGZpbGVfdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT... pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0ZWRfYXQsIGV2ZW50X3R5cGUsIGFjdG9yX2xvZ2luLCByZXBvX25hbWUsdGl0bGUsYWN0aW9uLCBzdWJzdHJpbmcoYm9keSwgMSwgMjAwKSBhcyBib2R5X3RydW5jYXRlZCAgRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGZpbGVfdGltZSA+IG5vdygpIC0gSU5URVJWQUwgNjAgREFZIEFORCBldmVudF90eXBlPSdQdWxsUmVxdWVzdEV2ZW50JyBBTkQgdGl0bGUgSUxJS0UgJyU1LjYlJyBBTkQgdGl0bGUgSUxJS0UgJyV4eiUnIEFORCBib2R5IE5PVCBJTElLRSAnJVNueWslJyBhbmQgYm9keSBub3QgaWxpa2UgJyVDVkUtMjAyNC0zMDk0JScgYW5kIHRpdGxlIG5vdCBpbGlrZSAnJWRvd25ncmFkZSUnIE9SREVSIEJZIGZpbGVfdGltZSBERVNDIExJTUlUIDEwMA== https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...
- palijer 3y ago
  Yeah. It would be interesting to see who adopted to the compromised versions and how quickly, compared to how quickly they normally adopt new versions (not bots pulling upgrades, but how quickly maintainers approve and merge them) If there were a bunch of people who adopted it abnormally fast compared to usual, might point to there being more "bad actors" in this operation (said at the risk of sounding paranoid if this turns out to be a state run thing)
waynesonfire 3y ago
750 commits... is xz able to send e-mails yet?

moritonal 3y ago

Warning, drunk brain talking. But a LLM driven email based "collaborator" could play a very long gMw adding basic features to a code made whilst earning trust backed by a generated online presence. My money is on a resurgance in the Web of Trust.

w4ffl35 3y ago
Clearly a human is even better at it.
llmblockchain 3y ago
State level actor? China?
- logankeenan 3y ago
  You're likely being downvoted because the Github profile looking like east Asian isn't evidence of where the attacker/attackers are from. Nation states will go to long lengths to disguise their identity. Using broken Russian English when they are not Russian, putting comments in the code of another language, and all sorts of other things to create misdirection.
  - llmblockchain 3y ago
    That's certainly true-- at the very least it "seems" like Asian, but it could very well be from any nation. If they were patient enough to work up to this point they would likely not be dumb enough to leak such information.
  - account42 3y ago
    Otoh with these kind of kneejerk "antiracist" reactions the best way to disguise a chinese operative becomes to prentend you're chinese.
jnxx 3y ago
The web of trust is a really nice idea, but it works badly against that kind of attacks. Just consider that in the real world, most living people (all eight billions) are linked by only six degrees of separation. It really works, for code and for trusted social relations (like "I lend you 100 bucks and you pay me them back when you get your salary") mostly when you know the code author in person. This is also not a new insight. In the beginning of the naughties, there was a web site named kuro5hin.org, which experiemented with user ratings and trust networks. It turned out impossible to prevent take-overs.

lucasRW 3y ago

More likely that the account of that dev was breawched, dont you think ?

dang 3y ago

(I detached this subthread from https://news.ycombinator.com/item?id=39866275 https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)