9 ms·

And, Joey Hess has counted at least 750 commits to xz from that handle. https://hachyderm.io/@joeyh/112180715824680521 https://hachyderm.io/@joeyh/112180715824

by jnxx 3y ago

And, Joey Hess has counted at least 750 commits to xz from that handle. https://hachyderm.io/@joeyh/112180715824680521 https://hachyderm.io/@joeyh/112180715824680521 This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

ebfe1 3y ago
clickhouse has pretty good github_events dataset on playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/ https://ghe.clickhouse.tech/ Example of what this user JiaT75 did so far: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSdKaWFUNzUnIE9SREVSIEJZIGZpbGVfdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT... pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0ZWRfYXQsIGV2ZW50X3R5cGUsIGFjdG9yX2xvZ2luLCByZXBvX25hbWUsdGl0bGUsYWN0aW9uLCBzdWJzdHJpbmcoYm9keSwgMSwgMjAwKSBhcyBib2R5X3RydW5jYXRlZCAgRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGZpbGVfdGltZSA+IG5vdygpIC0gSU5URVJWQUwgNjAgREFZIEFORCBldmVudF90eXBlPSdQdWxsUmVxdWVzdEV2ZW50JyBBTkQgdGl0bGUgSUxJS0UgJyU1LjYlJyBBTkQgdGl0bGUgSUxJS0UgJyV4eiUnIEFORCBib2R5IE5PVCBJTElLRSAnJVNueWslJyBhbmQgYm9keSBub3QgaWxpa2UgJyVDVkUtMjAyNC0zMDk0JScgYW5kIHRpdGxlIG5vdCBpbGlrZSAnJWRvd25ncmFkZSUnIE9SREVSIEJZIGZpbGVfdGltZSBERVNDIExJTUlUIDEwMA== https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...
- palijer 3y ago
  Yeah. It would be interesting to see who adopted to the compromised versions and how quickly, compared to how quickly they normally adopt new versions (not bots pulling upgrades, but how quickly maintainers approve and merge them) If there were a bunch of people who adopted it abnormally fast compared to usual, might point to there being more "bad actors" in this operation (said at the risk of sounding paranoid if this turns out to be a state run thing)
waynesonfire 3y ago
750 commits... is xz able to send e-mails yet?
- wyldfire 3y ago
  No. But if you have any centrifuges they will probably exhibit inconsistent behavior.
  - shermantanktop 3y ago
    Maybe it’s the centrifuges which will send the mail, making the world’s first uranium-enriching spam botnet.
- lovasoa 3y ago
  Yes, it sends an email containing your private key on installation.
- soraminazuki 3y ago
  It's hardly surprising given that parsing is generally considered to be a tricky problem. Plus, it's a 15 years old project that's widely used. 750 commits is nothing to sneer about. No wonder the original maintainer got burned out.
- account42 3y ago
  750 is not that many commits if you follow a commit style where each independent change gets its own commit.
indigodaddy 3y ago
Anyone have any level of confidence that for example EL7/8 would not be at risk even if more potential exploits at play?
- soraminazuki 3y ago
  I wouldn't count on it. RedHat packages contain lots of backported patches.
  - indigodaddy 3y ago
    Right, that notion was what was making me nervous
    - worthless-trash 3y ago
      At least in my group, the backports are hand picked to solve specific problems, not just random wholesale backports.
      - soraminazuki 3y ago
        Sure, they aren't backporting patches wholesale. But the patches that did get backported, if any, needs much more scrutiny given the situation. The thing about enterprise Linux distros is that they have a long support period. Bug fixes and security patches pile up. Fortunately though, xz doesn't seem to have a lot of backported patches. https://git.centos.org/rpms/xz/blob/c8s/f/SPECS/xz.spec https://git.centos.org/rpms/xz/blob/c8s/f/SPECS/xz.spec https://git.centos.org/rpms/xz/blob/c7/f/SPECS/xz.spec https://git.centos.org/rpms/xz/blob/c7/f/SPECS/xz.spec But take glibc for example. The amount of patches makes my head spin. https://git.centos.org/rpms/glibc/blob/c8s/f/SPECS/glibc.spec https://git.centos.org/rpms/glibc/blob/c8s/f/SPECS/glibc.spe...
      - bobba27 3y ago
        For the duration of a major release, up until ~x.4 pretty much everything from upstream gets backported with a delay of 6-12 months, depending on how conservative to change the rhel engineer maintaining this part of the kernel is. After ~x.4 things slow down and only "important" fixes get backported but no new features. After ~x.7 or so different processes and approvals come into play and virtually nothing except high severity bugs or something that "important customer" needs will be backported.
jnxx 3y ago
If this is a conspiracy or a state-sponsored attack, they might have gone specifically for embedded devices and the linux kernel. Here archived from tukaani.org: https://web.archive.org/web/20110831134700/http://tukaani.org/xz/embedded.html https://web.archive.org/web/20110831134700/http://tukaani.or... > XZ Embedded is a relatively small decompressor for the XZ format. It was developed with the Linux kernel in mind, but is easily usable in other projects too. > *Features* > * Compiled code 8-20 KiB > [...] > * All the required memory is allocated at initialization time. This is targeted at embedded and real-time stuff. Could even be part of boot loaders in things like buildroot or RTEMS. And this means potentially millions of devices, from smart toasters or toothbrushes to satellites and missiles which most can't be updated with security fixes.
- jnxx 3y ago
  [flagged]
  - masklinn 3y ago
    That is just technical disagreements and sour grapes by someone involved in a competing format (Lzip). There’s no evidence Lasse did anything “wrong” beyond looking for / accepting co-maintainers, something package authors are taken to task for not doing every time they have life catching up or get fed up and can’t / won’t spend as much time on the thing.
    - jnxx 3y ago
      [flagged]
      - matsemann 3y ago
        > But is this appropiate here? Yes, nothing points to the inventor of the format and maintainer for decades has done anything with the format to make it suspect. If so, the recent backdoor wouldn't be needed. It's good to be skeptic, but don't drag people through the mud without anything to back it up.
codedokode 3y ago
> If the code is complex, there could be many more exploits hiding. Then the code should not be complex. Low-level hacks and tricks (like pointer juggling) should be not allowed and simplicity and readability should be preferred.
- sgarland 3y ago
  For tools like compression programs, you’d generally prefer performance over everything (except data corruption, of course).
  - jononor 3y ago
    Probably you would prefer no backdoors also? Performance without correctness or trustworthiness is useless.
    - sgarland 3y ago
      Yes, but my point was that at the level of performance tools like this are expected to operate at, it’s highly probable that you’ll need to get into incredibly esoteric code. Look at ffmpeg – tons of hand-written Assembly, because they need it. To be clear, I have no idea how to solve this problem; I just don’t think saying that all code must be non-hacky is the right approach.
      - jononor 3y ago
        Performance can be bought with better hardware. It gets cheaper and cheaper every year. Trustworthiness cannot be purchased in the same way. I do not understand why performance would ever trumph clean code, especially for for code that processes user provided input.
        patchguard 3y ago
        > Performance can be bought with better hardware. I hate this argument. If current hardware promises you a theoretical throughput of 100 MB/s for an operation, someone will try to hit that limit. Your program that has no hard to understand code but gives me 5 MB/s will loose in the face of a faster one, even if that means writing harder to understand code.
pmarreck 3y ago
I have some questions. 1) Are there no legit code reviews from contributors like this? How did this get accepted into main repos while flying under the radar? When I do a code review, I try to understand the actual code I'm reviewing. Call me crazy I guess! 2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date.
- WhyNotHugo 3y ago
  > 2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date. Any government which uses GNU/Linux in their infrastructure can pitch this as an attempt to backdoor their servers. The real question is: will we ever even know who was behind this? If it was some mercenary hacker intending to resell the backdoor, maybe. But if it was someone working with an intelligence agency in US/China/Israel/Russia/etc, I doubt they'll ever be exposed.
  - darkelder 3y ago
    Reflecting on the idea of introducing a validation structure for software contributions, akin to what RPKI does for BGP routing, I see significant potential to enhance security and accountability in software development. Such a system could theoretically bring greater transparency and responsibility, particularly in an ecosystem where contributions come from all corners. Implementing verifiable identity proofs for contributors might be challenging, but it also presents an opportunity to bolster security without compromising privacy and the freedom to contribute under pseudonyms. The accountability of those accepting pull requests would also become clearer, potentially reducing the risk of malicious code being incorporated. Of course, establishing a robust validation chain for software would require the commitment of everyone in the development ecosystem, including platforms like GitHub. However, I view this not as a barrier but as an essential step towards evolving our approach to security and collaboration in software development.
- treffer 3y ago
  The actual inclusion code was never in the repo. The blobs were hidden as lzma test files. So you review would need to guess from 2 new test files that those are, decompressed, a backdoor and could be injected which was never in the git history. This was explicitly build to evade such reviews.

by jnxx 3y ago

ebfe1 3y ago

clickhouse has pretty good github_events dataset on playground that folks can use to do some research - some info on the dataset https://ghe.clickhouse.tech/ https://ghe.clickhouse.tech/ Example of what this user JiaT75 did so far: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGFjdG9yX2xvZ2luPSdKaWFUNzUnIE9SREVSIEJZIGZpbGVfdGltZSBERVND https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT... pull requests mentioning xz, 5.6 without downgrade, cve being mentioned in the last 60 days: https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0ZWRfYXQsIGV2ZW50X3R5cGUsIGFjdG9yX2xvZ2luLCByZXBvX25hbWUsdGl0bGUsYWN0aW9uLCBzdWJzdHJpbmcoYm9keSwgMSwgMjAwKSBhcyBib2R5X3RydW5jYXRlZCAgRlJPTSBnaXRodWJfZXZlbnRzIFdIRVJFIGZpbGVfdGltZSA+IG5vdygpIC0gSU5URVJWQUwgNjAgREFZIEFORCBldmVudF90eXBlPSdQdWxsUmVxdWVzdEV2ZW50JyBBTkQgdGl0bGUgSUxJS0UgJyU1LjYlJyBBTkQgdGl0bGUgSUxJS0UgJyV4eiUnIEFORCBib2R5IE5PVCBJTElLRSAnJVNueWslJyBhbmQgYm9keSBub3QgaWxpa2UgJyVDVkUtMjAyNC0zMDk0JScgYW5kIHRpdGxlIG5vdCBpbGlrZSAnJWRvd25ncmFkZSUnIE9SREVSIEJZIGZpbGVfdGltZSBERVNDIExJTUlUIDEwMA== https://play.clickhouse.com/play?user=play#U0VMRUNUIGNyZWF0Z...

palijer 3y ago
Yeah. It would be interesting to see who adopted to the compromised versions and how quickly, compared to how quickly they normally adopt new versions (not bots pulling upgrades, but how quickly maintainers approve and merge them) If there were a bunch of people who adopted it abnormally fast compared to usual, might point to there being more "bad actors" in this operation (said at the risk of sounding paranoid if this turns out to be a state run thing)

waynesonfire 3y ago

750 commits... is xz able to send e-mails yet?

wyldfire 3y ago
No. But if you have any centrifuges they will probably exhibit inconsistent behavior.
- shermantanktop 3y ago
  Maybe it’s the centrifuges which will send the mail, making the world’s first uranium-enriching spam botnet.
lovasoa 3y ago
Yes, it sends an email containing your private key on installation.
soraminazuki 3y ago
It's hardly surprising given that parsing is generally considered to be a tricky problem. Plus, it's a 15 years old project that's widely used. 750 commits is nothing to sneer about. No wonder the original maintainer got burned out.
account42 3y ago
750 is not that many commits if you follow a commit style where each independent change gets its own commit.

indigodaddy 3y ago

Anyone have any level of confidence that for example EL7/8 would not be at risk even if more potential exploits at play?

soraminazuki 3y ago
I wouldn't count on it. RedHat packages contain lots of backported patches.
- indigodaddy 3y ago
  Right, that notion was what was making me nervous
  - worthless-trash 3y ago
    At least in my group, the backports are hand picked to solve specific problems, not just random wholesale backports.
    - soraminazuki 3y ago
      Sure, they aren't backporting patches wholesale. But the patches that did get backported, if any, needs much more scrutiny given the situation. The thing about enterprise Linux distros is that they have a long support period. Bug fixes and security patches pile up. Fortunately though, xz doesn't seem to have a lot of backported patches. https://git.centos.org/rpms/xz/blob/c8s/f/SPECS/xz.spec https://git.centos.org/rpms/xz/blob/c8s/f/SPECS/xz.spec https://git.centos.org/rpms/xz/blob/c7/f/SPECS/xz.spec https://git.centos.org/rpms/xz/blob/c7/f/SPECS/xz.spec But take glibc for example. The amount of patches makes my head spin. https://git.centos.org/rpms/glibc/blob/c8s/f/SPECS/glibc.spec https://git.centos.org/rpms/glibc/blob/c8s/f/SPECS/glibc.spe...
    - bobba27 3y ago
      For the duration of a major release, up until ~x.4 pretty much everything from upstream gets backported with a delay of 6-12 months, depending on how conservative to change the rhel engineer maintaining this part of the kernel is. After ~x.4 things slow down and only "important" fixes get backported but no new features. After ~x.7 or so different processes and approvals come into play and virtually nothing except high severity bugs or something that "important customer" needs will be backported.

jnxx 3y ago

If this is a conspiracy or a state-sponsored attack, they might have gone specifically for embedded devices and the linux kernel. Here archived from tukaani.org: https://web.archive.org/web/20110831134700/http://tukaani.org/xz/embedded.html https://web.archive.org/web/20110831134700/http://tukaani.or... > XZ Embedded is a relatively small decompressor for the XZ format. It was developed with the Linux kernel in mind, but is easily usable in other projects too. > *Features* > * Compiled code 8-20 KiB > [...] > * All the required memory is allocated at initialization time. This is targeted at embedded and real-time stuff. Could even be part of boot loaders in things like buildroot or RTEMS. And this means potentially millions of devices, from smart toasters or toothbrushes to satellites and missiles which most can't be updated with security fixes.

jnxx 3y ago
[flagged]
- masklinn 3y ago
  That is just technical disagreements and sour grapes by someone involved in a competing format (Lzip). There’s no evidence Lasse did anything “wrong” beyond looking for / accepting co-maintainers, something package authors are taken to task for not doing every time they have life catching up or get fed up and can’t / won’t spend as much time on the thing.
  - jnxx 3y ago
    [flagged]
    - matsemann 3y ago
      > But is this appropiate here? Yes, nothing points to the inventor of the format and maintainer for decades has done anything with the format to make it suspect. If so, the recent backdoor wouldn't be needed. It's good to be skeptic, but don't drag people through the mud without anything to back it up.

codedokode 3y ago

> If the code is complex, there could be many more exploits hiding. Then the code should not be complex. Low-level hacks and tricks (like pointer juggling) should be not allowed and simplicity and readability should be preferred.

sgarland 3y ago
For tools like compression programs, you’d generally prefer performance over everything (except data corruption, of course).
- jononor 3y ago
  Probably you would prefer no backdoors also? Performance without correctness or trustworthiness is useless.
  - sgarland 3y ago
    Yes, but my point was that at the level of performance tools like this are expected to operate at, it’s highly probable that you’ll need to get into incredibly esoteric code. Look at ffmpeg – tons of hand-written Assembly, because they need it. To be clear, I have no idea how to solve this problem; I just don’t think saying that all code must be non-hacky is the right approach.
    - jononor 3y ago
      Performance can be bought with better hardware. It gets cheaper and cheaper every year. Trustworthiness cannot be purchased in the same way. I do not understand why performance would ever trumph clean code, especially for for code that processes user provided input.
      - patchguard 3y ago
        > Performance can be bought with better hardware. I hate this argument. If current hardware promises you a theoretical throughput of 100 MB/s for an operation, someone will try to hit that limit. Your program that has no hard to understand code but gives me 5 MB/s will loose in the face of a faster one, even if that means writing harder to understand code.

pmarreck 3y ago

I have some questions. 1) Are there no legit code reviews from contributors like this? How did this get accepted into main repos while flying under the radar? When I do a code review, I try to understand the actual code I'm reviewing. Call me crazy I guess! 2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date.

WhyNotHugo 3y ago
> 2) Is there no legal recourse to this? We're talking about someone who managed to root any linux server that stays up-to-date. Any government which uses GNU/Linux in their infrastructure can pitch this as an attempt to backdoor their servers. The real question is: will we ever even know who was behind this? If it was some mercenary hacker intending to resell the backdoor, maybe. But if it was someone working with an intelligence agency in US/China/Israel/Russia/etc, I doubt they'll ever be exposed.
- darkelder 3y ago
  Reflecting on the idea of introducing a validation structure for software contributions, akin to what RPKI does for BGP routing, I see significant potential to enhance security and accountability in software development. Such a system could theoretically bring greater transparency and responsibility, particularly in an ecosystem where contributions come from all corners. Implementing verifiable identity proofs for contributors might be challenging, but it also presents an opportunity to bolster security without compromising privacy and the freedom to contribute under pseudonyms. The accountability of those accepting pull requests would also become clearer, potentially reducing the risk of malicious code being incorporated. Of course, establishing a robust validation chain for software would require the commitment of everyone in the development ecosystem, including platforms like GitHub. However, I view this not as a barrier but as an essential step towards evolving our approach to security and collaboration in software development.
treffer 3y ago
The actual inclusion code was never in the repo. The blobs were hidden as lzma test files. So you review would need to guess from 2 new test files that those are, decompressed, a backdoor and could be injected which was never in the git history. This was explicitly build to evade such reviews.