4 ms·

Ubuntu still ships 5.4.5 on 24.03 (atm). I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github

Ubuntu still ships 5.4.5 on 24.03 (atm). I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though. So for 5.4.5 the tagged release and download on github differ. It does change format strings, e.g. +#: src/xz/args.c:735 +#, fuzzy +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout" +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard" There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs. [Edit] formatting

fransje26 3y ago
Thanks for the heads up.
chasil 3y ago
RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.
mort96 3y ago
FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code. Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?
- treffer 3y ago
  Thank you, formatting fixed. My TLDR is that I would regard all commits by JiaT75 as potentially compromised. Given the ability to manipulate gitnhistory I am not sure if a simple time based revert is enough. It would be great to compare old copies of the repo with the current state. There is no guarantee that the history wasn't tampered with. Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.
  - tomrod 3y ago
    Not just Jia. There are some other accounts of concern with associated activity or short term/bot-is names.
    - jdright 3y ago
      yes, like this one: https://github.com/facebook/folly/pull/2153 https://github.com/facebook/folly/pull/2153
      - account42 3y ago
        Interesting, this would suggest exploits other than the known sshd one.
  - ogurechny 3y ago
    Zstd belongs to the class of speed-optimized compressors providing “tolerable” compression ratios. Their intended use case is wrapping some easily compressible data with negligible (in the grand scale) performance impact. So when you have a server which sends gigabits of text per second, or caches gigabytes of text, or processes a queue with millions of text protocol messages, you can add compression on one side and decompression on the other to shrink them without worrying too much about CPU usage. Xz is an implant of 7zip's LZMA(2) compression into traditional Unix archiver skeleton. It trades long compression times and giant dictionaries (that need lots of memory) for better (“much-better-than-deflate”) compression ratios. Therefore, zstd, no matter how fashionable that name might be in some circles, is not a replacement for xz. It should also be noted that those LZMA-based archive formats might not be considered state-of-the-art today. If you worry about data density, there are options for both faster compression at the same size, and better compression in the same amount of time (provided that data is generally compressible). 7zip and xz are widespread and well tested, though, and allow decompression to be fast, which might be important in some cases. Alternatives often decompress much slowly. This is also a trade-off between total time spent on X nodes compressing data, and Y nodes decompressing data. When X is 1, and Y is in the millions (say, software distribution), you can spend A LOT of time compressing even for relatively minuscule gains without affecting the scales. It should also be noted that many (or most) decoders of top compressing archivers are implemented as virtual machines executing chains of transform and unpack operations defined in archive file over pieces of data also saved there. Or, looking from a different angle, complex state machines initializing their state using complex data in the archive. Compressor tries to find most suitable combination of basic steps based on input data, and stores the result in the archive. (This is logically completed in neural network compression tools which learn what to do with data from data itself.) As some people may know, implementing all that byte juggling safely and effectively is a herculean task, and compression tools had exploits in the past because of that. Switching to a better solution might introduce a lot more potentially exploited bugs.
jwilk 3y ago
"#, fuzzy" means the translation is out-of-date and it will be discarded at compile time.
- treffer 3y ago
  I tried to get the translation to trigger by switching to french and it does not show. You are right. So it's just odd that the tags and release tarballs diverge.

fransje26 3y ago

Thanks for the heads up.

chasil 3y ago

RHEL9 is shipping 5.2.5; RHEL8 is on 5.2.4.

mort96 3y ago

FYI, your formatting is broken. Hacker News doesn't support backtick code blocks, you have to indent code. Anyway, so... the xz project has been compromised for a long time, at least since 5.4.5. I see that this JiaT75 guy has been the primary guy in charge of at least the GitHub releases for years. Should we view all releases after he got involved as probably compromised?

treffer 3y ago
Thank you, formatting fixed. My TLDR is that I would regard all commits by JiaT75 as potentially compromised. Given the ability to manipulate gitnhistory I am not sure if a simple time based revert is enough. It would be great to compare old copies of the repo with the current state. There is no guarantee that the history wasn't tampered with. Overall the only safe action would IMHO to establish a new upstream from an assumed good state, then fully audit it. At that point we should probably just abandon it and use zstd instead.
- tomrod 3y ago
  Not just Jia. There are some other accounts of concern with associated activity or short term/bot-is names.
  - jdright 3y ago
    yes, like this one: https://github.com/facebook/folly/pull/2153 https://github.com/facebook/folly/pull/2153
    - account42 3y ago
      Interesting, this would suggest exploits other than the known sshd one.
- ogurechny 3y ago
  Zstd belongs to the class of speed-optimized compressors providing “tolerable” compression ratios. Their intended use case is wrapping some easily compressible data with negligible (in the grand scale) performance impact. So when you have a server which sends gigabits of text per second, or caches gigabytes of text, or processes a queue with millions of text protocol messages, you can add compression on one side and decompression on the other to shrink them without worrying too much about CPU usage. Xz is an implant of 7zip's LZMA(2) compression into traditional Unix archiver skeleton. It trades long compression times and giant dictionaries (that need lots of memory) for better (“much-better-than-deflate”) compression ratios. Therefore, zstd, no matter how fashionable that name might be in some circles, is not a replacement for xz. It should also be noted that those LZMA-based archive formats might not be considered state-of-the-art today. If you worry about data density, there are options for both faster compression at the same size, and better compression in the same amount of time (provided that data is generally compressible). 7zip and xz are widespread and well tested, though, and allow decompression to be fast, which might be important in some cases. Alternatives often decompress much slowly. This is also a trade-off between total time spent on X nodes compressing data, and Y nodes decompressing data. When X is 1, and Y is in the millions (say, software distribution), you can spend A LOT of time compressing even for relatively minuscule gains without affecting the scales. It should also be noted that many (or most) decoders of top compressing archivers are implemented as virtual machines executing chains of transform and unpack operations defined in archive file over pieces of data also saved there. Or, looking from a different angle, complex state machines initializing their state using complex data in the archive. Compressor tries to find most suitable combination of basic steps based on input data, and stores the result in the archive. (This is logically completed in neural network compression tools which learn what to do with data from data itself.) As some people may know, implementing all that byte juggling safely and effectively is a herculean task, and compression tools had exploits in the past because of that. Switching to a better solution might introduce a lot more potentially exploited bugs.

jwilk 3y ago

"#, fuzzy" means the translation is out-of-date and it will be discarded at compile time.

treffer 3y ago
I tried to get the translation to trigger by switching to french and it does not show. You are right. So it's just odd that the tags and release tarballs diverge.