4 ms·

Patchelf is almost always broken, and when they merge fixes they don't release a new version right away. Between 2014-2019 running patchelf+strip would corrupt

Patchelf is almost always broken, and when they merge fixes they don't release a new version right away. Between 2014-2019 running patchelf+strip would corrupt binaries, after the fix running patchelf twice on the same binary would corrupt it. And now patchelf doesn't work on all binaries, there's a bugfix merged https://github.com/NixOS/patchelf/pull/230 https://github.com/NixOS/patchelf/pull/230 merged Nov 19, 2020, but no release since then. Also patchelf exposed a bug in ldconfig noted 11 years ago and only fixed in glibc 2.31: https://nix-dev.science.uu.narkive.com/q6Ww5fyO/ldconfig-problem-with-patchelf-and-64-bit-libs https://nix-dev.science.uu.narkive.com/q6Ww5fyO/ldconfig-pro... And currently patchelf still has a bug: https://github.com/NixOS/patchelf/pull/275 https://github.com/NixOS/patchelf/pull/275 Apparently elf files are hard.

jchw 5y ago
To be fair, patching existing binaries isn’t all that easy to begin with. Once you need to start moving around structures, a lot of bets are off. In case of NixOS, I’m sure the long nix store paths added often overflow structures and require workarounds to avoid breaking images. Hacking around Win32 PE images, I’ve run into a lot of tricky subtle issues over time...
- stabbles 5y ago
  Yeah, it wasn't meant to criticize the project as a whole; I'm happy that at least someone took on the project of "growing" the rpath section... My only point of critique is they should immediately tag a new version after a bug fix, cause many distro's have buggy versions of patchelf.
ornxka 5y ago
I'm not really sure why it has to be that hard, like, why don't we just use base64-encoded JSON or something?
- KMag 5y ago
  > why don't we just use base64-encoded JSON or something? Base64 would be counterproductive, increasing both space and parsing time... presumably out of fear that JSON would contain a naughty byte for a greenfield file format. It would be much better to just design tho format to not have any naughty bytes. Parsing time for exacutables and libraries is definitely on the critical startup path. You really want a length-delimited format, or better yet, one where offsets to various structures are stored at fixed offsets so you can find everything in O(1) time with a tiny constant factor. Compiler writers and tool authors are perfectly comfortable working with binary file formats. There's nothing more inherently future-compatible about JSON than a forward-compatible binary format like flatbuffers. Having to escape and then unescape naughty bytes is a huge downside for text-based formats that are hardly ever read by humans. On a side note, Zlib DEFLATE / gzip / LZMA etc. aren't magic for getting rid of space overheads. Try gzip -9'ing your system's wordlist, now convert it to UTF-16 and gzip -9'ing it. You'll see a several percentage increase in size, despite an entropy change of at most a constant and small number of bits (-log2(P(UTF-16)/P(UTF-8)). I've frequently seen huge JSON proponents use hand-wavy arguments that gzip will reduce any size differences to zero. It's also nice if the file format is very close to being able to just be mmap()ed into the process's address space and only require minimal patching to a minimum number of pages in order to be an optimized in-memory representation. Also, there's a huge amount of momentum behind executable formats. Incremental improvements by adding new features in new segment types or appending new fields to old data structures (where there's no ambiguity) is much preferred to wholly new formats. So, creating a new debugging symbol section that's just flatbuffers is workable. Replacing the whole ecosystem with base64-'d JSON would have way more downsides than upsides. On another side note, you need to be very careful with JSONifying floating point values. Many libraries don't give you bit-perfect round-tripping of IEEE-754 double precision values.
  - wyldfire 5y ago
    YHBT. HAND.
    - KMag 5y ago
      Heh. Hopefully some of the production systems I work with are also well-played trollings.
- geocar 5y ago
  > why don't we just use base64-encoded JSON or something? You should try it and let us know how that works.
deleted 5y ago
[deleted]
phendrenad2 5y ago
elf files ARE hard, the documentation is best described as "adequate". You can often learn a lot of things looking at how GCC generates ELF files that might not be obvious from looking at the ELF spec.
mwill 5y ago
Nixos has been my main distro for ages now, and I gotta admit, I'm still a little bit surprised every time patchelf works
nmfisher 5y ago
patchelf is a lifesaver for handling Android/iOS with libraries compiled outside the regular toolchain (both platforms can be very finicky about soname/rpath). On one occasion, however, I did spend days chasing a crash that turned out to be a bug when using an older version of patchelf on ARM binaries. So great tool, but definitely be wary of introducing weird flakiness.

jchw 5y ago

To be fair, patching existing binaries isn’t all that easy to begin with. Once you need to start moving around structures, a lot of bets are off. In case of NixOS, I’m sure the long nix store paths added often overflow structures and require workarounds to avoid breaking images. Hacking around Win32 PE images, I’ve run into a lot of tricky subtle issues over time...

stabbles 5y ago
Yeah, it wasn't meant to criticize the project as a whole; I'm happy that at least someone took on the project of "growing" the rpath section... My only point of critique is they should immediately tag a new version after a bug fix, cause many distro's have buggy versions of patchelf.

ornxka 5y ago

I'm not really sure why it has to be that hard, like, why don't we just use base64-encoded JSON or something?

KMag 5y ago
> why don't we just use base64-encoded JSON or something? Base64 would be counterproductive, increasing both space and parsing time... presumably out of fear that JSON would contain a naughty byte for a greenfield file format. It would be much better to just design tho format to not have any naughty bytes. Parsing time for exacutables and libraries is definitely on the critical startup path. You really want a length-delimited format, or better yet, one where offsets to various structures are stored at fixed offsets so you can find everything in O(1) time with a tiny constant factor. Compiler writers and tool authors are perfectly comfortable working with binary file formats. There's nothing more inherently future-compatible about JSON than a forward-compatible binary format like flatbuffers. Having to escape and then unescape naughty bytes is a huge downside for text-based formats that are hardly ever read by humans. On a side note, Zlib DEFLATE / gzip / LZMA etc. aren't magic for getting rid of space overheads. Try gzip -9'ing your system's wordlist, now convert it to UTF-16 and gzip -9'ing it. You'll see a several percentage increase in size, despite an entropy change of at most a constant and small number of bits (-log2(P(UTF-16)/P(UTF-8)). I've frequently seen huge JSON proponents use hand-wavy arguments that gzip will reduce any size differences to zero. It's also nice if the file format is very close to being able to just be mmap()ed into the process's address space and only require minimal patching to a minimum number of pages in order to be an optimized in-memory representation. Also, there's a huge amount of momentum behind executable formats. Incremental improvements by adding new features in new segment types or appending new fields to old data structures (where there's no ambiguity) is much preferred to wholly new formats. So, creating a new debugging symbol section that's just flatbuffers is workable. Replacing the whole ecosystem with base64-'d JSON would have way more downsides than upsides. On another side note, you need to be very careful with JSONifying floating point values. Many libraries don't give you bit-perfect round-tripping of IEEE-754 double precision values.
- wyldfire 5y ago
  YHBT. HAND.
  - KMag 5y ago
    Heh. Hopefully some of the production systems I work with are also well-played trollings.
geocar 5y ago
> why don't we just use base64-encoded JSON or something? You should try it and let us know how that works.

deleted 5y ago

[deleted]

phendrenad2 5y ago

elf files ARE hard, the documentation is best described as "adequate". You can often learn a lot of things looking at how GCC generates ELF files that might not be obvious from looking at the ELF spec.

mwill 5y ago

Nixos has been my main distro for ages now, and I gotta admit, I'm still a little bit surprised every time patchelf works

nmfisher 5y ago

patchelf is a lifesaver for handling Android/iOS with libraries compiled outside the regular toolchain (both platforms can be very finicky about soname/rpath). On one occasion, however, I did spend days chasing a crash that turned out to be a bug when using an older version of patchelf on ARM binaries. So great tool, but definitely be wary of introducing weird flakiness.