6 ms·

Shrinking While Linking

49 points by ingve 11mo ago

kragen 11mo ago
These are extremely practical tips on using binutils to shrink your libraries. It might also be worthwhile to compile the library without per-function sections in the first place, which hopefully can be done without patching rustc. On many platforms this produces significantly smaller code in GCC.
- jneem 11mo ago
  I had a quick look and couldn't find a way to turn off per-function sections in rustc. But I think it's a pretty good default for rust, because the unit of compilation is pretty large and so it's common to have a lot of unused functions. It's really only a problem for distributing static libraries, since binaries and shared libraries already lose their per-function sections.
  - kragen 11mo ago
    I would rather say it's really only potentially beneficial for distributing static libraries, since binaries and shared libraries already lose their per-function sections. So they pay the cost of the fluffier code without getting much benefit, if any.
lionkor 11mo ago
> Back in the day, your compiler or assembler would turn each source file into an “object” file containing the compiled code. Lots and lots of code is still C and C++. That's not really "back in the day".
- dcminter 11mo ago
  I'd be very surprised if the C/C++ code being written today was comparable in volume to the collective pile of JavaScript, Java, C#, Go, Python, and PHP. Contrast to the heyday of C and C++ when not much else got a look in (Pascal perhaps? Perl probably...) I think it's fair.
  - kragen 11mo ago
    Yes, surely much more code is being written in other languages than in C, but probably more C is being written now than ever before, too.
    - dcminter 11mo ago
      Sure, but I think that "back in the day your compiler would ..." is a reasonable characterisation. Back in the day, yes, your compiler almost certainly would do that. Now, your compiler might possibly do that. For most working stiffs it won't though. Ok, my side of the shed's painted now :D
      - kragen 11mo ago
        I agree.
  - happyweasel 11mo ago
    hmm how can I reuse this useful Go library in python... Oh I can't.. hmm and how can I reuse this useful java library in php ? Oh I can't. Oh and which of the programming languages you mentioned can and do use C libraries? All of them. Reminds me of that coworker who thought that OpenCV was basically written in python.
    - dcminter 11mo ago
      I'm not claiming that there is no C or C++ out there. But it's such a nit pick when for most developers, no, their day-to-day work absolutely does not involve the creation of object files. Sure, akshuwally, there are still C and C++ devs out there. Meanwhile a friend has just embarked upon a career as a pro COBOL developer. What of it? Edit: Also, in the spirit of akshewally, I have just googled up this monster! My word, PHP and Java AND XML... it's like the unholy trinity of HackerNewsbane... https://php-java-bridge.sourceforge.net/pjb/ https://php-java-bridge.sourceforge.net/pjb/
      - actionfromafar 11mo ago
Joker_vD 11mo ago
> A static library is nothing but a bundle of object files, wrapped in an ancient and never-quite-standardized archive format. To this day I'm astonished that it's not just tar. Or pax. Or even cpio! Or literally any file format that has any other use.
- xyzzy_plugh 11mo ago
  tar/pax are kind of terrible formats. They are hard to implement correctly. I'm glad they are not used more often. cpio is pretty reasonable though. zip is actually pretty great and I've been growing increasingly fond of it over the years.
  - Joker_vD 11mo ago
    The thing is, there is always tar(1) even in the most basic of distributions. And everyone uses tar.gz's or .bz2's or whatever for distributing all kinds of things, so tar is pretty ubiquitous. But the moment you want to do some C development, or any binutils-related, nope, install and use ar(1) which is used for literally one single purpose and nothing else. Because reasons.
    - hyperman1 11mo ago
      Im not sure how ar does it, but tar has no centralised directory. The only way to get file 100 is to walk trough the 99 files before. This kills random access speed.
      - Joker_vD 11mo ago
        Ar puts a file called "/" as the first file of the archive. Inside, there is a number N, then a list of N file offsets, and then a list of N null-terminated strings. It's a symbol table of sorts: each null-terminated string is a symbol name, and the corresponding file offset points at the archive header for the object file that contains the symbol. The filenames themselves are not recorded centrally since it's not really needed.
    - 11mo ago
nextaccountic 11mo ago
The real problem are generics. They cause a blowup in size because they are instantiated at every type combination that is actually called. This can sometimes be a performance boost (typically not because it avoids an indirect call, but because it enables optimizations such as inlining). But it can also makes code slower (not by much), or have little effect on performance Rust lets you choose between generics and trait objects, but this is a viral change that sometimes means that large sections of code must be rewritten. There is also an optimization that turns generics into virtual dispatch if deemed beneficial, but I'm not sure how well it works
compiler-guy 11mo ago
"If you compiled your code with support for link-time optimization (LTO), it contains another copy (in the form of LLVM bitcode — more on that later) of all our code and the code of all our dependencies." The author then goes and removes the lto-related code via objcopy. Which is fine, and great, and useful if you care about size more than having link-time optimization enabled for your end users. But if you are just going to turn around and remove it, you probably shouldn't include it in the first place, which makes the original analysis of 132MB vs 15MB a little misleading. It should be more like 50MB vs 15.

49 points by ingve 11mo ago

kragen 11mo ago

These are extremely practical tips on using binutils to shrink your libraries. It might also be worthwhile to compile the library without per-function sections in the first place, which hopefully can be done without patching rustc. On many platforms this produces significantly smaller code in GCC.

jneem 11mo ago
I had a quick look and couldn't find a way to turn off per-function sections in rustc. But I think it's a pretty good default for rust, because the unit of compilation is pretty large and so it's common to have a lot of unused functions. It's really only a problem for distributing static libraries, since binaries and shared libraries already lose their per-function sections.
- kragen 11mo ago
  I would rather say it's really only potentially beneficial for distributing static libraries, since binaries and shared libraries already lose their per-function sections. So they pay the cost of the fluffier code without getting much benefit, if any.

lionkor 11mo ago

> Back in the day, your compiler or assembler would turn each source file into an “object” file containing the compiled code. Lots and lots of code is still C and C++. That's not really "back in the day".

dcminter 11mo ago
I'd be very surprised if the C/C++ code being written today was comparable in volume to the collective pile of JavaScript, Java, C#, Go, Python, and PHP. Contrast to the heyday of C and C++ when not much else got a look in (Pascal perhaps? Perl probably...) I think it's fair.
- kragen 11mo ago
  Yes, surely much more code is being written in other languages than in C, but probably more C is being written now than ever before, too.
  - dcminter 11mo ago
    Sure, but I think that "back in the day your compiler would ..." is a reasonable characterisation. Back in the day, yes, your compiler almost certainly would do that. Now, your compiler might possibly do that. For most working stiffs it won't though. Ok, my side of the shed's painted now :D
    - kragen 11mo ago
      I agree.
- happyweasel 11mo ago
  hmm how can I reuse this useful Go library in python... Oh I can't.. hmm and how can I reuse this useful java library in php ? Oh I can't. Oh and which of the programming languages you mentioned can and do use C libraries? All of them. Reminds me of that coworker who thought that OpenCV was basically written in python.
  - dcminter 11mo ago
    I'm not claiming that there is no C or C++ out there. But it's such a nit pick when for most developers, no, their day-to-day work absolutely does not involve the creation of object files. Sure, akshuwally, there are still C and C++ devs out there. Meanwhile a friend has just embarked upon a career as a pro COBOL developer. What of it? Edit: Also, in the spirit of akshewally, I have just googled up this monster! My word, PHP and Java AND XML... it's like the unholy trinity of HackerNewsbane... https://php-java-bridge.sourceforge.net/pjb/ https://php-java-bridge.sourceforge.net/pjb/
    - actionfromafar 11mo ago

Joker_vD 11mo ago

> A static library is nothing but a bundle of object files, wrapped in an ancient and never-quite-standardized archive format. To this day I'm astonished that it's not just tar. Or pax. Or even cpio! Or literally any file format that has any other use.

xyzzy_plugh 11mo ago
tar/pax are kind of terrible formats. They are hard to implement correctly. I'm glad they are not used more often. cpio is pretty reasonable though. zip is actually pretty great and I've been growing increasingly fond of it over the years.
- Joker_vD 11mo ago
  The thing is, there is always tar(1) even in the most basic of distributions. And everyone uses tar.gz's or .bz2's or whatever for distributing all kinds of things, so tar is pretty ubiquitous. But the moment you want to do some C development, or any binutils-related, nope, install and use ar(1) which is used for literally one single purpose and nothing else. Because reasons.
  - hyperman1 11mo ago
    Im not sure how ar does it, but tar has no centralised directory. The only way to get file 100 is to walk trough the 99 files before. This kills random access speed.
    - Joker_vD 11mo ago
      Ar puts a file called "/" as the first file of the archive. Inside, there is a number N, then a list of N file offsets, and then a list of N null-terminated strings. It's a symbol table of sorts: each null-terminated string is a symbol name, and the corresponding file offset points at the archive header for the object file that contains the symbol. The filenames themselves are not recorded centrally since it's not really needed.
  - 11mo ago

nextaccountic 11mo ago

The real problem are generics. They cause a blowup in size because they are instantiated at every type combination that is actually called. This can sometimes be a performance boost (typically not because it avoids an indirect call, but because it enables optimizations such as inlining). But it can also makes code slower (not by much), or have little effect on performance Rust lets you choose between generics and trait objects, but this is a viral change that sometimes means that large sections of code must be rewritten. There is also an optimization that turns generics into virtual dispatch if deemed beneficial, but I'm not sure how well it works

compiler-guy 11mo ago

"If you compiled your code with support for link-time optimization (LTO), it contains another copy (in the form of LLVM bitcode — more on that later) of all our code and the code of all our dependencies." The author then goes and removes the lto-related code via objcopy. Which is fine, and great, and useful if you care about size more than having link-time optimization enabled for your end users. But if you are just going to turn around and remove it, you probably shouldn't include it in the first place, which makes the original analysis of 132MB vs 15MB a little misleading. It should be more like 50MB vs 15.