4 ms·

That's pretty much the only reason I cringe every time somebody writes a piece of client code in golang. While I really like the language itself, statically li

by tribaal 11y ago

That's pretty much the only reason I cringe every time somebody writes a piece of client code in golang. While I really like the language itself, statically linking everything is creating exactly what this article describes. Burn karma, burn. I don't care.

Galanwe 11y ago
I dont really understand the link between software bloat and static linking...
- adrusi 11y ago
  When you statically link and executable, all the code from the libraries that it links to (at least the code that is directly or indirectly called by the application) is bundled into the binary. If you have 13 applications running that each statically link in 60% of a 2MB library, then you're using 15.6MB of memory where with dynamic linking you only need 2MB.
  - TheLoneWolfling 11y ago
    On the flip side, if you have a small tool that pulls in 1% of that 2MB library you only need 20KB of memory, whereas with dynamic linking you still need the full 2MB. Ditto, static linking leads to a lot more optimization opportunities, both in terms of performance and in terms of size. I would be very surprised if that 13 x 60% of a 2MB library was actually anywhere near 15.6MB.
- deleted 11y ago
  [deleted]
vezzy-fnord 11y ago
Not gonna burn your karma, however: a) You are failing to account for the bloat caused by the presence of the dynamic linker, the dynamic loader, all the code that must operate under the assumptions of a dynamically linked environment, and the various auxiliaries used to treat shared library hell issues like WinSxS and libtool. b) Most modern static linking implementations do things like sharing of text segments across processes and other dedup techniques, so they're not actually that bloated at all.
- tribaal 11y ago
  So maybe my comment stems from ignorance. Are you saying that two Golang projects sharing the same (large, for the sake of the argument) dependency would not take (size of the dependency * 2) on i.e. the Ubuntu live CD (as was the example in the fine article)?
  - ajuc 11y ago
    You have program X, and Y, both depending on big library L with 1000 symbols. X uses 1 symbol l1 from L, Y use another symbol l2 from L. With dynamic linking you need the whole library, no matter how much of it you use. With static linking (and --gc-sections -fdata-sections) you have parts of L compiled into X and Y binaries, but just the functions used (and their dependencies). It's possible that l1 and l2 share dependencies - and these will be duplicated. But still static linking is likely to be net win. Dynamic linking is mostly about maintanance.
  - orbifold 11y ago
    A statically linked binary can in principle only include code from the libraries it actually needs, so the overhead for some programs can be quite small and not as large as you might imagine. It also reduces the memory footprint at runtime.
  - TheLoneWolfling 11y ago
    Correct. I would not be surprised if those two Golang projects take less than the size of the dependency. 2x is a degenerate worst-case. With static linking, you only need to pull in things you actually need (and you can optimize from there!), whereas with dynamic linking you need to pull in the entire dependency regardless. (You only need to pull in one copy regardless of how many things are using the library, true. But you need to pull in one full copy always.) There are very few cases where you actually use the entire library in a project - and if you are indeed using an entire library in a project your project is necessarily large enough compared to the library that it's not much overhead percentage-wise.
    - tribaal 11y ago
PythonicAlpha 11y ago
I just don't agree. One of the best features I found in Golang was the statically linkage concept. There is just to much trouble with tools that come either in dozens of versions for different OS versions (Ubuntu, Debian, Mint, CentOS .... all Linux, but no Linux is the same Linux ..., it goes on: Ubuntu 12.04, 12.10, 14.x, ..., Debian Sid, Debian with and without systemd ...) While statically linking might add some bloat, I don't see, that the real bloat-factor is because of this. The real bloat-factor for example comes from programs that rely to much on template libraries (yes, I mean C++), that just unroll everything into new statements and because many even simple tasks have a multitude of solutions in the code ... each aside each other. When hundreds of developers are developing on one application, many of them invent the wheel again ...
lsaferite 11y ago
But doesn't static linking JUST link in the used features of the linked libs? Wouldn't that actually be better? Then I don't have 5 versions of a huge dynamic lib for 5 programs that all use slightly incompatible version and that only use 1 or 2 methods.
- cbd1984 11y ago
  > But doesn't static linking JUST link in the used features of the linked libs? No, not if it works like it does in C and C++. Lisp used to do that, back when Symbolics still existed; it was called tree-shaking, as in "shake the call graph tree to see what falls out". The main function calls functions FOO, BAR, BAZ, which call FROTZ and QUUX, which call... and remove everything not in that tree. Of course, this gets to be equivalent to solving the Halting Problem the moment the EVAL function is called, so if the tree-shaker found any mention of EVAL anywhere in the code (probably after it had removed the provably dead code) it punted and linked in the whole Lisp runtime. It was the only viable solution, really, especially given how creative Lispers can get...
  - adrusi 11y ago
    In C and C++ you can conservatively eliminate unused code for statically linked libraries using --gc-sections and -fdata-sections. Go uses the Plan9 compiler architecture, which means that its linker knows more about the code and is responsible for more optimizations than its Unix equivalent, so it quite effectively eliminates dead code. Of course, libraries are often highly coupled within themselves, so not much code can be eliminated, and then they might pull in additional dependencies, etc., etc.
    - cbd1984 11y ago
      Interesting. I didn't realize there had been any substantive new work done on linkers in decades.
  - lispm 11y ago
    Lisp still does that. The commercial Lisps Allegro CL and LispWorks have that as a part of their respective delivery toolkit. See for example the LispWorks documentation: 4.6 How Delivery makes an image smaller, http://www.lispworks.com/documentation/lw70/DV/html/delivery-22.htm#pgfId-854332 http://www.lispworks.com/documentation/lw70/DV/html/delivery...

by tribaal 11y ago

Galanwe 11y ago

I dont really understand the link between software bloat and static linking...

adrusi 11y ago
When you statically link and executable, all the code from the libraries that it links to (at least the code that is directly or indirectly called by the application) is bundled into the binary. If you have 13 applications running that each statically link in 60% of a 2MB library, then you're using 15.6MB of memory where with dynamic linking you only need 2MB.
- TheLoneWolfling 11y ago
  On the flip side, if you have a small tool that pulls in 1% of that 2MB library you only need 20KB of memory, whereas with dynamic linking you still need the full 2MB. Ditto, static linking leads to a lot more optimization opportunities, both in terms of performance and in terms of size. I would be very surprised if that 13 x 60% of a 2MB library was actually anywhere near 15.6MB.
deleted 11y ago
[deleted]

vezzy-fnord 11y ago

Not gonna burn your karma, however: a) You are failing to account for the bloat caused by the presence of the dynamic linker, the dynamic loader, all the code that must operate under the assumptions of a dynamically linked environment, and the various auxiliaries used to treat shared library hell issues like WinSxS and libtool. b) Most modern static linking implementations do things like sharing of text segments across processes and other dedup techniques, so they're not actually that bloated at all.

tribaal 11y ago
So maybe my comment stems from ignorance. Are you saying that two Golang projects sharing the same (large, for the sake of the argument) dependency would not take (size of the dependency * 2) on i.e. the Ubuntu live CD (as was the example in the fine article)?
- ajuc 11y ago
  You have program X, and Y, both depending on big library L with 1000 symbols. X uses 1 symbol l1 from L, Y use another symbol l2 from L. With dynamic linking you need the whole library, no matter how much of it you use. With static linking (and --gc-sections -fdata-sections) you have parts of L compiled into X and Y binaries, but just the functions used (and their dependencies). It's possible that l1 and l2 share dependencies - and these will be duplicated. But still static linking is likely to be net win. Dynamic linking is mostly about maintanance.
- orbifold 11y ago
  A statically linked binary can in principle only include code from the libraries it actually needs, so the overhead for some programs can be quite small and not as large as you might imagine. It also reduces the memory footprint at runtime.
- TheLoneWolfling 11y ago
  Correct. I would not be surprised if those two Golang projects take less than the size of the dependency. 2x is a degenerate worst-case. With static linking, you only need to pull in things you actually need (and you can optimize from there!), whereas with dynamic linking you need to pull in the entire dependency regardless. (You only need to pull in one copy regardless of how many things are using the library, true. But you need to pull in one full copy always.) There are very few cases where you actually use the entire library in a project - and if you are indeed using an entire library in a project your project is necessarily large enough compared to the library that it's not much overhead percentage-wise.
  - tribaal 11y ago

PythonicAlpha 11y ago

I just don't agree. One of the best features I found in Golang was the statically linkage concept. There is just to much trouble with tools that come either in dozens of versions for different OS versions (Ubuntu, Debian, Mint, CentOS .... all Linux, but no Linux is the same Linux ..., it goes on: Ubuntu 12.04, 12.10, 14.x, ..., Debian Sid, Debian with and without systemd ...) While statically linking might add some bloat, I don't see, that the real bloat-factor is because of this. The real bloat-factor for example comes from programs that rely to much on template libraries (yes, I mean C++), that just unroll everything into new statements and because many even simple tasks have a multitude of solutions in the code ... each aside each other. When hundreds of developers are developing on one application, many of them invent the wheel again ...

lsaferite 11y ago

But doesn't static linking JUST link in the used features of the linked libs? Wouldn't that actually be better? Then I don't have 5 versions of a huge dynamic lib for 5 programs that all use slightly incompatible version and that only use 1 or 2 methods.

cbd1984 11y ago
> But doesn't static linking JUST link in the used features of the linked libs? No, not if it works like it does in C and C++. Lisp used to do that, back when Symbolics still existed; it was called tree-shaking, as in "shake the call graph tree to see what falls out". The main function calls functions FOO, BAR, BAZ, which call FROTZ and QUUX, which call... and remove everything not in that tree. Of course, this gets to be equivalent to solving the Halting Problem the moment the EVAL function is called, so if the tree-shaker found any mention of EVAL anywhere in the code (probably after it had removed the provably dead code) it punted and linked in the whole Lisp runtime. It was the only viable solution, really, especially given how creative Lispers can get...
- adrusi 11y ago
  In C and C++ you can conservatively eliminate unused code for statically linked libraries using --gc-sections and -fdata-sections. Go uses the Plan9 compiler architecture, which means that its linker knows more about the code and is responsible for more optimizations than its Unix equivalent, so it quite effectively eliminates dead code. Of course, libraries are often highly coupled within themselves, so not much code can be eliminated, and then they might pull in additional dependencies, etc., etc.
  - cbd1984 11y ago
    Interesting. I didn't realize there had been any substantive new work done on linkers in decades.
- lispm 11y ago
  Lisp still does that. The commercial Lisps Allegro CL and LispWorks have that as a part of their respective delivery toolkit. See for example the LispWorks documentation: 4.6 How Delivery makes an image smaller, http://www.lispworks.com/documentation/lw70/DV/html/delivery-22.htm#pgfId-854332 http://www.lispworks.com/documentation/lw70/DV/html/delivery...