6 ms·

Run htop or similar, sort by "shared memory" column and see how much more memory you'd need per process if shared linking did not exist. I think the author's u

Run htop or similar, sort by "shared memory" column and see how much more memory you'd need per process if shared linking did not exist. I think the author's using a wrong method to make a point. Dynamic linking feels out of place for most long-running server-side apps (typical SaaS workload). One can argue that in a mostly CLI-environment there's also not much benefit. But even an empty Ubuntu desktop runs ~400 processes and dynamic linking makes perfect sense. libc alone would have to exist in hundreds reincarnations consuming hundred+ megabytes of RAM and I'm not even talking about much, much heavier GTK+ / cairo / freetype / etc libraries needed for GUI applications.

ddevault 6y ago
That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM. Statically linking would, as the article shows, only use on average about 4% of the symbols available from the libraries, and the majority of this would not end up in RAM with your statically linked binary. And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.
- loeg 6y ago
  > Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. Come on. You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data. Even if you had measured the figure you're accusing GP of ignoring, the tone of your remark is just aggressively condescending and inappropriate. Tone it down. To address your other claims: > Shared memory is page-aligned entire libraries dropped into RAM. There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. > the majority of [the small fraction of static linked library used] would not end up in RAM with your statically linked binary. Huh? Why do you claim that? > And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. I think that claim is plausible! But it wasn't an option presented on your blog post, nor was it discussed by GP. Prior to that comment, discussion was only around 100% vs 0% dynamic linking.
  - bjourne 6y ago
    > There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. But most code isn't performance critical. Thus trying to align functions to page boundaries is just wasting memory. Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage.
    - MaulingMonkey 6y ago
      Not only is it wasting memory, but I wouldn't be convinced it'd not be outright hurting performance by increasing cache line aliasing, increasing TLB overhead and misses/invalidations! Avoiding page alignment can be a performance gain! https://pvk.ca/Blog/2012/07/30/binary-search-is-a-pathological-case-for-caches/ https://pvk.ca/Blog/2012/07/30/binary-search-is-a-pathologic... Like, sure, mapping your executable's code section in at a page boundary is probably fine, but I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy.
      - loeg 6y ago
        > I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy. Yes — which is why no one does that. I don't know where bjourne came up with the idea.
    - loeg 6y ago
      > But most code isn't performance critical. Wasting TLB slots on your unimportant code still pessimizes your hot code. > Thus trying to align functions to page boundaries is just wasting memory. No one aligns individual functions to page boundaries; you align the entire loadable code segment to a page (or preferably, superpage) boundary. > Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage. This is a different kind of optimization (avoiding cache line contention) than I was talking about (optimizing TLB slot use). Yes, forced page-alignment doesn't help cache line contention anymore than forced cacheline-alignment. But that's irrelevant for code, generally: (outside of self-modifying code, which is extremely uncommon) code doesn't share a cacheline with memory that will be mutated and thus doesn't contend in that way.
- cryptonector 6y ago
  @ddevault: performance is only part of the story. The poverty of semantics of C static linking is atrocious -- it could be fixed, but until it is we should not statically link C programs.
- old-gregg 6y ago
  That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM First of all, I seriously doubt it (haven't looked closely, but if library-loading is similar to mmap, it should only count actually used segments). But that shouldn't even matter, as most of these libraries are fully utilized. All of libc is used collectively by 400+ processes, that is also true for the complex multi-layer GUI machinery. The output of ldd $(which gnome-calculator) is terrifying, run it under a profiler and see how many functions get hit, you'll be amazed. Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. The coldness of your numbers did not impress. And calling a reasonable engineering trade-off "cargo cult" doesn't get you any points either. Static linking is better than dynamic linking. Sometimes. And vice versa. That's how engineering is different from science, there are no absolute truths, only trade-offs.
jnwatson 6y ago
Go executables are statically linked. It makes deployment a breeze. I think you overestimate how much saving you get from dynamically linking libc. Each executable uses only a small portion of libc, so the average savings is going to be in the handful of kilobytes per executable.
- glouwbug 6y ago
  True, but a couple github imports and 10 lines of code generates a 100mb binary. But, to be fair, I guess we're okay with shipping huge binaries now a days because we're literally shipping whole environments with docker anyway.
  - yjftsjthsd-h 6y ago
    To be fair, you're not really supposed to be shipping a full system in a docker image if you can help it; you're supposed to layer your application over the smallest base that will support it (whether that's scratch, distroless, alpine, or a minimal debian base). Of course, I'll be the first to agree that "supposed to" and reality have little in common; if I had a dollar for every time I've seen a final image that still included the whole compiler chain...
    - majewsky 6y ago
      In my experience, most who build fat images with the compiler chain and everything do so because they're simply not aware that multi-stage builds are a thing now.
      - yjftsjthsd-h 6y ago
        Yep. And just general awareness of how Docker works and how to best use it. And that's not a knock on them; it's a whole topic unto itself, and lots of these folks are developers who are just trying to get their stuff working without having to go off and learn all about Docker, and I have a hard time blaming them for that. Which, of course, is why they keep a handful of us sysadmin folks on staff to help tidy up;)
a1369209993 6y ago
> Run htop or similar, sort by "shared memory" The top two entries are 80 and 36 kB respectively. RES is 2.3 giga-bytes (over four orders of magnitude larger) between them. Even multiplying the top SHR by your ~400 processes gives 32 MB (still two orders of magnitude off). That is not a counter argument; that is a agreement that dynamic linking is useless. Edit: RES, not VIRT.

ddevault 6y ago

That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM. Statically linking would, as the article shows, only use on average about 4% of the symbols available from the libraries, and the majority of this would not end up in RAM with your statically linked binary. And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. Put the cold hard numbers right in front of someone's face and still the cargo cult wins out.

loeg 6y ago
> Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. Come on. You did not actually measure the figure GP mentioned and which you are disputing. Your methodology and assumption — that 4% external symbol use translates into 4% size used — is a plausible guess, but you haven't supported it with data. Even if you had measured the figure you're accusing GP of ignoring, the tone of your remark is just aggressively condescending and inappropriate. Tone it down. To address your other claims: > Shared memory is page-aligned entire libraries dropped into RAM. There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. > the majority of [the small fraction of static linked library used] would not end up in RAM with your statically linked binary. Huh? Why do you claim that? > And if you used a more selective approach, dynamically linking to no more than perhaps a dozen high-impact libraries and statically linking the rest, you'd get a lot of the benefits and few of the drawbacks. I think that claim is plausible! But it wasn't an option presented on your blog post, nor was it discussed by GP. Prior to that comment, discussion was only around 100% vs 0% dynamic linking.
- bjourne 6y ago
  > There is a good reason to page- or superpage-align code generally; it burns some virtual memory but reduces TLB overhead and therefore misses / invalidations, which are very costly. You would want to do the same with executable code in a static-linked binary. But most code isn't performance critical. Thus trying to align functions to page boundaries is just wasting memory. Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage.
  - MaulingMonkey 6y ago
    Not only is it wasting memory, but I wouldn't be convinced it'd not be outright hurting performance by increasing cache line aliasing, increasing TLB overhead and misses/invalidations! Avoiding page alignment can be a performance gain! https://pvk.ca/Blog/2012/07/30/binary-search-is-a-pathological-case-for-caches/ https://pvk.ca/Blog/2012/07/30/binary-search-is-a-pathologic... Like, sure, mapping your executable's code section in at a page boundary is probably fine, but I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy.
    - loeg 6y ago
      > I think trying to align individual functions to page boundaries would be a counterproductive mistake as a general strategy. Yes — which is why no one does that. I don't know where bjourne came up with the idea.
  - loeg 6y ago
    > But most code isn't performance critical. Wasting TLB slots on your unimportant code still pessimizes your hot code. > Thus trying to align functions to page boundaries is just wasting memory. No one aligns individual functions to page boundaries; you align the entire loadable code segment to a page (or preferably, superpage) boundary. > Even in performance critical code, aligning to cache line sizes is enough and aligning to page boundaries doesn't provide any advantage. This is a different kind of optimization (avoiding cache line contention) than I was talking about (optimizing TLB slot use). Yes, forced page-alignment doesn't help cache line contention anymore than forced cacheline-alignment. But that's irrelevant for code, generally: (outside of self-modifying code, which is extremely uncommon) code doesn't share a cacheline with memory that will be mutated and thus doesn't contend in that way.
cryptonector 6y ago
@ddevault: performance is only part of the story. The poverty of semantics of C static linking is atrocious -- it could be fixed, but until it is we should not statically link C programs.
old-gregg 6y ago
That is an extremely misleading figure. Shared memory is page-aligned entire libraries dropped into RAM First of all, I seriously doubt it (haven't looked closely, but if library-loading is similar to mmap, it should only count actually used segments). But that shouldn't even matter, as most of these libraries are fully utilized. All of libc is used collectively by 400+ processes, that is also true for the complex multi-layer GUI machinery. The output of ldd $(which gnome-calculator) is terrifying, run it under a profiler and see how many functions get hit, you'll be amazed. Put the cold hard numbers right in front of someone's face and still the cargo cult wins out. The coldness of your numbers did not impress. And calling a reasonable engineering trade-off "cargo cult" doesn't get you any points either. Static linking is better than dynamic linking. Sometimes. And vice versa. That's how engineering is different from science, there are no absolute truths, only trade-offs.

jnwatson 6y ago

Go executables are statically linked. It makes deployment a breeze. I think you overestimate how much saving you get from dynamically linking libc. Each executable uses only a small portion of libc, so the average savings is going to be in the handful of kilobytes per executable.

glouwbug 6y ago
True, but a couple github imports and 10 lines of code generates a 100mb binary. But, to be fair, I guess we're okay with shipping huge binaries now a days because we're literally shipping whole environments with docker anyway.
- yjftsjthsd-h 6y ago
  To be fair, you're not really supposed to be shipping a full system in a docker image if you can help it; you're supposed to layer your application over the smallest base that will support it (whether that's scratch, distroless, alpine, or a minimal debian base). Of course, I'll be the first to agree that "supposed to" and reality have little in common; if I had a dollar for every time I've seen a final image that still included the whole compiler chain...
  - majewsky 6y ago
    In my experience, most who build fat images with the compiler chain and everything do so because they're simply not aware that multi-stage builds are a thing now.
    - yjftsjthsd-h 6y ago
      Yep. And just general awareness of how Docker works and how to best use it. And that's not a knock on them; it's a whole topic unto itself, and lots of these folks are developers who are just trying to get their stuff working without having to go off and learn all about Docker, and I have a hard time blaming them for that. Which, of course, is why they keep a handful of us sysadmin folks on staff to help tidy up;)

a1369209993 6y ago

> Run htop or similar, sort by "shared memory" The top two entries are 80 and 36 kB respectively. RES is 2.3 giga-bytes (over four orders of magnitude larger) between them. Even multiplying the top SHR by your ~400 processes gives 32 MB (still two orders of magnitude off). That is not a counter argument; that is a agreement that dynamic linking is useless. Edit: RES, not VIRT.