5 ms·

> I had observed binaries beyond 25GiB, including debug symbols. How is this possible? These companies prefer to statically build their services to speed up sta

> I had observed binaries beyond 25GiB, including debug symbols. How is this possible? These companies prefer to statically build their services to speed up startup and simplify deployment. Statically including all code in some of the world’s largest codebases is a recipe for massive binaries. I am very sympathetic to wanting nice static binaries that can be shipped around as a single artifact[0], but... surely at some point we have to ask if it's worth it? If nothing else, that feels like a little bit of a code smell; surely if your actual executable code doesn't even fit in 2GB it's time to ask if that's really one binary's worth of code or if you're actually staring at like... a dozen applications that deserve to be separate? Or get over it the other way and accept that sometimes the single artifact you ship is a tarball / OCI image / EROFS image for systemd[1] to mount+run / self-extracting archive[2] / ... [0] Seriously, one of my background projects right now is trying to figure out if it's really that hard to make fat ELF binaries. [1] https://systemd.io/PORTABLE_SERVICES/ https://systemd.io/PORTABLE_SERVICES/ [2] https://justine.lol/ape.html https://justine.lol/ape.html > "PKZIP Executables Make Pretty Good Containers"

jmmv 10mo ago
This is something that always bothered me while I was working at Google too: we had an amazing compute and storage infrastructure that kept getting crazier and crazier over the years (in terms of performance, scalability and redundancy) but everything in operations felt slow because of the massive size of binaries. Running a command line binary? Slow. Building a binary for deployment? Slow. Deploying a binary? Slow. The answer to an ever-increasing size of binaries was always "let's make the infrastructure scale up!" instead of "let's... not do this crazy thing maybe?". By the time I left, there were some new initiatives towards the latter and the feeling that "maybe we should have put limits much earlier" but retrofitting limits into the existing bloat was going to be exceedingly difficult.
- joatmon-snoo 10mo ago
  There's a lot of tooling built on static binaries: - google-wide profiling: the core C++ team can collect data on how much of fleet CPU % is spent in absl::flat_hash_map re-bucketing (you can find papers on this publicly) - crashdump telemetry - dapper stack trace -> codesearch Borg literally had to pin the bash version because letting the bash version float caused bugs. I can't imagine how much harder debugging L7 proxy issues would be if I had to follow a .so rabbit hole. I can believe shrinking binary size would solve a lot of problems, and I can imagine ways to solve the .so versioning problem, but for every problem you mention I can name multiple other probable causes (eg was startup time really execvp time, or was it networked deps like FFs).
  - Filligree 10mo ago
    There’s no way my proxy binary actually requires 25GB of code, or even the 3GB it is. Sounds to me like the answer is a tree shaker.
    - Sesse__ 10mo ago
      Google implemented the C++ equivalent of a tree shaker in their build system around 2009.
      - setheron 10mo ago
        the front-end services to be "fast" AFAIK probably include nearly all the services you need to avoid hops -- so you can't really shake that much away.
  - deleted 10mo ago
    [deleted]
  - MaskRay 10mo ago
    We are missing tooling to partition a huge binary into a few larger shared objects. As my https://maskray.me/blog/2023-05-14-relocation-overflow-and-code-models#:~:text=static https://maskray.me/blog/2023-05-14-relocation-overflow-and-c... (linked by author, thanks! But I maintain lld/ELF instead of "wrote" it - it's engineer work of many folks) Quoting the relevant paragraphs below: ## Static linking In this section, we will deviate slightly from the main topic to discuss static linking. By including all dependencies within the executable itself, it can run without relying on external shared objects. This eliminates the potential risks associated with updating dependencies separately. Certain users prefer static linking or mostly static linking for the sake of deployment convenience and performance aspects: * Link-time optimization is more effective when all dependencies are known. Providing shared object information during executable optimization is possible, but it may not be a worthwhile engineering effort. * Profiling techniques are more efficient dealing with one single executable. * The traditional ELF dynamic linking approach incurs overhead to support [symbol interposition](https://maskray.me/blog/2021-05-16-elf-interposition-and-bsymbolic https://maskray.me/blog/2021-05-16-elf-interposition-and-bsy...). * Dynamic linking involves PLT and GOT, which can introduce additional overhead. Static linking eliminates the overhead. * Loading libraries in the dynamic loader has a time complexity `O(|libs|^2*|libname|)`. The existing implementations are designed to handle tens of shared objects, rather than a thousand or more. Furthermore, the current lack of techniques to partition an executable into a few larger shared objects, as opposed to numerous smaller shared objects, exacerbates the overhead issue. In scenarios where the distributed program contains a significant amount of code (related: software bloat), employing full or mostly static linking can result in very large executable files. Consequently, certain relocations may be close to the distance limit, and even a minor disruption (e.g. add a function or introduce a dependency) can trigger relocation overflow linker errors.
- lenkite 10mo ago
  Maybe I am missing something, but why didn't they just leverage dynamic libraries ?
  - tmoertel 10mo ago
    One reason is that using static binaries greatly simplifies the problem of establishing Binary Provenance, upon which security claims and many other important things rely. In environments like Google’s it's important to know that what you have deployed to production is exactly what you think it is. See for more: https://google.github.io/building-secure-and-reliable-systems/raw/ch14.html https://google.github.io/building-secure-and-reliable-system...
    - inkyoto 10mo ago
      > One reason is that using static binaries greatly simplifies the problem of establishing Binary Provenance, upon which security claims and many other important things rely. It depends. If it is a vulnerability stemming from libc, then every single binary has to be re-linked and redeployed, which can lead to a situation where something has been accidentally left out due to a unaccounted for artefact. One solution could be bundling the binary or related multiple binaries with the operating system image but that would incur a multidimensional overhead that would be unacceptable for most people and then we would be talking about «an application binary statically linked into the operating system» so to speak.
      - tmoertel 10mo ago
        > If it is a vulnerability stemming from libc, then every single binary has to be re-linked and redeployed, which can lead to a situation where something has been accidentally left out due to a unaccounted for artefact. The whole point of Binary Provenance is that there are no unaccounted-for artifacts: Every build should produce binary provenance describing exactly how a given binary artifact was built: the inputs, the transformation, and the entity that performed the build. So, to use your example, you'll always know which artefacts were linked against that bad version of libc. See https://google.github.io/building-secure-and-reliable-systems/raw/ch14.html#:~:text=to%20insider%20risk.-,Binary%20Provenance,-Every%20build%20should https://google.github.io/building-secure-and-reliable-system...
- darubedarob 10mo ago
  I think google of all companies could build a good autostripper reducing binaries by adding partial load assembly on misses. It cant be much slower then shovelling a full monorepo assembly plus symbols into ram.
  - loeg 10mo ago
    The low-hanging fruit is just not shipping the debuginfo, of course.
    - usefulcat 10mo ago
      Is compressed debug info a thing? It seems likely to compress well, and if it's rarely used then it might be a worthwhile thing to do?
      - loeg 10mo ago
        It is: https://maskray.me/blog/2022-01-23-compressed-debug-sections https://maskray.me/blog/2022-01-23-compressed-debug-sections But the compression ratio isn't magical (approx. 1:0.25, for both zlib and zstd in the examples given). You'd probably still want to set aside debuginfo in separate files.
      - Gibbon1 10mo ago
        Small brained primate comment. With embedded firmware you only flash the .text and and flash to the device. But you still can debug using the .elf file. In my case if I get a bus fault I'll pull the offending address off the stack and use bintools and the .elf to show me who was naughty. I think if you have a crash dump you should be able to make sense of things as long as you keep the unstripped .elf file around.
- bfrog 10mo ago
  Sounds like Google could really use Nix
forrestthewoods 10mo ago
If you have 25gb of executables then I don’t think it matters if that’s one binary executable or a hundred. Something has gone horribly horribly wrong. I don’t think I’ve ever seen a 4gb binary yet. I have seen instances where a PDB file hit 4gb and that caused problems. Debug symbols getting that large is totally plausible. I’m ok with that at least.
- wolfi1 10mo ago
  I did, it was a Spring Boot fat jar with a NLP, I had to deploy it to the biggest instance AWS could offer, the costs were enormous
  - selkin 10mo ago
    Java bytecode is always dynamically linked.
    - wolfi1 10mo ago
      still, if I remember correctly I had to reserve 6gig of memory so that the jvm could actually start
- throwawaymobule 10mo ago
  A few ps3 games I've seen had 4GB or more binaries. This was a problem because code signing meant it needed to be completely replaced by updates.
  - swiftcoder 10mo ago
    > A few ps3 games I've seen had 4GB or more binaries. Is this because they are embedding assets into the binary? I find it hard to believe anyone was carrying around enough code to fill 4GB in the PS3 era...
    - throwawaymobule 10mo ago
      I assume so, there were rarely any other files on the disc in this case. It varied between games, one of the battlefields (3 or bad company 2) was what I was thinking of. It generally improved with later releases. The 4GB file size was significant, since it meant I couldn't run them from a backup on a fat32 usb drive. There are workarounds for many games nowadays.
shevy-java 10mo ago
> https://systemd.io/PORTABLE_SERVICES/ https://systemd.io/PORTABLE_SERVICES/ Systemd and portable?
- yjftsjthsd-h 10mo ago
  Portable across systemd/Linux systems, yes:)
jcelerier 10mo ago
What's wild to me is not using -gsplit-dwarf to have separate debug info and "normal-sized" binaries
- jeffbee 10mo ago
  Google contributed the code, and the entire concept, of DWARF fission to both GCC and LLVM. This suggests that rather than overlooking something obvious that they'll be embarrassed to learn on HN, they were aware of the issues and were using the solutions before you'd even heard of them.
  - sionisrecur 10mo ago
    A case of the left hand not knowing what the right hand is doing?
    - jeffbee 10mo ago
      There's no contradiction, no missing link in the facts of the story. They have a huge program, it is 2GiB minus epsilon of .text, and a much larger amount of DWARF stuff. The article is about how to use different code models to potentially go beyond 2GiB of text, and the size of the DWARF sections is irrelevant trivia.
      - jcelerier 10mo ago
        > They have a huge program, it is 2GiB minus epsilon of .text, but the article says 25+GiB including debug symbols, in a single binary? also, I appreciate your enthusiasm in assuming that because some people do something in an organization, it is applied consistently everywhere. Hell, if it were microsoft other departments would try to shoot down the "debug tooling optimization" dpt
        jeffbee 10mo ago
        ELF is just a container format and you can put literally anything into one of its sections. Whether the DWARF sections are in "the binary" or in another named file is really quite beside the point.

jmmv 10mo ago

This is something that always bothered me while I was working at Google too: we had an amazing compute and storage infrastructure that kept getting crazier and crazier over the years (in terms of performance, scalability and redundancy) but everything in operations felt slow because of the massive size of binaries. Running a command line binary? Slow. Building a binary for deployment? Slow. Deploying a binary? Slow. The answer to an ever-increasing size of binaries was always "let's make the infrastructure scale up!" instead of "let's... not do this crazy thing maybe?". By the time I left, there were some new initiatives towards the latter and the feeling that "maybe we should have put limits much earlier" but retrofitting limits into the existing bloat was going to be exceedingly difficult.

joatmon-snoo 10mo ago
There's a lot of tooling built on static binaries: - google-wide profiling: the core C++ team can collect data on how much of fleet CPU % is spent in absl::flat_hash_map re-bucketing (you can find papers on this publicly) - crashdump telemetry - dapper stack trace -> codesearch Borg literally had to pin the bash version because letting the bash version float caused bugs. I can't imagine how much harder debugging L7 proxy issues would be if I had to follow a .so rabbit hole. I can believe shrinking binary size would solve a lot of problems, and I can imagine ways to solve the .so versioning problem, but for every problem you mention I can name multiple other probable causes (eg was startup time really execvp time, or was it networked deps like FFs).
- Filligree 10mo ago
  There’s no way my proxy binary actually requires 25GB of code, or even the 3GB it is. Sounds to me like the answer is a tree shaker.
  - Sesse__ 10mo ago
    Google implemented the C++ equivalent of a tree shaker in their build system around 2009.
    - setheron 10mo ago
      the front-end services to be "fast" AFAIK probably include nearly all the services you need to avoid hops -- so you can't really shake that much away.
- deleted 10mo ago
  [deleted]
- MaskRay 10mo ago
  We are missing tooling to partition a huge binary into a few larger shared objects. As my https://maskray.me/blog/2023-05-14-relocation-overflow-and-code-models#:~:text=static https://maskray.me/blog/2023-05-14-relocation-overflow-and-c... (linked by author, thanks! But I maintain lld/ELF instead of "wrote" it - it's engineer work of many folks) Quoting the relevant paragraphs below: ## Static linking In this section, we will deviate slightly from the main topic to discuss static linking. By including all dependencies within the executable itself, it can run without relying on external shared objects. This eliminates the potential risks associated with updating dependencies separately. Certain users prefer static linking or mostly static linking for the sake of deployment convenience and performance aspects: * Link-time optimization is more effective when all dependencies are known. Providing shared object information during executable optimization is possible, but it may not be a worthwhile engineering effort. * Profiling techniques are more efficient dealing with one single executable. * The traditional ELF dynamic linking approach incurs overhead to support [symbol interposition](https://maskray.me/blog/2021-05-16-elf-interposition-and-bsymbolic https://maskray.me/blog/2021-05-16-elf-interposition-and-bsy...). * Dynamic linking involves PLT and GOT, which can introduce additional overhead. Static linking eliminates the overhead. * Loading libraries in the dynamic loader has a time complexity `O(|libs|^2*|libname|)`. The existing implementations are designed to handle tens of shared objects, rather than a thousand or more. Furthermore, the current lack of techniques to partition an executable into a few larger shared objects, as opposed to numerous smaller shared objects, exacerbates the overhead issue. In scenarios where the distributed program contains a significant amount of code (related: software bloat), employing full or mostly static linking can result in very large executable files. Consequently, certain relocations may be close to the distance limit, and even a minor disruption (e.g. add a function or introduce a dependency) can trigger relocation overflow linker errors.
lenkite 10mo ago
Maybe I am missing something, but why didn't they just leverage dynamic libraries ?
- tmoertel 10mo ago
  One reason is that using static binaries greatly simplifies the problem of establishing Binary Provenance, upon which security claims and many other important things rely. In environments like Google’s it's important to know that what you have deployed to production is exactly what you think it is. See for more: https://google.github.io/building-secure-and-reliable-systems/raw/ch14.html https://google.github.io/building-secure-and-reliable-system...
  - inkyoto 10mo ago
    > One reason is that using static binaries greatly simplifies the problem of establishing Binary Provenance, upon which security claims and many other important things rely. It depends. If it is a vulnerability stemming from libc, then every single binary has to be re-linked and redeployed, which can lead to a situation where something has been accidentally left out due to a unaccounted for artefact. One solution could be bundling the binary or related multiple binaries with the operating system image but that would incur a multidimensional overhead that would be unacceptable for most people and then we would be talking about «an application binary statically linked into the operating system» so to speak.
    - tmoertel 10mo ago
      > If it is a vulnerability stemming from libc, then every single binary has to be re-linked and redeployed, which can lead to a situation where something has been accidentally left out due to a unaccounted for artefact. The whole point of Binary Provenance is that there are no unaccounted-for artifacts: Every build should produce binary provenance describing exactly how a given binary artifact was built: the inputs, the transformation, and the entity that performed the build. So, to use your example, you'll always know which artefacts were linked against that bad version of libc. See https://google.github.io/building-secure-and-reliable-systems/raw/ch14.html#:~:text=to%20insider%20risk.-,Binary%20Provenance,-Every%20build%20should https://google.github.io/building-secure-and-reliable-system...
darubedarob 10mo ago
I think google of all companies could build a good autostripper reducing binaries by adding partial load assembly on misses. It cant be much slower then shovelling a full monorepo assembly plus symbols into ram.
- loeg 10mo ago
  The low-hanging fruit is just not shipping the debuginfo, of course.
  - usefulcat 10mo ago
    Is compressed debug info a thing? It seems likely to compress well, and if it's rarely used then it might be a worthwhile thing to do?
    - loeg 10mo ago
      It is: https://maskray.me/blog/2022-01-23-compressed-debug-sections https://maskray.me/blog/2022-01-23-compressed-debug-sections But the compression ratio isn't magical (approx. 1:0.25, for both zlib and zstd in the examples given). You'd probably still want to set aside debuginfo in separate files.
    - Gibbon1 10mo ago
      Small brained primate comment. With embedded firmware you only flash the .text and and flash to the device. But you still can debug using the .elf file. In my case if I get a bus fault I'll pull the offending address off the stack and use bintools and the .elf to show me who was naughty. I think if you have a crash dump you should be able to make sense of things as long as you keep the unstripped .elf file around.
bfrog 10mo ago
Sounds like Google could really use Nix

forrestthewoods 10mo ago

If you have 25gb of executables then I don’t think it matters if that’s one binary executable or a hundred. Something has gone horribly horribly wrong. I don’t think I’ve ever seen a 4gb binary yet. I have seen instances where a PDB file hit 4gb and that caused problems. Debug symbols getting that large is totally plausible. I’m ok with that at least.

wolfi1 10mo ago
I did, it was a Spring Boot fat jar with a NLP, I had to deploy it to the biggest instance AWS could offer, the costs were enormous
- selkin 10mo ago
  Java bytecode is always dynamically linked.
  - wolfi1 10mo ago
    still, if I remember correctly I had to reserve 6gig of memory so that the jvm could actually start
throwawaymobule 10mo ago
A few ps3 games I've seen had 4GB or more binaries. This was a problem because code signing meant it needed to be completely replaced by updates.
- swiftcoder 10mo ago
  > A few ps3 games I've seen had 4GB or more binaries. Is this because they are embedding assets into the binary? I find it hard to believe anyone was carrying around enough code to fill 4GB in the PS3 era...
  - throwawaymobule 10mo ago
    I assume so, there were rarely any other files on the disc in this case. It varied between games, one of the battlefields (3 or bad company 2) was what I was thinking of. It generally improved with later releases. The 4GB file size was significant, since it meant I couldn't run them from a backup on a fat32 usb drive. There are workarounds for many games nowadays.

shevy-java 10mo ago

> https://systemd.io/PORTABLE_SERVICES/ https://systemd.io/PORTABLE_SERVICES/ Systemd and portable?

yjftsjthsd-h 10mo ago
Portable across systemd/Linux systems, yes:)

jcelerier 10mo ago

What's wild to me is not using -gsplit-dwarf to have separate debug info and "normal-sized" binaries

jeffbee 10mo ago
Google contributed the code, and the entire concept, of DWARF fission to both GCC and LLVM. This suggests that rather than overlooking something obvious that they'll be embarrassed to learn on HN, they were aware of the issues and were using the solutions before you'd even heard of them.
- sionisrecur 10mo ago
  A case of the left hand not knowing what the right hand is doing?
  - jeffbee 10mo ago
    There's no contradiction, no missing link in the facts of the story. They have a huge program, it is 2GiB minus epsilon of .text, and a much larger amount of DWARF stuff. The article is about how to use different code models to potentially go beyond 2GiB of text, and the size of the DWARF sections is irrelevant trivia.
    - jcelerier 10mo ago
      > They have a huge program, it is 2GiB minus epsilon of .text, but the article says 25+GiB including debug symbols, in a single binary? also, I appreciate your enthusiasm in assuming that because some people do something in an organization, it is applied consistently everywhere. Hell, if it were microsoft other departments would try to shoot down the "debug tooling optimization" dpt
      - jeffbee 10mo ago
        ELF is just a container format and you can put literally anything into one of its sections. Whether the DWARF sections are in "the binary" or in another named file is really quite beside the point.