6 ms·
Huge Binaries
- gerikson 10mo agoThe HN de-sensationalize algo for submission titles needs tweaking. Original title is simply "Huge Binaries".
- acosmism 10mo agoagreed. Binaries is a bit too sensational for my taste. this can be further optimized.
- binaryturtle 10mo ago"Bins"? :)
- fuzzfactor 10mo ago"Files So Big They Might As Well Be Trinaries".
- doubletwoyou 10mo ago25 GiB for a single binary sounds horrifying at some point surely some dynamic linking is warranted
- nneonneo 10mo agoTo be fair, this is with debug symbols. Debug builds of Chrome were in the 5GB range several years ago; no doubt that’s increased since then. I can remember my poor laptop literally running out of RAM during the linking phase due to the sheer size of the object files being linked. Why are debug symbols so big? For C++, they’ll include detailed type information for every instantiation of every type everywhere in your program, including the types of every field (recursively), method signatures, etc. etc., along with the types and locations of local variables in every method (updated on every spill and move), line number data, etc. etc. for every specialization of every function. This produces a lot of data even for “moderate”-sized projects. Worse: for C++, you don’t win much through dynamic linking because dynamically linking C++ libraries sucks so hard. Templates defined in header files can’t easily be put in shared libraries; ABI variations mean that dynamic libraries generally have to be updated in sync; and duplication across modules is bound to happen (thanks to inlined functions and templates). A single “stuck” or outdated .so might completely break a deployment too, which is a much worse situation than deploying a single binary (either you get a new version or an old one, not a broken service).
- tempay 10mo agoI’ve seen LLVM dependent builds hit well over 30GB. At that point it started breaking several package managers.
- 01HNNWZ0MV43FF 10mo agoI've hit the same thing in Rust, probably for the same reasons. Isn't the simple solution to use detached debug files? I think Windows and Linux both support them. That's how phones like Android and iOS get useful crash reports out of small binaries, they just upload the stack trace and some service like Sentry translates that back into source line numbers. (It's easy to do manually too) I'm surprised the author didn't mention it first. A 25 GB exe might be 1 GB of code and 24 GB of debug crud.
- dwattttt 10mo ago> I think Windows and Linux both support them. Detached debug files has been the default (only?) option in MS's compiler since at least the 90s. I'm not sure at what point it became hip to do that around Linux.
- 0xbadcafebee 10mo agoTo be fair, they worked at Google, their engineering decisions are not normal. They might just decide that 25 GiB binaries are worth a 0.25% speedup at start time, potentially resulting in tens of millions of dollars' worth of difference. Nobody should do things the way Google does, but it's interesting to think about.
- flohofwoe 10mo agoThe overall size wouldn't get smaller just because it is dynamically linked, on the contrary (because DLLs are a dead code elimination barrier). 25 GB is insane either way, something must have gone horribly wrong very early in the development process (also why, even ship with debug information included, that doesn't make sense in the first place).
- dilyevsky 10mo agoWon't make a bit of difference because everything is in a sort of container (not Docker) anyway. Unless you're suggesting those libraries to be distributed as base image to every possible Borg machine your app can run on which is an obvious non-starter.
- yjftsjthsd-h 10mo ago> I had observed binaries beyond 25GiB, including debug symbols. How is this possible? These companies prefer to statically build their services to speed up startup and simplify deployment. Statically including all code in some of the world’s largest codebases is a recipe for massive binaries. I am very sympathetic to wanting nice static binaries that can be shipped around as a single artifact[0], but... surely at some point we have to ask if it's worth it? If nothing else, that feels like a little bit of a code smell; surely if your actual executable code doesn't even fit in 2GB it's time to ask if that's really one binary's worth of code or if you're actually staring at like... a dozen applications that deserve to be separate? Or get over it the other way and accept that sometimes the single artifact you ship is a tarball / OCI image / EROFS image for systemd[1] to mount+run / self-extracting archive[2] / ... [0] Seriously, one of my background projects right now is trying to figure out if it's really that hard to make fat ELF binaries. [1] https://systemd.io/PORTABLE_SERVICES/ https://systemd.io/PORTABLE_SERVICES/ [2] https://justine.lol/ape.html https://justine.lol/ape.html > "PKZIP Executables Make Pretty Good Containers"
- jmmv 10mo agoThis is something that always bothered me while I was working at Google too: we had an amazing compute and storage infrastructure that kept getting crazier and crazier over the years (in terms of performance, scalability and redundancy) but everything in operations felt slow because of the massive size of binaries. Running a command line binary? Slow. Building a binary for deployment? Slow. Deploying a binary? Slow. The answer to an ever-increasing size of binaries was always "let's make the infrastructure scale up!" instead of "let's... not do this crazy thing maybe?". By the time I left, there were some new initiatives towards the latter and the feeling that "maybe we should have put limits much earlier" but retrofitting limits into the existing bloat was going to be exceedingly difficult.
- joatmon-snoo 10mo agoThere's a lot of tooling built on static binaries: - google-wide profiling: the core C++ team can collect data on how much of fleet CPU % is spent in absl::flat_hash_map re-bucketing (you can find papers on this publicly) - crashdump telemetry - dapper stack trace -> codesearch Borg literally had to pin the bash version because letting the bash version float caused bugs. I can't imagine how much harder debugging L7 proxy issues would be if I had to follow a .so rabbit hole. I can believe shrinking binary size would solve a lot of problems, and I can imagine ways to solve the .so versioning problem, but for every problem you mention I can name multiple other probable causes (eg was startup time really execvp time, or was it networked deps like FFs).
- a_t48 10mo agoI've seen terrible, terrible binary sizes with Eigen + debug symbols, due to how Eigen lazy evaluation works (I think). Every math expression ends up as a new template instantiation.
- forrestthewoods 10mo agoEigen is one of the worst libraries when it comes to both exe size and compile times. <shudder>
- a_t48 10mo agoIn terms of compile times, boost geometry is somehow worse. You're encouraged to import boost/geometry.hpp, which includes every module, which stalls compile times by several seconds just to parse all the templates. It's not terrible if you include just the headers you need, but that's not the "default" that most people use.
- forrestthewoods 10mo agoboost is on my “do not ever use ever oh my god what are you doing stop it” list. It’s so bad.
- a_t48 10mo agoSame.
- stncls 10mo ago> The simplest solution however is to use -mcmodel=large which changes all the relative CALL instructions to absolute JMP. Makes sense, but in the assembly output just after, there is not a single JMP instruction. Instead, CALL <immediate> is replaced with putting the address in a 64-bit register, then CALL <register>, which makes even more sense. But why mention the JMP thing then? Is it a mistake or am I missing something? (I know some calls are replaced by JMP, but that's done regardless of -mcmodel=large)
- dwattttt 10mo agoI would assume loose language, referring to a CALL as a JMP. However of the two reasons given to dislike the large code model, register pressure isn't relevant to that particular snippet. It's performing a call, ABIs define registers that are not preserved over calls; writing the destination to one of those won't affect register pressure.
- loeg 10mo agoI think the author is just noting that the construction is similar to an 8-byte JMP instruction. The text now reads: > The simplest solution however is to use -mcmodel=large which changes all the relative CALL instructions to absolute 64bit ones; kind of like a JMP. (We still need to use CALL in order to push a return address.)
- 10000truths 10mo agoDebug symbol size shouldn't be influencing relocation jump distances - debug info has its own ELF section. Regardless of whether you're FAANG or not, nothing you're running should require an executable with a 2 GB large .text section. If you're bumping into that limit, then your build process likely lacks dead code elimination in the linking step. You should be using LTO for release builds. Even the traditional solution (compile your object files with -ffunction-sections and link with --gc-sections) does a good job of culling dead code at function-level granularity.
- yablak 10mo agoFAANGs we're deeply involved in designing LTO. See, e.g., https://research.google/pubs/thinlto-scalable-and-incremental-lto/ https://research.google/pubs/thinlto-scalable-and-incrementa... And other refs. And yet...
- jeffbee 10mo agoGoogle also uses identical code folding. It's a pretty silly idea that a shop that big doesn't know about the compiler flags.
- Orphis 10mo agoGoogle is made of many thousands of individuals. Some experts will be aware of all those, some won't. In my team, many didn't know about those details as they were handled by other builds teams for specific products or entire domains at once. But since each product in some different domains had to actively enable those optimizations for themselves, they were occasionally forgotten, and I found a few in the app I worked for (but not directly on).
- jeffbee 10mo agoICF seems like a good one to keep in the box of flags people don't know about because like everything in life it's a tradeoff and keeping that one problematic artifact under 2GiB is pretty much the only non-debatable use case for it.
- deleted 10mo ago[deleted]
- yablak 10mo ago> We would like to keep our small code-model. What other strategies can we pursue? Move all the hot BBs near each other, right? Facebook's solution: https://github.com/llvm/llvm-project/blob/main/bolt%2FREADME.md https://github.com/llvm/llvm-project/blob/main/bolt%2FREADME... Google's: https://lists.llvm.org/pipermail/llvm-dev/2019-September/135393.html https://lists.llvm.org/pipermail/llvm-dev/2019-September/135...
- setheron 10mo agobut for x86_64, as of right now, if only a single call needs more than 31bits you have to upgrade the whole code section to large code model. BOLT AFAIU is more about cache locality of putting hot code near each other and not really breaking the 2GiB barrier.
- jeffbee 10mo agoWhy? Can't the linker or post-link optimizer reduce all near calls, leaving the more complicated mov with immediate form only where required?
- mananaysiempre 10mo agoOnce the compiler has generated a 32-bit relative jump with an R_X86_64_PLT32 relocation, it’s too late. (A bit surprising for it to be a PLT relocation, but it does make some sense upon reflection, and the linker turns it into a direct call if you’re statically linking.) I think only RISC-V was brave enough to allow potentially size-changing linker relaxation, and incidentally they screwed it up (the bug tracker says “too late to change”, which brings me great sadness given we’re talking about a new platform). On x86-64 it would probably be easier to point the relative call to a synthesized trampoline that does a 64-bit one, but it seems nobody has bothered thus far. You have to admit that sounds pretty painful.
- shevy-java 10mo ago25GB seems excessive, but I keep on having the basic compile toolchain as statically compiled executables. It simply works better when things go awry.
- wyldfire 10mo ago> What other strategies can we pursue? You can use thunks/trampolines. lld can make them for some architectures, presumably also for x86_64. Though I don't know why it didn't in your case. But, like the large code model it can be expensive to add trampolines, both in icache performance and just execution if a trampoline is in a particularly hot path.
- setheron 10mo agoIn many ways that is what the PLT is also. This is what my next post will explore. I ran into some issues with the GOT that I'll have to explore solutions for. I'm writing this for myself mostly. The whole idea for code models when you have thunks feels unnecessary.
- setheron 10mo agofollow-up: https://fzakaria.com/2025/12/29/huge-binaries-i-thunk-therefore-i-am https://fzakaria.com/2025/12/29/huge-binaries-i-thunk-theref...
- wyldfire 10mo ago> With this information, the necessity of code-models feels unecessary [sic]. Why trigger the cost for every callsite when we can do-so piecemeal as necessary with the opportunity to use profiles to guide us on which methods to migrate to thunks. Does the linker have access to the same hotness information that the compiler uses during PGO? Well -- presumably it could, even if it doesn't now. But it would be like a heuristic with a hotness threshold? Do linkers "do" heuristics?
- meisel 10mo ago> Responses to my publication submissions often claimed such problems did not exist I see this often even in communities of software engineers, where people who are unaware of certain limitations at scale will announce that the research is unnecessary
- loeg 10mo agoSure! But there's a sleight of hand in the numbers here where we're talking about 25GB binaries with debuginfo and then 2GB maximum offsets in the .text section. Of those 25GB binaries, probably 24.5 of them are debuginfo. You have to get into truly huge binaries before >2GB calls become an issue. (I wonder but have no particular insight into if LTO builds can do smarter things here -- most calls are local, but the handful of far calls can use the more expensive spelling.)
- benlivengood 10mo agoAt Google I worked with one statistics aggregation binary[0] that was ~25GB stripped. The distributed build system wouldn't even build the debug version because it exceeded the maximum configured size for any object file. I never asked if anyone had tried factoring it into separate pipelines but my intuition is that the extra processing overhead wouldn't have been worth splitting the business logic that way; once the exact set of necessary input logs are in memory you might as well do everything you need to them given the dramatically larger ratio of data size to code size. [0] https://research.google/pubs/ubiq-a-scalable-and-fault-tolerant-log-processing-infrastructure/ https://research.google/pubs/ubiq-a-scalable-and-fault-toler...
- inkyoto 10mo ago> […] 2GB maximum offsets in the .text section … on the x86 ISA because it encodes the 32-bit jump/call offset directly in the opcode. Whilst most RISC architecture do allow PC-relative branches, the offset is relatively small as 32-bit opcodes do not have enough room to squeeze a large offset in. «Long» jumps and calls are indirect branches / calls done via registers where the entirety of 64 bits is available (address alignment rules apply in RISC architectures). The target address has to be loaded / calculated beforehand, though. Available in RISC and x86 64-bit architectures.
- reactordev 10mo agoOh man, that first paragraph. “Such problems don’t exist…” what a gaslighting response to a publication submittal. The least they could do is ask where this problem emerges and you can hand wavy your answer without revealing business IP. Also, we, as an industry of software engineers, need to re-examine these hard defaults we thought could never be achieved. Such as the .text limits. Anyway, very good read.
- nicebyte 10mo agoshameless plug: if you want to understand the content of this post better, first read the first half of my article on jumps [1] (up to syscall). goes into detail about relocations and position-independent code. [1] https://gpfault.net/posts/asm-tut-4.html https://gpfault.net/posts/asm-tut-4.html
- MaskRay 10mo agoNote, sections without the SHF_ALLOC flag, such as `.debug_*` sections, do not contribute to the relocation distance pressure. Many 10+GiB binaries (likely due to not using split DWARF) might have much smaller code+data and not even close to the limit. However, Google, Meta, and ByteDance have encountered x86-64 relocation distance issue with their huge C++ server binaries. To my knowledge industry users in other domains haven't run into this problem. To address this, Google adopted the medium code model approximately two years ago for its sanitizer and PGO instrumentation builds. CUDA fat binaries also caused problems. I suggest that linker script `INSERT BEFORE/AFTER` for orphan sections (https://reviews.llvm.org/D74375 https://reviews.llvm.org/D74375 ) served as a key mitigation. I hope that a range extension thunk ABI, similar to AArch64/Power, is defined for the x86-64 psABI. It is better than the current long branch pessimization we have with -mcmodel=large. --- It seems that nobody has run into this .eh_frame_hdr implementation limitation yet * `.eh_frame_hdr -> .text`: GNU ld and ld.lld only support 32-bit offsets (`table_enc = DW_EH_PE_datarel | DW_EH_PE_sdata4;`) as of Dec 2025.