21 ms·

Why are my Go executable files so large?

468 points by caiobegotti 7y ago

swiftcoder 7y ago
This is where letting a large enterprise guide the development of a piece of widely-used software becomes questionable. At a FAANG the constraints are fundamentally different. At work I routinely see CLIs programs clocking in at a gigabyte, because it was simpler to statically link the entire shared dependency tree than to figure out the actual set of dependencies, and once your binary grows that big, running LTO adds too much time to one's builds. And disk space is basically free at FAANG scale...
- tomphoolery 7y ago
  Disk space in general is pretty much free these days. 123mb for a whole database is really not that big of a deal, IMHO. For example, my local PostgreSQL docker image is 140mb plus the alpine OS (5mb). And the Ruby on Rails application using that image clocks in at a little over 1gb, also using the ruby alpine image as a base (50mb). With my company, the cost really started to become a burden with data transfer. But transferring images to and from the AWS container registry is so expensive that we actually build production images inside the Kubernetes cluster (plus the cluster has access to all the secrets and stuff), even though it was a bit harder to implement. If you're FAANG and you can run your own stable and highly-available cloud, data transfer rates don't matter, so you can deploy your application the "right" way in a containerized world.
  - gumby 7y ago
    Yes but when running a long-lived application on a lot of data it's typically important to keep the executable small both so it can be "hot" and to leave more room for data. At scale this could be even more important, not less, than for a smaller operation. Of course the real (i.e. explicitly stated by Pike) driver for go was the assertion that inexperienced new hires write poor code and so the harder it is for them to get into trouble the better, even at the cost of other issues.
    - boardwaalk 7y ago
      Executable size on disk doesn't dictate effective size at higher levels of the memory hierarchy. IOW, you're paying by the page/cache line. If the extra bloat (debug information) in your executable isn't interspersed within your actual code (it shouldn't be), you aren't paying for it in runtime efficiency.
    - lazyjones 7y ago
      > Yes but when running a long-lived application on a lot of data it's typically important to keep the executable small both so it can be "hot" and to leave more room for data. Who really runs server applications these days where data/rss are not a large multiple of the actual code segments? People happily run JVM server processes these days, how much code/data do you think that pulls in just to start up?
      - deleted 7y ago
        [deleted]
- bpodgursky 7y ago
  Mmmm I don't know if I agree with that logic. I'd argue the opposite -- as a startup, you can't afford to micro-optimize. Labor, time and opportunity cost dwarf all but the grossest resource waste. If you need to use 100GB/k8s node instead of 50GB, it will have 0 effect on the success of your venture. At Google scale, it becomes worth it to optimize: - You are delivering more product per engineer, and more product means more resources. Instead of a single customer instance which costs $1/month more, you have 100,000 customer instances, costing a significant amount more. It becomes worth trimming margins. - You have economies of scale, and it might be worth it for an engineer to spend a month trimming 2% of the cost of a software deliverable. The common refrain for startups is "do things that don't scale", and this is for good reason. Google has to actually worry about fixing things AFTER they are scaled.
  - fwip 7y ago
    I think you're both right. A 125MB binary is relatively beefy, but still easily fits in RAM. The amount of disk that you're spending on a single executable (your database, in this instance) is tiny in comparison to the amount of data stored in that database. It's definitely worth it for Google to trim 2% off of their storage requirements - but if your binary is 0.1% of your storage, it's barely even worth glancing at.
  - TheReveller 7y ago
    There are plenty of efforts and tooling that reduce the number of dependencies and the deploy size of binaries at Google. The notion that we don't care about size isn't true. But it's true that it's not worth optimising first, it's done by first evaluating the impact across the fleet and then prioritising the most effective changes.
- BubRoss 7y ago
  LTO is no magic bullet for binary size either. A binary that does nothing will still link in the whole C library. It doesn't end up decreasing large program sizes that much in my experience either.
- techsin101 7y ago
  Tree shaking?
- akhilcacharya 7y ago
  Genuinely wondering how a CLI would be 1GB, even at Facebook. Never encountered anything _close_ at Amazon.
  - swiftcoder 7y ago
    I never saw one a singular tool that big at Amazon either, but the general concept held. The AWS service I worked on had ~1,000 jars in its dependency closure. All of those had to be uploaded to S3 after each build, and then downloaded onto each EC2 instance during deployment. We're talking on the order of a terabyte of data transfer each time we deployed to a thousand instance fleet (ideally deploying weekly)
  - deleted 7y ago
    [deleted]
  - lkramer 7y ago
    Not your usually usecase, but the binary for running the Bloomberg terminal is well over 1GB (In fact running into the the 4GB executable boundary was a problem).
    - akhilcacharya 7y ago
      Oh, that's fascinating. I never checked the binary size but that makes sense.
    - saagarjha 7y ago
      I’m curious, what’s in it? Which sections take up the space?
      - lkramer 7y ago
        Sorry, I should have clarified, this is the server side binary. I believe the part that's run on the user's computer is just a kind of dumb terminal. The reason it's so big (or at least was several years ago) is that every single function is compiled into one binary monolith.
- jrockway 7y ago
  > This is where letting a large enterprise guide the development of a piece of widely-used software becomes questionable. At a FAANG the constraints are fundamentally different. I see where you're going with this, but the conclusion of the article is that Go isn't (what you call) FAANG-y enough for them. From the article: "This design choice was intended to lower the start-up time of programs [...] This performance goal is not relevant to server software with long-running processes, like CockroachDB, and its incurred space cost is particularly inconvenient for large, feature-rich programs." What they want is a way to pass 1000 parameters to the compiler so that nobody ever gets a working binary unless they have a team of release engineers to make one for you. The Go team and their users at Google easily have access to said team of release engineers, but took the opposite approach. They optimized for programs that start fast even though most of those teams at Google are writing programs that run for a long time. They optimized for a compiler and runtime that don't have a billion knobs to turn, even though they can afford to pay an army of engineers to tune those knobs. So I actually think this has very little to do with optimizing for FAANG. This is more about every program getting the same compiler options, which is merely a philosophy that some people on the Go team have, not really a philosophy that every FAANG has. (They use Java at Google too, and Java is more than happy to give you knobs to turn. It also doesn't start up very fast.)
  - BossingAround 7y ago
    > It also doesn't start up very fast This has been changing rapidly with GraalVM and projects like Quarkus, which can build a static binary of a Java program that starts, as you'd expect, much faster (still not performance of a C/C++ program, but way faster than a regular Java program).
    - jhgb 7y ago
      Did I miss something important about GraalVM? I thought it was another runtime on top of JVM, not a replacement for it. For a Java program, for example, GraalVM would presumably have no impact since it wouldn't be used for it.
      - 7y ago
zerr 7y ago
70MB source seems somewhat bloat for such project, no?
- thrill 7y ago
  For an ACID compliant, resilient, consistent, distributed, auto-sharding and auto-tuning for low latency, highly scalable SQL database? No.
  - BubRoss 7y ago
    70MB of source is such an extreme amount I don't know how it could be reasonably justified, there must be an enormous amount of waste. All of sqlite is 6MB.
    - paulddraper 7y ago
      When you have the features of sqlite, you can have the size of sqlite. PostreSQL is 36MB. [1] Granted Go is much terser than C and has a larger standard library, but we're not at absurd levels. [1] find src -name '*.c' -o -name '*.h' | xargs cat | wc -c
- newnewpdro 7y ago
  Have you seen idiomatic Go? The error handling alone is a verbose mess.
- frumiousirc 7y ago
  It also seemed large to me so I poked around. 36 MB are just the .go files. In comparison, postgresql has about 38 MB of .c and .h files. So, my mind is changed.
- Scarbutt 7y ago
  There is a lot of copy/paste in Go.
  - dana321 7y ago
sys_64738 7y ago
I think the GO runtime libraries are linked into the binary so there are no external dependencies for GO itself.
- turk73 7y ago
  Exactly. The whole point of Go is to fix the issue with Python where you have to have all the right libraries installed wherever you intend to run.
warent 7y ago
So what is the solution then? Will they just have to fork Go and compress the table again like before? It's completely insane that it would eventually surpass the size of the program itself.
- kseistrup 7y ago
  Is the line table in go executables absolutely essential? Shouldn't they be strippable, the way you can strip debug symbols from a C binary?
  - Carpetsmoker 7y ago
    They are, and many people don't include them in production builds. This article is incomplete for not mentioning it.
    - nneonneo 7y ago
      I did a quick search and turned up nothing about stripping the pclntab (note: distinct from the DWARF line-number tables which can be stripped). A post on Google Groups suggests the opposite - pclntab cannot be stripped because the runtime needs the info for GC - https://groups.google.com/forum/m/#!topic/golang-nuts/hEdGYnqokZc https://groups.google.com/forum/m/#!topic/golang-nuts/hEdGYn....
    - caiobegotti 7y ago
      You're right, but that wouldn't make the main problem go away though. I just built a simple plugin for Kubernetes's kubectl and it's about 32MB with go build's -ldflags -s -w where 16MB of that is still the pclntab mentioned in the article.
      - Carpetsmoker 7y ago
        The problem with removing that completely is that you won't get any information on panics. I don't think this is what you really want, and the current behaviour strikes me as a reasonable middle ground. 32M is really large for a simple plugin, and to be honest I think that says just as much about Kubernetes as it says about Go.
kris-s 7y ago
Great writeup, I believe there is a open issue from Rob Pike 2013 this would fall under: https://github.com/golang/go/issues/6853 https://github.com/golang/go/issues/6853
- ChrisSD 7y ago
  Except that this write up says this was a deliberate design decision to trade space for speed. That's not something to be fixed, unless you convince Go to make different trade offs or to provide more optimization options.
  - kris-s 7y ago
    The linked issue is tagged "NeedsFix".
- syockit 7y ago
  In fact, there's a new issue opened (again, from Rob Pike) in response to this article: https://github.com/golang/go/issues/36313 https://github.com/golang/go/issues/36313
aequitas 7y ago
There is an project called TinyGo [0] which brings Go to embedded systems where binary and memory size matter even more. [0] https://archive.fosdem.org/2019/schedule/event/go_on_microcontrollers/ https://archive.fosdem.org/2019/schedule/event/go_on_microco...
Carpetsmoker 7y ago
You can compress with upx (at the cost of increased startup time in the order of hundreds of ms, which is okay for servers) and/or not include all debug symbols. Doing both usually shaves >60% off a binary.
- oefrha 7y ago
  <deleted> Edit: somehow read “doing both” as “doing either”. Just ignore this.
  - wongarsu 7y ago
    That's already 30% off for free, now run that through upx [1] 1: https://upx.github.io/ https://upx.github.io/
  - robfig 7y ago
    UPX compresses the binary and introduces a decompression step on startup, so you run exactly the input binary Stripped means that the compiler does not include debug symbols. They are completely different - you can use either or both.
- amscanne 7y ago
  UPX transforms demand-paged, reclaimable page cache memory into a blob of unreclaimable anonymous memory. It makes no sense for most use cases where I’ve seen it. It adds runtime costs both in terms of start-up and memory usage. Maybe it helps in terms of binary sizes for downloads — but those are often compressed anyways! E.g. Your docker images are compressed and UPX’ed binaries in a layer aren’t buying you anything (just adding runtime costs).
  - Carpetsmoker 7y ago
    For a lot of applications the increase in startup size and memory usage are negligible. The startup time is increased by dozens or hundreds of ms, which is not a lot for a server application. I tried to measure the increase in memory, but wasn't really able to, so either it's a very subtle difference or it's very small. Not everything uses Docker.
selljamhere 7y ago
Would it be possible to add a flag for the compiler to disable the line table pre-expansion?
- Exuma 7y ago
  That was my question but I would imagine the problem is then that you can't debug production. It looks like a more common solution is dSYM, .dbg, .pdb or other things (read from another commenter)
- w0rd-driven 7y ago
  I can’t exactly tell if we’re saying the same thing but my thought was a flag to switch between the 1.2 way for faster startup and the earlier approach for longer running processes. The trade off is added complexity in identifying your binary usage patterns and keeping both methods in the tooling. These kind of changes may not be breaking in a technical sense but it’s very unexpected behavior if you’re one to notice patterns like file sizes changing in such a significant way over time. An answer of “stick with v1.1x indefinitely if you want the old behavior” only feels like a very temporary answer.
fuckyah 7y ago
try rust
PeterisP 7y ago
Is it just me or something like that runtime.pclntab shouldn't be included in production builds at all? I mean, it makes all sense while you're developing and testing, but it should be reasonably possible to strip it out from production build binaries, and instead put it in a separate file so that if you do get a crash with a stack trace, then some external script can transform the program counters to line numbers, not have it embedded in every deployed binary.
- mynegation 7y ago
  Not sure if it is just you but normally you DO want this information in the production build. It is quite a bad situation to have a run time exception in PROD and having no idea how it happened. Sure, there is defensive programming and checks and asserts but most of the time you cannot foresee everything. I get the point about external symbols and location database, but oftentimes time is precious and having fully laid out stack trace in the log will allow you to get to the root much faster.
  - Exuma 7y ago
    So what would a solution be? How do other languages solve that problem?
    - zorked 7y ago
      That is debug information. Just have it stored elsewhere (not on the binary you ship everywhere) and use that in conjunction with your core dump to debug.
    - asveikau 7y ago
      A lot of them have symbol files separate from the binary. Unixy tooling doesn't do this by default but for example objcopy(1) in binutils can copy symbols to another file before you run strip(1), and on Mac my memory is rusty but I think it may be dsymutil(1) that lets you copy to a .dSYM bundle. Microsoft has its .pdb files and never even keeps debug info inside the binary proper.
    - 7y ago
throw_m239339 7y ago
Being an Electron and a Go developer, I'm not complaining too much about the size of Go executables. Electron backed softwareon the other hand...
- dana321 7y ago
  Have you tried this? I've played around with it in the past. https://github.com/asticode/go-astilectron https://github.com/asticode/go-astilectron
nneonneo 7y ago
This is where go’s insistence on reinventing the wheel feels terribly misplaced. Every major debug format has a way to associate code locations with line numbers. Every major debug format also has a way to separate the debug data from the main executable (.dSYM, .dbg, .pdb). In other words, the problem that the massive pclntab table (over 25% of a stripped binary!) is trying to solve is already a well-trodden and solved problem. But go, being go, insists on doing things their own way. The same holds for their wacky calling convention (everything on the stack even when register calling convention is the platform default) and their zero-reliance on libc (to the point of rolling their own syscall code and inducing weird breakage). Sure, the existing solutions might not be perfect, but reinventing the wheel gets tiresome after a while. Contrast this with Rust, which has made an overt effort to fit into existing tooling: symbols are mangled using the C++ mangler so that gdb and friends understand them, rust outputs nice normal DWARF stuff on Linux so gdb debugging just works, Rust uses platform calling convention as much as possible, etc. It means that a wealth of existing tooling just works.
- krasin 7y ago
  I agree with some of your points; but zero-reliance on libc is the reason why it's so easy to use Go in containers; and Docker is one of the primary reasons why Go is popular. It's what they have got right.
  - Skunkleton 7y ago
    You could statically link in libc and get the same effect.
    - IshKebab 7y ago
      Says someone who has never actually tried to do that. You can statically link with Musl. But you can't really statically link with Glibc.
      - josteink 7y ago
        Technically speaking OP only suggested statically linking (a) libc, not glibc specifically, and musl is a libc.
evmar 7y ago
The next time you need to make an HTML treemap like this, try my tool: https://github.com/evmar/webtreemap https://github.com/evmar/webtreemap It provides a command line app that accepts simple space-delimited data and outputs an HTML file. See the doc: https://github.com/evmar/webtreemap#command-line https://github.com/evmar/webtreemap#command-line (It also is available as a JS library for linking in web apps, but the command line app is the one that I end up using the most. I actually built it to visualize binary size exactly like this post and then later generalized it.)
- leetrout 7y ago
  Any plans to fix (or close if they are fixed) the existing issues?
- pacoverdi 7y ago
  Another option is to generate a text file in the format expected by flamegraph. Especially useful when the data is hierarchical. https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph Exemple for Java: https://github.com/pcdv/deps-flamegraph/blob/master/README.md https://github.com/pcdv/deps-flamegraph/blob/master/README.m...
fxtentacle 7y ago
TLDR: They included debugging information when they did not intend to.
- techsin101 7y ago
  What's the fix?
  - layoutIfNeeded 7y ago
    Don’t use Go.
lazyjones 7y ago
The author guessed a few things wrong: * fmt.Println pulling in 300KB isn't proof that Go's standard library isn't "well modularized". It's the wonders of Unicode and other code that is actually used. * 900K for the runtime isn't surprising when you have complex garbage collection and goroutine scheduling among other things
- majewsky 7y ago
  > fmt.Println pulling in 300KB isn't proof that Go's standard library isn't "well modularized". It's the wonders of Unicode and other code that is actually used. I would guess a large part is that it has to pull in the entire reflection library to support format verbs like %#v which renders the argument as a Go literal.
- knz42 7y ago
  The C library supports unicode too nowadays (at least the GNU C library does) yet linking a simple program that uses printf does not import 300KB of code. The answer comes from the reflection code as the other commenter states.
tytso 7y ago
I'm not sure why people are so worried about the size of the executable file here. If the runtime.pclntab table is never[1] used then it won't be paged into memory, and disk space is mostly free these days. [1] Well, hardly _ever_! (Sorry not sorry for the obligatory Gilbert and Sullivan reference.) If you're using the Go executable on a system without virtual memory support, yeah, that's going to suck, but it appears the Go runtime is horribly bloated and not really suited for super-tiny 16-bit processors in the micro-embedded space. But for something like Cockroachdb, why worry about the file size?
- caiobegotti 7y ago
  Bandwidth [to transfer big binaries around] is not free however.
  - pushpop 7y ago
    True, but you’d need a lot of transfers before it starts to add up but if you run into that edge case then thankfully you don’t actually need the file to be executable during transit so you probably should compress the file for transit and decompress it at the other end (assuming your CI/CD pipeline allows for that) or compile it on the destiny nodes (since Go’s compiler is fast). Admittedly neither are perfect solutions but software architecting is always about making smart compromises.
- F-0X 7y ago
  > disk space is mostly free these days. This is the only "argument" ever presented, and I don't think it is any good. I care about file sizes. I want to get the most out of my hardware. Not needing to buy another drive is always going to be cheaper for me and every other user.
  - trevor-e 7y ago
    Then don't use Go, you aren't their target audience in that case. And I don't mean this in a harsh way, just that Google is clearly opinionated in how they are building Go.
kazinator 7y ago
> there is about 70MB of source code currently in CockroachDB 19.1 That's what is insane here; way more so than Go executable size issues.
aasasd 7y ago
> runtime.pclntab Ah yes, an example of Google's long and descriptive identifier names, a lament of which surfaced here recently. (https://news.ycombinator.com/item?id=21843180 https://news.ycombinator.com/item?id=21843180) > I’m glad we now live in a futuristic utopia where keyboard farts like p, idxcrpm, and x3 are rare.
- esprehn 7y ago
  What's interesting is that Go encourages letter soup while the rest of Google styles encourage long descriptive names. Sometimes I think it was intentionally done to troll the rest of Google that used long_descriptive_names_ with an 80 col limit (Java argued up to 100 but no one else did, not even JS which uses Java's long names). I always thought it was funny walking around the office seeing everyone with their squished and wrapped code on a huge 32in monitor. Lots of the code in google3 looks like haiku squished to the right margin.
  - aasasd 7y ago
    Excellently thought out. Very rational. Truly a language for the twenty-first century.
nickcw 7y ago
> prior to 1.2, the Go linker was emitting a compressed line table, and the program would decompress it upon initialization at run-time. in Go 1.2, a decision was made to pre-expand the line table in the executable file into its final format suitable for direct use at run-time, without an additional decompression step. This is a good choice I think and the author of the article missed the most important point - it uses less memory to have an uncompressed table. This sounds paradoxical but if a table has to be expanded at runtime then it has to be loaded into memory. However if a table is part of the executable, the OS won't even load it into memory unless it is used and will only page the bits into memory that are used. You see the same effect when you compress a binary with UPX (for example) - the size on disk gets smaller, but because the entire executable is decompressed into RAM rather than demand paged in, then it uses more memory.
- deleted 7y ago
  [deleted]
- zamadatix 7y ago
  If you decompress it to an mmaped file it'll be one of the first things written to disk under memory pressure anyways and instantly available in normal situations. With the ever decreasing cost of flash and it's ever increasing speed relative to the CPU compression is not really worth what it used to be to startup times 10 years ago though.
  - ceph_ 7y ago
    Swapping is fine for workstations and home computers. But high performance machines running in a production environments will absolutely have swap disabled. The performance difference between RAM and disk is not an acceptable tradeoff. RAM will be tightly provisioned and jobs will be killed rather than letting the machine OOM.
    - Galanwe 7y ago
      Are you arguing you prefer your production jobs to be killed rather than slowed ?
dana321 7y ago
Go is a hybrid language. Its not a jvm, but its runtime has jvm-like features such as garbage collection and reflection but also a thread scheduling system called goroutines. I love the fact it is monolithic in nature.. One exe is all you need no matter which platform you use. Everything is statically compiled into the binary. No bundling the jvm and a load of jar files, or lib*.so dependencies.
- int_19h 7y ago
  Toolkits to precompile a Java app into a single native binary with the runtime and stdlib inside have been around for 20 years or so.
andy_ppp 7y ago
Because Go isn’t C?
narven 7y ago
Use upx https://upx.github.io/ https://upx.github.io/
bsaul 7y ago
sidenote : currently trying to get up to date with the best way to get distributed acid key value storage those days. Is coackroach the new standard ? I tried to find benchmarks comparing it to things like postgres for various use case but only found articles that read like ads.
- redis_mlc 7y ago
  > the best way to get distributed acid key value storage You will need to define "best way", "distributed" and "acid" based on your requirements. For most people, multi-master MySQL and Redis with vector clocks is a great combination. > Is coackroach the new standard No database less than 5 - 10 years old works in production, so no, it's not a standard. > I tried to find benchmarks comparing it to things like postgres for various use case Well, without more detailed requirements, good luck. Also comparing pg to distributed databases is ... like comparing apples and oranges. Source: experienced DBA.
  - bsaul 7y ago
    distributed : able to scale horizontally, ideally over multiple availability zone, to ensure both performance and data safety. acid : able to perform bank account money transfert style transaction. last time i checked, pg was able to shard and work in multiple replication configuration, ensuring a good scalability story up to terabytes of active data.. Does multi master mysql enable row locking ?
ausjke 7y ago
go really should be the same size as c with static link, somehow it's 4x larger in size, why is that? upx can help, with upx, c static binary is still much smaller. both removed debugging info.
boulos 7y ago
Disclosure: I work on Google Cloud (but this applies generally). There seem to be a lot of arguments about disk space (arguably free in 2020), memory (free-ish because as tytso points out, they won’t get backed) and then bandwidth. I think most people saying “bandwidth” might mean “time to fetch”, because GCS, GCR, S3, etc. all have free egress on the transfer from them to GCE/EC2. If you have a self-hosted Docker Hub or something on an EC2 instance, that’s not the case (you may pay Zone to Zone egress of $.01/GB). If you were paying a penny per gigabyte egress, a 100MB-ish binary is only .1 GB and therefore at best 1/1000th of a dollar. On the 16 vCPU hosts that CockroachDB prefers (see their recent benchmarks), that’s equivalent to about 5 seconds of runtime. A fair retort is that in container land, this becomes death by 1000 cuts as each binary includes all the same stuff over and over again (for minikube, the .iso is nearly 1 GB [1] now, but not because of the 50MB binary itself). Even so, a 1GB image takes almost as long to pull from a object store like GCS as a 100 MB image (at many Gbps, the constant factors dominate). If you’re trying to run something this large as a function on Cloud Run or Lambda or Knative, you’ll probably be sad (you’ve burned about 800 vcpu seconds, economically, of compute time) but that’s why there are layers. tl;dr: your 50 or 100 MB binary doesn’t “cost” much, but a 1 GB container image without shared layers does. [1] https://github.com/kubernetes/minikube/issues/5013 https://github.com/kubernetes/minikube/issues/5013
Timothycquinn 7y ago
Why are they not considering keeping the pre 1.2 compressed runtime.pclntab and have the data be decompressed to a separate file on first run. This way, memory footprint is kept low whilst keeping executable size down?
kccqzy 7y ago
Language flame wars aside, Bloaty McBloatface is a wonderful tool to analyze why the binary size is big: https://github.com/google/bloaty https://github.com/google/bloaty I frequently use this tool to answer questions for C++ binaries, another language that has a penchant for producing large executables.
_bxg1 7y ago
> prior to 1.2, the Go linker was emitting a compressed line table, and the program would decompress it upon initialization at run-time. in Go 1.2, a decision was made to pre-expand the line table in the executable file into its final format suitable for direct use at run-time, without an additional decompression step. Sounds like a good case for a flag.
todotask 7y ago
Rob Pike submit issue: https://github.com/golang/go/issues/36313 https://github.com/golang/go/issues/36313

468 points by caiobegotti 7y ago

This is where letting a large enterprise guide the development of a piece of widely-used software becomes questionable. At a FAANG the constraints are fundamentally different. At work I routinely see CLIs programs clocking in at a gigabyte, because it was simpler to statically link the entire shared dependency tree than to figure out the actual set of dependencies, and once your binary grows that big, running LTO adds too much time to one's builds. And disk space is basically free at FAANG scale...

tomphoolery 7y ago
Disk space in general is pretty much free these days. 123mb for a whole database is really not that big of a deal, IMHO. For example, my local PostgreSQL docker image is 140mb plus the alpine OS (5mb). And the Ruby on Rails application using that image clocks in at a little over 1gb, also using the ruby alpine image as a base (50mb). With my company, the cost really started to become a burden with data transfer. But transferring images to and from the AWS container registry is so expensive that we actually build production images inside the Kubernetes cluster (plus the cluster has access to all the secrets and stuff), even though it was a bit harder to implement. If you're FAANG and you can run your own stable and highly-available cloud, data transfer rates don't matter, so you can deploy your application the "right" way in a containerized world.
- gumby 7y ago
  Yes but when running a long-lived application on a lot of data it's typically important to keep the executable small both so it can be "hot" and to leave more room for data. At scale this could be even more important, not less, than for a smaller operation. Of course the real (i.e. explicitly stated by Pike) driver for go was the assertion that inexperienced new hires write poor code and so the harder it is for them to get into trouble the better, even at the cost of other issues.
  - boardwaalk 7y ago
    Executable size on disk doesn't dictate effective size at higher levels of the memory hierarchy. IOW, you're paying by the page/cache line. If the extra bloat (debug information) in your executable isn't interspersed within your actual code (it shouldn't be), you aren't paying for it in runtime efficiency.
  - lazyjones 7y ago
    > Yes but when running a long-lived application on a lot of data it's typically important to keep the executable small both so it can be "hot" and to leave more room for data. Who really runs server applications these days where data/rss are not a large multiple of the actual code segments? People happily run JVM server processes these days, how much code/data do you think that pulls in just to start up?
    - deleted 7y ago
      [deleted]
bpodgursky 7y ago
Mmmm I don't know if I agree with that logic. I'd argue the opposite -- as a startup, you can't afford to micro-optimize. Labor, time and opportunity cost dwarf all but the grossest resource waste. If you need to use 100GB/k8s node instead of 50GB, it will have 0 effect on the success of your venture. At Google scale, it becomes worth it to optimize: - You are delivering more product per engineer, and more product means more resources. Instead of a single customer instance which costs $1/month more, you have 100,000 customer instances, costing a significant amount more. It becomes worth trimming margins. - You have economies of scale, and it might be worth it for an engineer to spend a month trimming 2% of the cost of a software deliverable. The common refrain for startups is "do things that don't scale", and this is for good reason. Google has to actually worry about fixing things AFTER they are scaled.
- fwip 7y ago
  I think you're both right. A 125MB binary is relatively beefy, but still easily fits in RAM. The amount of disk that you're spending on a single executable (your database, in this instance) is tiny in comparison to the amount of data stored in that database. It's definitely worth it for Google to trim 2% off of their storage requirements - but if your binary is 0.1% of your storage, it's barely even worth glancing at.
- TheReveller 7y ago
  There are plenty of efforts and tooling that reduce the number of dependencies and the deploy size of binaries at Google. The notion that we don't care about size isn't true. But it's true that it's not worth optimising first, it's done by first evaluating the impact across the fleet and then prioritising the most effective changes.
BubRoss 7y ago
LTO is no magic bullet for binary size either. A binary that does nothing will still link in the whole C library. It doesn't end up decreasing large program sizes that much in my experience either.
techsin101 7y ago
Tree shaking?
akhilcacharya 7y ago
Genuinely wondering how a CLI would be 1GB, even at Facebook. Never encountered anything _close_ at Amazon.
- swiftcoder 7y ago
  I never saw one a singular tool that big at Amazon either, but the general concept held. The AWS service I worked on had ~1,000 jars in its dependency closure. All of those had to be uploaded to S3 after each build, and then downloaded onto each EC2 instance during deployment. We're talking on the order of a terabyte of data transfer each time we deployed to a thousand instance fleet (ideally deploying weekly)
- deleted 7y ago
  [deleted]
- lkramer 7y ago
  Not your usually usecase, but the binary for running the Bloomberg terminal is well over 1GB (In fact running into the the 4GB executable boundary was a problem).
  - akhilcacharya 7y ago
    Oh, that's fascinating. I never checked the binary size but that makes sense.
  - saagarjha 7y ago
    I’m curious, what’s in it? Which sections take up the space?
    - lkramer 7y ago
      Sorry, I should have clarified, this is the server side binary. I believe the part that's run on the user's computer is just a kind of dumb terminal. The reason it's so big (or at least was several years ago) is that every single function is compiled into one binary monolith.
jrockway 7y ago
> This is where letting a large enterprise guide the development of a piece of widely-used software becomes questionable. At a FAANG the constraints are fundamentally different. I see where you're going with this, but the conclusion of the article is that Go isn't (what you call) FAANG-y enough for them. From the article: "This design choice was intended to lower the start-up time of programs [...] This performance goal is not relevant to server software with long-running processes, like CockroachDB, and its incurred space cost is particularly inconvenient for large, feature-rich programs." What they want is a way to pass 1000 parameters to the compiler so that nobody ever gets a working binary unless they have a team of release engineers to make one for you. The Go team and their users at Google easily have access to said team of release engineers, but took the opposite approach. They optimized for programs that start fast even though most of those teams at Google are writing programs that run for a long time. They optimized for a compiler and runtime that don't have a billion knobs to turn, even though they can afford to pay an army of engineers to tune those knobs. So I actually think this has very little to do with optimizing for FAANG. This is more about every program getting the same compiler options, which is merely a philosophy that some people on the Go team have, not really a philosophy that every FAANG has. (They use Java at Google too, and Java is more than happy to give you knobs to turn. It also doesn't start up very fast.)
- BossingAround 7y ago
  > It also doesn't start up very fast This has been changing rapidly with GraalVM and projects like Quarkus, which can build a static binary of a Java program that starts, as you'd expect, much faster (still not performance of a C/C++ program, but way faster than a regular Java program).
  - jhgb 7y ago
    Did I miss something important about GraalVM? I thought it was another runtime on top of JVM, not a replacement for it. For a Java program, for example, GraalVM would presumably have no impact since it wouldn't be used for it.
    - 7y ago

zerr 7y ago

70MB source seems somewhat bloat for such project, no?

thrill 7y ago
For an ACID compliant, resilient, consistent, distributed, auto-sharding and auto-tuning for low latency, highly scalable SQL database? No.
- BubRoss 7y ago
  70MB of source is such an extreme amount I don't know how it could be reasonably justified, there must be an enormous amount of waste. All of sqlite is 6MB.
  - paulddraper 7y ago
    When you have the features of sqlite, you can have the size of sqlite. PostreSQL is 36MB. [1] Granted Go is much terser than C and has a larger standard library, but we're not at absurd levels. [1] find src -name '*.c' -o -name '*.h' | xargs cat | wc -c
newnewpdro 7y ago
Have you seen idiomatic Go? The error handling alone is a verbose mess.
frumiousirc 7y ago
It also seemed large to me so I poked around. 36 MB are just the .go files. In comparison, postgresql has about 38 MB of .c and .h files. So, my mind is changed.
Scarbutt 7y ago
There is a lot of copy/paste in Go.
- dana321 7y ago

sys_64738 7y ago

I think the GO runtime libraries are linked into the binary so there are no external dependencies for GO itself.

turk73 7y ago
Exactly. The whole point of Go is to fix the issue with Python where you have to have all the right libraries installed wherever you intend to run.

warent 7y ago

So what is the solution then? Will they just have to fork Go and compress the table again like before? It's completely insane that it would eventually surpass the size of the program itself.

kseistrup 7y ago
Is the line table in go executables absolutely essential? Shouldn't they be strippable, the way you can strip debug symbols from a C binary?
- Carpetsmoker 7y ago
  They are, and many people don't include them in production builds. This article is incomplete for not mentioning it.
  - nneonneo 7y ago
    I did a quick search and turned up nothing about stripping the pclntab (note: distinct from the DWARF line-number tables which can be stripped). A post on Google Groups suggests the opposite - pclntab cannot be stripped because the runtime needs the info for GC - https://groups.google.com/forum/m/#!topic/golang-nuts/hEdGYnqokZc https://groups.google.com/forum/m/#!topic/golang-nuts/hEdGYn....
  - caiobegotti 7y ago
    You're right, but that wouldn't make the main problem go away though. I just built a simple plugin for Kubernetes's kubectl and it's about 32MB with go build's -ldflags -s -w where 16MB of that is still the pclntab mentioned in the article.
    - Carpetsmoker 7y ago
      The problem with removing that completely is that you won't get any information on panics. I don't think this is what you really want, and the current behaviour strikes me as a reasonable middle ground. 32M is really large for a simple plugin, and to be honest I think that says just as much about Kubernetes as it says about Go.

kris-s 7y ago

Great writeup, I believe there is a open issue from Rob Pike 2013 this would fall under: https://github.com/golang/go/issues/6853 https://github.com/golang/go/issues/6853

ChrisSD 7y ago
Except that this write up says this was a deliberate design decision to trade space for speed. That's not something to be fixed, unless you convince Go to make different trade offs or to provide more optimization options.
- kris-s 7y ago
  The linked issue is tagged "NeedsFix".
syockit 7y ago
In fact, there's a new issue opened (again, from Rob Pike) in response to this article: https://github.com/golang/go/issues/36313 https://github.com/golang/go/issues/36313

aequitas 7y ago

There is an project called TinyGo [0] which brings Go to embedded systems where binary and memory size matter even more. [0] https://archive.fosdem.org/2019/schedule/event/go_on_microcontrollers/ https://archive.fosdem.org/2019/schedule/event/go_on_microco...

Carpetsmoker 7y ago

You can compress with upx (at the cost of increased startup time in the order of hundreds of ms, which is okay for servers) and/or not include all debug symbols. Doing both usually shaves >60% off a binary.

oefrha 7y ago
<deleted> Edit: somehow read “doing both” as “doing either”. Just ignore this.
- wongarsu 7y ago
  That's already 30% off for free, now run that through upx [1] 1: https://upx.github.io/ https://upx.github.io/
- robfig 7y ago
  UPX compresses the binary and introduces a decompression step on startup, so you run exactly the input binary Stripped means that the compiler does not include debug symbols. They are completely different - you can use either or both.
amscanne 7y ago
UPX transforms demand-paged, reclaimable page cache memory into a blob of unreclaimable anonymous memory. It makes no sense for most use cases where I’ve seen it. It adds runtime costs both in terms of start-up and memory usage. Maybe it helps in terms of binary sizes for downloads — but those are often compressed anyways! E.g. Your docker images are compressed and UPX’ed binaries in a layer aren’t buying you anything (just adding runtime costs).
- Carpetsmoker 7y ago
  For a lot of applications the increase in startup size and memory usage are negligible. The startup time is increased by dozens or hundreds of ms, which is not a lot for a server application. I tried to measure the increase in memory, but wasn't really able to, so either it's a very subtle difference or it's very small. Not everything uses Docker.

selljamhere 7y ago

Would it be possible to add a flag for the compiler to disable the line table pre-expansion?

Exuma 7y ago
That was my question but I would imagine the problem is then that you can't debug production. It looks like a more common solution is dSYM, .dbg, .pdb or other things (read from another commenter)
w0rd-driven 7y ago
I can’t exactly tell if we’re saying the same thing but my thought was a flag to switch between the 1.2 way for faster startup and the earlier approach for longer running processes. The trade off is added complexity in identifying your binary usage patterns and keeping both methods in the tooling. These kind of changes may not be breaking in a technical sense but it’s very unexpected behavior if you’re one to notice patterns like file sizes changing in such a significant way over time. An answer of “stick with v1.1x indefinitely if you want the old behavior” only feels like a very temporary answer.

fuckyah 7y ago

try rust

PeterisP 7y ago

Is it just me or something like that runtime.pclntab shouldn't be included in production builds at all? I mean, it makes all sense while you're developing and testing, but it should be reasonably possible to strip it out from production build binaries, and instead put it in a separate file so that if you do get a crash with a stack trace, then some external script can transform the program counters to line numbers, not have it embedded in every deployed binary.

mynegation 7y ago
Not sure if it is just you but normally you DO want this information in the production build. It is quite a bad situation to have a run time exception in PROD and having no idea how it happened. Sure, there is defensive programming and checks and asserts but most of the time you cannot foresee everything. I get the point about external symbols and location database, but oftentimes time is precious and having fully laid out stack trace in the log will allow you to get to the root much faster.
- Exuma 7y ago
  So what would a solution be? How do other languages solve that problem?
  - zorked 7y ago
    That is debug information. Just have it stored elsewhere (not on the binary you ship everywhere) and use that in conjunction with your core dump to debug.
  - asveikau 7y ago
    A lot of them have symbol files separate from the binary. Unixy tooling doesn't do this by default but for example objcopy(1) in binutils can copy symbols to another file before you run strip(1), and on Mac my memory is rusty but I think it may be dsymutil(1) that lets you copy to a .dSYM bundle. Microsoft has its .pdb files and never even keeps debug info inside the binary proper.
  - 7y ago

throw_m239339 7y ago

Being an Electron and a Go developer, I'm not complaining too much about the size of Go executables. Electron backed softwareon the other hand...

dana321 7y ago
Have you tried this? I've played around with it in the past. https://github.com/asticode/go-astilectron https://github.com/asticode/go-astilectron

nneonneo 7y ago

This is where go’s insistence on reinventing the wheel feels terribly misplaced. Every major debug format has a way to associate code locations with line numbers. Every major debug format also has a way to separate the debug data from the main executable (.dSYM, .dbg, .pdb). In other words, the problem that the massive pclntab table (over 25% of a stripped binary!) is trying to solve is already a well-trodden and solved problem. But go, being go, insists on doing things their own way. The same holds for their wacky calling convention (everything on the stack even when register calling convention is the platform default) and their zero-reliance on libc (to the point of rolling their own syscall code and inducing weird breakage). Sure, the existing solutions might not be perfect, but reinventing the wheel gets tiresome after a while. Contrast this with Rust, which has made an overt effort to fit into existing tooling: symbols are mangled using the C++ mangler so that gdb and friends understand them, rust outputs nice normal DWARF stuff on Linux so gdb debugging just works, Rust uses platform calling convention as much as possible, etc. It means that a wealth of existing tooling just works.

krasin 7y ago
I agree with some of your points; but zero-reliance on libc is the reason why it's so easy to use Go in containers; and Docker is one of the primary reasons why Go is popular. It's what they have got right.
- Skunkleton 7y ago
  You could statically link in libc and get the same effect.
  - IshKebab 7y ago
    Says someone who has never actually tried to do that. You can statically link with Musl. But you can't really statically link with Glibc.
    - josteink 7y ago
      Technically speaking OP only suggested statically linking (a) libc, not glibc specifically, and musl is a libc.

evmar 7y ago

The next time you need to make an HTML treemap like this, try my tool: https://github.com/evmar/webtreemap https://github.com/evmar/webtreemap It provides a command line app that accepts simple space-delimited data and outputs an HTML file. See the doc: https://github.com/evmar/webtreemap#command-line https://github.com/evmar/webtreemap#command-line (It also is available as a JS library for linking in web apps, but the command line app is the one that I end up using the most. I actually built it to visualize binary size exactly like this post and then later generalized it.)

leetrout 7y ago
Any plans to fix (or close if they are fixed) the existing issues?
pacoverdi 7y ago
Another option is to generate a text file in the format expected by flamegraph. Especially useful when the data is hierarchical. https://github.com/brendangregg/FlameGraph https://github.com/brendangregg/FlameGraph Exemple for Java: https://github.com/pcdv/deps-flamegraph/blob/master/README.md https://github.com/pcdv/deps-flamegraph/blob/master/README.m...

fxtentacle 7y ago

TLDR: They included debugging information when they did not intend to.

techsin101 7y ago
What's the fix?
- layoutIfNeeded 7y ago
  Don’t use Go.

lazyjones 7y ago

The author guessed a few things wrong: * fmt.Println pulling in 300KB isn't proof that Go's standard library isn't "well modularized". It's the wonders of Unicode and other code that is actually used. * 900K for the runtime isn't surprising when you have complex garbage collection and goroutine scheduling among other things

majewsky 7y ago
> fmt.Println pulling in 300KB isn't proof that Go's standard library isn't "well modularized". It's the wonders of Unicode and other code that is actually used. I would guess a large part is that it has to pull in the entire reflection library to support format verbs like %#v which renders the argument as a Go literal.
knz42 7y ago
The C library supports unicode too nowadays (at least the GNU C library does) yet linking a simple program that uses printf does not import 300KB of code. The answer comes from the reflection code as the other commenter states.

tytso 7y ago

I'm not sure why people are so worried about the size of the executable file here. If the runtime.pclntab table is never[1] used then it won't be paged into memory, and disk space is mostly free these days. [1] Well, hardly _ever_! (Sorry not sorry for the obligatory Gilbert and Sullivan reference.) If you're using the Go executable on a system without virtual memory support, yeah, that's going to suck, but it appears the Go runtime is horribly bloated and not really suited for super-tiny 16-bit processors in the micro-embedded space. But for something like Cockroachdb, why worry about the file size?

caiobegotti 7y ago
Bandwidth [to transfer big binaries around] is not free however.
- pushpop 7y ago
  True, but you’d need a lot of transfers before it starts to add up but if you run into that edge case then thankfully you don’t actually need the file to be executable during transit so you probably should compress the file for transit and decompress it at the other end (assuming your CI/CD pipeline allows for that) or compile it on the destiny nodes (since Go’s compiler is fast). Admittedly neither are perfect solutions but software architecting is always about making smart compromises.
F-0X 7y ago
> disk space is mostly free these days. This is the only "argument" ever presented, and I don't think it is any good. I care about file sizes. I want to get the most out of my hardware. Not needing to buy another drive is always going to be cheaper for me and every other user.
- trevor-e 7y ago
  Then don't use Go, you aren't their target audience in that case. And I don't mean this in a harsh way, just that Google is clearly opinionated in how they are building Go.

kazinator 7y ago

> there is about 70MB of source code currently in CockroachDB 19.1 That's what is insane here; way more so than Go executable size issues.

aasasd 7y ago

> runtime.pclntab Ah yes, an example of Google's long and descriptive identifier names, a lament of which surfaced here recently. (https://news.ycombinator.com/item?id=21843180 https://news.ycombinator.com/item?id=21843180) > I’m glad we now live in a futuristic utopia where keyboard farts like p, idxcrpm, and x3 are rare.

esprehn 7y ago
What's interesting is that Go encourages letter soup while the rest of Google styles encourage long descriptive names. Sometimes I think it was intentionally done to troll the rest of Google that used long_descriptive_names_ with an 80 col limit (Java argued up to 100 but no one else did, not even JS which uses Java's long names). I always thought it was funny walking around the office seeing everyone with their squished and wrapped code on a huge 32in monitor. Lots of the code in google3 looks like haiku squished to the right margin.
- aasasd 7y ago
  Excellently thought out. Very rational. Truly a language for the twenty-first century.

nickcw 7y ago

> prior to 1.2, the Go linker was emitting a compressed line table, and the program would decompress it upon initialization at run-time. in Go 1.2, a decision was made to pre-expand the line table in the executable file into its final format suitable for direct use at run-time, without an additional decompression step. This is a good choice I think and the author of the article missed the most important point - it uses less memory to have an uncompressed table. This sounds paradoxical but if a table has to be expanded at runtime then it has to be loaded into memory. However if a table is part of the executable, the OS won't even load it into memory unless it is used and will only page the bits into memory that are used. You see the same effect when you compress a binary with UPX (for example) - the size on disk gets smaller, but because the entire executable is decompressed into RAM rather than demand paged in, then it uses more memory.

deleted 7y ago
[deleted]
zamadatix 7y ago
If you decompress it to an mmaped file it'll be one of the first things written to disk under memory pressure anyways and instantly available in normal situations. With the ever decreasing cost of flash and it's ever increasing speed relative to the CPU compression is not really worth what it used to be to startup times 10 years ago though.
- ceph_ 7y ago
  Swapping is fine for workstations and home computers. But high performance machines running in a production environments will absolutely have swap disabled. The performance difference between RAM and disk is not an acceptable tradeoff. RAM will be tightly provisioned and jobs will be killed rather than letting the machine OOM.
  - Galanwe 7y ago
    Are you arguing you prefer your production jobs to be killed rather than slowed ?

dana321 7y ago

Go is a hybrid language. Its not a jvm, but its runtime has jvm-like features such as garbage collection and reflection but also a thread scheduling system called goroutines. I love the fact it is monolithic in nature.. One exe is all you need no matter which platform you use. Everything is statically compiled into the binary. No bundling the jvm and a load of jar files, or lib*.so dependencies.

int_19h 7y ago
Toolkits to precompile a Java app into a single native binary with the runtime and stdlib inside have been around for 20 years or so.

andy_ppp 7y ago

Because Go isn’t C?

narven 7y ago

Use upx https://upx.github.io/ https://upx.github.io/

bsaul 7y ago

sidenote : currently trying to get up to date with the best way to get distributed acid key value storage those days. Is coackroach the new standard ? I tried to find benchmarks comparing it to things like postgres for various use case but only found articles that read like ads.

redis_mlc 7y ago
> the best way to get distributed acid key value storage You will need to define "best way", "distributed" and "acid" based on your requirements. For most people, multi-master MySQL and Redis with vector clocks is a great combination. > Is coackroach the new standard No database less than 5 - 10 years old works in production, so no, it's not a standard. > I tried to find benchmarks comparing it to things like postgres for various use case Well, without more detailed requirements, good luck. Also comparing pg to distributed databases is ... like comparing apples and oranges. Source: experienced DBA.
- bsaul 7y ago
  distributed : able to scale horizontally, ideally over multiple availability zone, to ensure both performance and data safety. acid : able to perform bank account money transfert style transaction. last time i checked, pg was able to shard and work in multiple replication configuration, ensuring a good scalability story up to terabytes of active data.. Does multi master mysql enable row locking ?

ausjke 7y ago

go really should be the same size as c with static link, somehow it's 4x larger in size, why is that? upx can help, with upx, c static binary is still much smaller. both removed debugging info.

boulos 7y ago

Disclosure: I work on Google Cloud (but this applies generally). There seem to be a lot of arguments about disk space (arguably free in 2020), memory (free-ish because as tytso points out, they won’t get backed) and then bandwidth. I think most people saying “bandwidth” might mean “time to fetch”, because GCS, GCR, S3, etc. all have free egress on the transfer from them to GCE/EC2. If you have a self-hosted Docker Hub or something on an EC2 instance, that’s not the case (you may pay Zone to Zone egress of $.01/GB). If you were paying a penny per gigabyte egress, a 100MB-ish binary is only .1 GB and therefore at best 1/1000th of a dollar. On the 16 vCPU hosts that CockroachDB prefers (see their recent benchmarks), that’s equivalent to about 5 seconds of runtime. A fair retort is that in container land, this becomes death by 1000 cuts as each binary includes all the same stuff over and over again (for minikube, the .iso is nearly 1 GB [1] now, but not because of the 50MB binary itself). Even so, a 1GB image takes almost as long to pull from a object store like GCS as a 100 MB image (at many Gbps, the constant factors dominate). If you’re trying to run something this large as a function on Cloud Run or Lambda or Knative, you’ll probably be sad (you’ve burned about 800 vcpu seconds, economically, of compute time) but that’s why there are layers. tl;dr: your 50 or 100 MB binary doesn’t “cost” much, but a 1 GB container image without shared layers does. [1] https://github.com/kubernetes/minikube/issues/5013 https://github.com/kubernetes/minikube/issues/5013

Timothycquinn 7y ago

Why are they not considering keeping the pre 1.2 compressed runtime.pclntab and have the data be decompressed to a separate file on first run. This way, memory footprint is kept low whilst keeping executable size down?

kccqzy 7y ago

Language flame wars aside, Bloaty McBloatface is a wonderful tool to analyze why the binary size is big: https://github.com/google/bloaty https://github.com/google/bloaty I frequently use this tool to answer questions for C++ binaries, another language that has a penchant for producing large executables.

_bxg1 7y ago

todotask 7y ago

Rob Pike submit issue: https://github.com/golang/go/issues/36313 https://github.com/golang/go/issues/36313