10 ms·

Shrink Go binaries 7x with this one weird trick

151 points by FiloSottile 11y ago

jonathonf 11y ago
This "one weird trick" of stripping the binary, then compressing with goupx.
- FiloSottile 11y ago
  First paragraph: > Ok, I lied, there's no weird trick. However, you can easily reduce a Go binary size by more than 6 times with some flags and common tools.
- prymitive 11y ago
  Hard disk drive manufacturers hate him
sofaofthedamned 11y ago
Doesn't upx mess with same page merging though?
- FiloSottile 11y ago
  As long as you don't mind having the entire uncompressed binary in memory, I don't think I see how. Can you elaborate?
  - jsnell 11y ago
    Every instance of the program will have its own private copy of the uncompressed data. For normal uncompressed binaries the pages of the on-disk binary would be shared instead. (Except for any pages that get modified, which would trigger copy-on-write. A classic example would be load-time dynamic linker relocations).
    - noselasd 10y ago
      Sure - but that's a different thing than same-page merging.
lossolo 11y ago
Do NOT strip Go binary: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717172 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717172
- deleted 11y ago
  [deleted]
- userbinator 11y ago
  Fixed in version golang/2:1.1.1-4 That bug is 3 years old and seems to be architecture-specific.
  - lossolo 11y ago
    You are missing the point. Read what was quoted about stripping binaries from Dave Cheney.
  - frio 11y ago
    "Closes: 717172 Changes: golang (2:1.1.1-4) unstable; urgency=low . * Disable stripping, it breaks go binaries on some architectures" The "fix" is that the Debian team stopped stripping the binaries ;).
- FiloSottile 11y ago
  Even if recent discussion is more mixed [0], that is why I linked to [1], recommended not to use strip(1), and explained -ldflags="-s -w" instead :) [0] https://groups.google.com/d/msg/golang-dev/ABppMOjYP6w/6JsfT2MQAwAJ https://groups.google.com/d/msg/golang-dev/ABppMOjYP6w/6JsfT... [1] https://github.com/docker/docker/blob/2a95488f7843a773de2b541a47d9b971a635bfff/project/PACKAGERS.md#stripping-binaries https://github.com/docker/docker/blob/2a95488f7843a773de2b54...
- mordocai 10y ago
  Just be clear here: This is saying don't strip using the standard strip utility. As the author says in their post, go has flags to do stripping for go binaries.
userbinator 11y ago
Note: I don't actually believe a 30MB static binary is a problem in this day and age Depends what that 30MB does, or how much of it is actually needed - it's less of a problem if all those bytes are used in some form or another, than if a significant portion of them are "dead code" that is never used and only serves to take up space in memory, on disk, and in communications channels whenever it's transferred somewhere. and I would not trade (build time | complexity | performance | debug-ability) for it On the contrary, smaller binaries are usually going to be better at all of those, except perhaps "debug-ability" if you keep the debugging symbols in the same file.
- dozzie 11y ago
  >> Note: I don't actually believe a 30MB static binary is a problem in this day and age > Depends what that 30MB does, or how much of it is actually needed [....] Moreover, let's consider a CLI tool suite. Dozen little tools, each doing perfectly what it was designed to (think of coreutils, util-linux, sysstat, binutils, or iproute2). Such a suite suddenly weighs 300MB, more than X.Org or PostgreSQL (or the two combined). Does it really do more than any of these two to justify the size? I doubt it.
  - userbinator 11y ago
    The availability of huge disks has probably skewed the perception of sizes to some extent. 300MB is more than sufficient for an entire operating system, including a GUI and several dozen CLI tools. A full install of Windows 98 was 175MB. On the other hand, internet speeds have not increased quite as dramatically, so we can still relate with stories of "this app is how big? And it only does what?"
    - skykooler 10y ago
      Indeed. Just the other day I was trying (with no luck) to find a simple drawing app for Android (something like Paint on windows) that was less than 18 MB. Seriously, that sort of app should not need huge executables like that.
shirro 11y ago
30MB binaries can be a bit of a problem for some people. I started developing a system from home and when deploying to a VPS for testing and production. Transferring Go binaries was like waiting for a floppy disk to copy due to poor Internet infrastructure. Compression didn't help much. It turned out to be significantly faster to push to a remote git and have a hook do the build. I think before you look to compress a go binary consider if there aren't other options.
- dietrichepp 11y ago
  I feel your pain. My home internet connection is 24 Mbit/s down, but only 1 Mbit/s up. I asked my ISP for more, but it's basically impossible to get. Thank goodness for multipart uploads, at least.
  - michaelcampbell 11y ago
    I've ameliorated this somewhat (I have 18/1), albeit at the expense of space, by uploading things to a common place, then copying or (sym)linking them to the final spot on the target computer. This gives me the ability to `rsync` them up rather than copy, using its ability to only transfer the pieces of the files that have changed. We have a ~20MB .war file that I have done this with over airport wifi, and it ended up only transfer a few 10's of KB (maybe 100) between compiles.
    - dietrichepp 10y ago
      I do something similar. I have a database of all my files on S3 and their SHA-256 hashes. This not only means that I can skip uploading files which haven't changed, but I can do S3->S3 copies for files which are the same. But it's still too often that I have 10s or 100s of MB of data to upload. At 1 Mbit/s, it takes a half hour for every 100 MB!
      - michaelcampbell 10y ago
        Indeed, sometimes it just sucks and you have to bite the bullet.
thejay 11y ago
Op discovered exe packer.
- rhexs 10y ago
  He's a full stack hacker, from the browser all the way down to ruby.
  - falcolas 10y ago
    Anybody who truly understands the technology and minutia behind modern browsers strikes me as almost more interesting and impressive than understanding Linux in this day and age. This isn't belittling operations folks, this is more of a comment on how complicated, complex, and obfuscated modern browsers are.
santaclaus 11y ago
> I don't actually believe a 30MB static binary is a problem in this day and age Well, binary size is pretty important for embedded work. Or mobile work.
bobwaycott 11y ago
This is the second time in about a week that a post on HN has included a variation of "this weird trick" (the other being jlongster's article dealing with React) in the title. I can't help but parse these articles as equivalent to all the horrible ad spam all over the web about "this one weird trick" that will flatten your belly, tone your butt, help you make millions, give you a bigger penis, eradicate your cellulite, and a host of other "improvements" to your life. Let's maybe spend a couple extra minutes coming up with less link-baity titles.
- foota 11y ago
  I think the title is a parody of those titles.
  - bobwaycott 10y ago
    I know. The first sentence of the post did not elude me.
  - Godel_unicode 10y ago
    The fact that it's a parody doesn't preclude it being annoying. Also, it's not actually that good a parody in my opinion. It's a click-bait title which links to an article with a really obvious message (lose weight:eat less::shrink binary:packer)
- ascendantlogic 10y ago
  It's fascinating to watch you be so literal that the subtle mocking of those exact things has gone completely over your head.
  - bobwaycott 10y ago
    About as fascinating as two commenters who think I can't recognize the mocking/parody, and still find it to be equivalent to all the shitty content on the web using this one particularly "weird" headline "trick"? Criticizing something is not a signal that one missed the joke.
codedokode 11y ago
Note that there are downsides with this method: - the startup time increases - if you run several instances of an app, they consume more memory because code uses anon mapping instead of file mapping - the whole executable file is loaded to memory at startup
- modeless 11y ago
  A bigger downside: various virus scanners flag UPX-packed executables as suspicious.
  - fileoffset 10y ago
    Not entirely true. Most AV will only flag 'non-standard' UPX packed executables as suspicious. Standard UPX packed binaries are easily unpacked by the scanner and the contents scanned without notifying user.
    - pilif 10y ago
      It's anecdotal of course, but a small application of ours went from about one false-positive per month to no false positive in over a year when I stopped UPX packing it.
      - SyneRyder 10y ago
        More anecdata here, but I've experienced the same thing as well (not quite as frequent as one false-positive a month though). UPX just seems to occasionally trip up the antivirus heuristic scans until you can get your application whitelisted.
- thrownaway2424 10y ago
  "whole executable file is loaded to memory at startup" might be something you were going to do anyway, if you are running a latency-sensitive service. There's nothing worse than having to page in parts of the program after it starts serving.
themartorana 11y ago
Honest question - mathematically, how can something be 7x smaller if x is not pre-defined? I always find it weird that anything can be more that 100% less than anything else because, you know, zero. Am I way, way off base? Should I perhaps retake high school? Edit: the smaller number is assumed 1x. Got it. Still feels weird.
- monochromatic 11y ago
  It's not well-defined nomenclature. Generally what people mean when they say 7x less is 1/7 as much.
- seeing 11y ago
  If a file is 100kb and is compressed to 50kb, that's 2x smaller. 50kb is still over zero.
- FiloSottile 11y ago
  You might be overthinking it a bit :) I always see it the X used as the multiplication symbol, meaning "seven times smaller". You know, like when in games you get a 2x multiplier. I doubt that is the algebra "X" ;)
- michaelcampbell 11y ago
  You're right, but as another poster noted "2x smaller" generally means 1/2 the original size. That usage, to me, is a horrible misuse of English, and is a pet peeve. You also see it with slower; "the unoptimized code is 2x slower". Huh? Do you mean "takes 2x the time"? I cringe every time.
  - maho 10y ago
    It makes perfect sense to me. I interpret "x" as shorthand/lazyhand notation of the Angloamerican multiplication symbol "×", which I verbally expand to the word "times". Furthermore, calling something "two times smaller" is equivalent to calling it "half the size", since "smallness" is simply the logical and mathematical inverse of "bigness", or "size". The same argument holds for unoptimized code being two times slower, or half as fast. (Slowness is the inverse of speed.)
pklausler 11y ago
Not actually specific to golang.
jedisct1 10y ago
Problem is that if you distribute UPX packed binaries, dumb antivirus will all raise an alarm.
fibo 10y ago
Bravo Filippo!
chetanahuja 10y ago
Note: I don't actually believe a 30MB static binary is a problem in this day and age If you expand your context from "day and age" to "country, bandwidth provider and pricing" you might start believing the problem a bit more
elcapitan 10y ago
Waiting for the rebuttal "ENLARGE YOUR BINARY WITH THIS SIMPLE PILL".
- sov 10y ago
  "With this simple .PLL!"
  - elcapitan 10y ago
    Had to look that up :D
- skoczymroczny 10y ago
  Admins hate him!
floatboth 10y ago
Also useful for Haskell: https://www.fpcomplete.com/blog/2015/05/haskell-web-server-in-5mb https://www.fpcomplete.com/blog/2015/05/haskell-web-server-i...
donatj 10y ago
Haha, the author linked to my poorly written poorly conceived old grumbling post. I'm honored. I should update the post as the binaries come out especially larger in the new versions of Go written in Go. I believe Hello Worldis up to at least 3mb.
- kevincox 10y ago
  Isn't the output supposed to be identical between the C and Go versions?

151 points by FiloSottile 11y ago

jonathonf 11y ago

This "one weird trick" of stripping the binary, then compressing with goupx.

FiloSottile 11y ago
First paragraph: > Ok, I lied, there's no weird trick. However, you can easily reduce a Go binary size by more than 6 times with some flags and common tools.
prymitive 11y ago
Hard disk drive manufacturers hate him

sofaofthedamned 11y ago

Doesn't upx mess with same page merging though?

FiloSottile 11y ago
As long as you don't mind having the entire uncompressed binary in memory, I don't think I see how. Can you elaborate?
- jsnell 11y ago
  Every instance of the program will have its own private copy of the uncompressed data. For normal uncompressed binaries the pages of the on-disk binary would be shared instead. (Except for any pages that get modified, which would trigger copy-on-write. A classic example would be load-time dynamic linker relocations).
  - noselasd 10y ago
    Sure - but that's a different thing than same-page merging.

lossolo 11y ago

Do NOT strip Go binary: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717172 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717172

deleted 11y ago
[deleted]
userbinator 11y ago
Fixed in version golang/2:1.1.1-4 That bug is 3 years old and seems to be architecture-specific.
- lossolo 11y ago
  You are missing the point. Read what was quoted about stripping binaries from Dave Cheney.
- frio 11y ago
  "Closes: 717172 Changes: golang (2:1.1.1-4) unstable; urgency=low . * Disable stripping, it breaks go binaries on some architectures" The "fix" is that the Debian team stopped stripping the binaries ;).
FiloSottile 11y ago
Even if recent discussion is more mixed [0], that is why I linked to [1], recommended not to use strip(1), and explained -ldflags="-s -w" instead :) [0] https://groups.google.com/d/msg/golang-dev/ABppMOjYP6w/6JsfT2MQAwAJ https://groups.google.com/d/msg/golang-dev/ABppMOjYP6w/6JsfT... [1] https://github.com/docker/docker/blob/2a95488f7843a773de2b541a47d9b971a635bfff/project/PACKAGERS.md#stripping-binaries https://github.com/docker/docker/blob/2a95488f7843a773de2b54...
mordocai 10y ago
Just be clear here: This is saying don't strip using the standard strip utility. As the author says in their post, go has flags to do stripping for go binaries.

userbinator 11y ago

Note: I don't actually believe a 30MB static binary is a problem in this day and age Depends what that 30MB does, or how much of it is actually needed - it's less of a problem if all those bytes are used in some form or another, than if a significant portion of them are "dead code" that is never used and only serves to take up space in memory, on disk, and in communications channels whenever it's transferred somewhere. and I would not trade (build time | complexity | performance | debug-ability) for it On the contrary, smaller binaries are usually going to be better at all of those, except perhaps "debug-ability" if you keep the debugging symbols in the same file.

dozzie 11y ago
>> Note: I don't actually believe a 30MB static binary is a problem in this day and age > Depends what that 30MB does, or how much of it is actually needed [....] Moreover, let's consider a CLI tool suite. Dozen little tools, each doing perfectly what it was designed to (think of coreutils, util-linux, sysstat, binutils, or iproute2). Such a suite suddenly weighs 300MB, more than X.Org or PostgreSQL (or the two combined). Does it really do more than any of these two to justify the size? I doubt it.
- userbinator 11y ago
  The availability of huge disks has probably skewed the perception of sizes to some extent. 300MB is more than sufficient for an entire operating system, including a GUI and several dozen CLI tools. A full install of Windows 98 was 175MB. On the other hand, internet speeds have not increased quite as dramatically, so we can still relate with stories of "this app is how big? And it only does what?"
  - skykooler 10y ago
    Indeed. Just the other day I was trying (with no luck) to find a simple drawing app for Android (something like Paint on windows) that was less than 18 MB. Seriously, that sort of app should not need huge executables like that.

shirro 11y ago

30MB binaries can be a bit of a problem for some people. I started developing a system from home and when deploying to a VPS for testing and production. Transferring Go binaries was like waiting for a floppy disk to copy due to poor Internet infrastructure. Compression didn't help much. It turned out to be significantly faster to push to a remote git and have a hook do the build. I think before you look to compress a go binary consider if there aren't other options.

dietrichepp 11y ago
I feel your pain. My home internet connection is 24 Mbit/s down, but only 1 Mbit/s up. I asked my ISP for more, but it's basically impossible to get. Thank goodness for multipart uploads, at least.
- michaelcampbell 11y ago
  I've ameliorated this somewhat (I have 18/1), albeit at the expense of space, by uploading things to a common place, then copying or (sym)linking them to the final spot on the target computer. This gives me the ability to `rsync` them up rather than copy, using its ability to only transfer the pieces of the files that have changed. We have a ~20MB .war file that I have done this with over airport wifi, and it ended up only transfer a few 10's of KB (maybe 100) between compiles.
  - dietrichepp 10y ago
    I do something similar. I have a database of all my files on S3 and their SHA-256 hashes. This not only means that I can skip uploading files which haven't changed, but I can do S3->S3 copies for files which are the same. But it's still too often that I have 10s or 100s of MB of data to upload. At 1 Mbit/s, it takes a half hour for every 100 MB!
    - michaelcampbell 10y ago
      Indeed, sometimes it just sucks and you have to bite the bullet.

thejay 11y ago

Op discovered exe packer.

rhexs 10y ago
He's a full stack hacker, from the browser all the way down to ruby.
- falcolas 10y ago
  Anybody who truly understands the technology and minutia behind modern browsers strikes me as almost more interesting and impressive than understanding Linux in this day and age. This isn't belittling operations folks, this is more of a comment on how complicated, complex, and obfuscated modern browsers are.

santaclaus 11y ago

> I don't actually believe a 30MB static binary is a problem in this day and age Well, binary size is pretty important for embedded work. Or mobile work.

bobwaycott 11y ago

This is the second time in about a week that a post on HN has included a variation of "this weird trick" (the other being jlongster's article dealing with React) in the title. I can't help but parse these articles as equivalent to all the horrible ad spam all over the web about "this one weird trick" that will flatten your belly, tone your butt, help you make millions, give you a bigger penis, eradicate your cellulite, and a host of other "improvements" to your life. Let's maybe spend a couple extra minutes coming up with less link-baity titles.

foota 11y ago
I think the title is a parody of those titles.
- bobwaycott 10y ago
  I know. The first sentence of the post did not elude me.
- Godel_unicode 10y ago
  The fact that it's a parody doesn't preclude it being annoying. Also, it's not actually that good a parody in my opinion. It's a click-bait title which links to an article with a really obvious message (lose weight:eat less::shrink binary:packer)
ascendantlogic 10y ago
It's fascinating to watch you be so literal that the subtle mocking of those exact things has gone completely over your head.
- bobwaycott 10y ago
  About as fascinating as two commenters who think I can't recognize the mocking/parody, and still find it to be equivalent to all the shitty content on the web using this one particularly "weird" headline "trick"? Criticizing something is not a signal that one missed the joke.

codedokode 11y ago

Note that there are downsides with this method: - the startup time increases - if you run several instances of an app, they consume more memory because code uses anon mapping instead of file mapping - the whole executable file is loaded to memory at startup

modeless 11y ago
A bigger downside: various virus scanners flag UPX-packed executables as suspicious.
- fileoffset 10y ago
  Not entirely true. Most AV will only flag 'non-standard' UPX packed executables as suspicious. Standard UPX packed binaries are easily unpacked by the scanner and the contents scanned without notifying user.
  - pilif 10y ago
    It's anecdotal of course, but a small application of ours went from about one false-positive per month to no false positive in over a year when I stopped UPX packing it.
    - SyneRyder 10y ago
      More anecdata here, but I've experienced the same thing as well (not quite as frequent as one false-positive a month though). UPX just seems to occasionally trip up the antivirus heuristic scans until you can get your application whitelisted.
thrownaway2424 10y ago
"whole executable file is loaded to memory at startup" might be something you were going to do anyway, if you are running a latency-sensitive service. There's nothing worse than having to page in parts of the program after it starts serving.

themartorana 11y ago

Honest question - mathematically, how can something be 7x smaller if x is not pre-defined? I always find it weird that anything can be more that 100% less than anything else because, you know, zero. Am I way, way off base? Should I perhaps retake high school? Edit: the smaller number is assumed 1x. Got it. Still feels weird.

monochromatic 11y ago
It's not well-defined nomenclature. Generally what people mean when they say 7x less is 1/7 as much.
seeing 11y ago
If a file is 100kb and is compressed to 50kb, that's 2x smaller. 50kb is still over zero.
FiloSottile 11y ago
You might be overthinking it a bit :) I always see it the X used as the multiplication symbol, meaning "seven times smaller". You know, like when in games you get a 2x multiplier. I doubt that is the algebra "X" ;)
michaelcampbell 11y ago
You're right, but as another poster noted "2x smaller" generally means 1/2 the original size. That usage, to me, is a horrible misuse of English, and is a pet peeve. You also see it with slower; "the unoptimized code is 2x slower". Huh? Do you mean "takes 2x the time"? I cringe every time.
- maho 10y ago
  It makes perfect sense to me. I interpret "x" as shorthand/lazyhand notation of the Angloamerican multiplication symbol "×", which I verbally expand to the word "times". Furthermore, calling something "two times smaller" is equivalent to calling it "half the size", since "smallness" is simply the logical and mathematical inverse of "bigness", or "size". The same argument holds for unoptimized code being two times slower, or half as fast. (Slowness is the inverse of speed.)

pklausler 11y ago

Not actually specific to golang.

jedisct1 10y ago

Problem is that if you distribute UPX packed binaries, dumb antivirus will all raise an alarm.

fibo 10y ago

Bravo Filippo!

chetanahuja 10y ago

Note: I don't actually believe a 30MB static binary is a problem in this day and age If you expand your context from "day and age" to "country, bandwidth provider and pricing" you might start believing the problem a bit more

elcapitan 10y ago

Waiting for the rebuttal "ENLARGE YOUR BINARY WITH THIS SIMPLE PILL".

sov 10y ago
"With this simple .PLL!"
- elcapitan 10y ago
  Had to look that up :D
skoczymroczny 10y ago
Admins hate him!

floatboth 10y ago

Also useful for Haskell: https://www.fpcomplete.com/blog/2015/05/haskell-web-server-in-5mb https://www.fpcomplete.com/blog/2015/05/haskell-web-server-i...

donatj 10y ago

Haha, the author linked to my poorly written poorly conceived old grumbling post. I'm honored. I should update the post as the binaries come out especially larger in the new versions of Go written in Go. I believe Hello Worldis up to at least 3mb.

kevincox 10y ago
Isn't the output supposed to be identical between the C and Go versions?