6 ms·

You can compress with upx (at the cost of increased startup time in the order of hundreds of ms, which is okay for servers) and/or not include all debug symbols

You can compress with upx (at the cost of increased startup time in the order of hundreds of ms, which is okay for servers) and/or not include all debug symbols. Doing both usually shaves >60% off a binary.

oefrha 7y ago
<deleted> Edit: somehow read “doing both” as “doing either”. Just ignore this.
- wongarsu 7y ago
  That's already 30% off for free, now run that through upx [1] 1: https://upx.github.io/ https://upx.github.io/
- robfig 7y ago
  UPX compresses the binary and introduces a decompression step on startup, so you run exactly the input binary Stripped means that the compiler does not include debug symbols. They are completely different - you can use either or both.
amscanne 7y ago
UPX transforms demand-paged, reclaimable page cache memory into a blob of unreclaimable anonymous memory. It makes no sense for most use cases where I’ve seen it. It adds runtime costs both in terms of start-up and memory usage. Maybe it helps in terms of binary sizes for downloads — but those are often compressed anyways! E.g. Your docker images are compressed and UPX’ed binaries in a layer aren’t buying you anything (just adding runtime costs).
- Carpetsmoker 7y ago
  For a lot of applications the increase in startup size and memory usage are negligible. The startup time is increased by dozens or hundreds of ms, which is not a lot for a server application. I tried to measure the increase in memory, but wasn't really able to, so either it's a very subtle difference or it's very small. Not everything uses Docker.
  - charleslmunger 7y ago
    UPXed binaries have their code pages mapped as dirty - this means the OS can't page them back to disk if needs or wants to. In some cases, that's an acceptable cost to pay - for low-latency servers you might want to mlock all your executable pages so there's no risk of a page fault and disk read killing your tail latency. Of course if you're doing that then you have to either pay the full cost of your binary size in memory, or you have to have some warm up phase and then hope that everything you need is loaded then. In the first case you suddenly care a lot about binary size, because memory is quite a bit more expensive than disk. But one valuable reason to use something like UPX is that you can attach a crappy and thus inexpensive disk to servers that you're not using for actual storage. Compression on disk lets you load from a slow disk faster, and of you weren't paging to disk anyway then UPX doesn't have much of a cost. But if you're on a traditional desktop operating system, UPX will increase your effective memory footprint, and force writing to swap instead of merely dropping pages. On Android, which doesn't swap, you'll significantly increase your memory footprint.
- jniedrauer 7y ago
  I have one system I target where, bizarrely, persistent storage is the bottleneck, not volatile memory or startup speed. In this one case, UPX makes a whole lot of sense.
- ComputerGuru 7y ago
  I used to UPX single-file binaries (think Windows freeware) because the download is runnable and does not need to be extracted (users get confused even today by zip files). I stopped because a number of big name corporate A.V. software simply flags all non-whitelisted UPX-compresses binaries as viruses.
barrkel 7y ago
UPX makes sense if you're trying to fit your executable onto fixed-size media from which it needs to execute (e.g. a floppy disk or USB drive), and almost nowhere else.
- Carpetsmoker 7y ago
  Reduced transfer times are good. Not everyone has a 100MB/s internet connection (in fact, most of the world doesn't).
  - deleted 7y ago
    [deleted]
  - TheDong 7y ago
    Okay, so transfer it using `rsync -z` to compress it over the wire, or gzip it first, and extract it on the other end. gzip probably gives you better compression ratios too.
  - Const-me 7y ago
    People don’t often directly download and run software binaries. On Windows, they download .msi packages, or .exe installers with embedded MSIs. On Linux, people download .deb or .rpm packages. All of the above packaging formats are already compressed. Also, even if you publish raw binaries without an installer package, HTTP protocol supports compression. Usually quite easy to implement, couple lines in a web server config.
    - Carpetsmoker 7y ago
      Yes, there are many solutions. upx just gives you an easy, transparent, and convenient way to compress a binary without worrying about web server configurations and whatnot.
      - Const-me 7y ago
        I have used UPX on Windows some 15 years ago and liked it a lot these days. My primary motivation was not network transfer speed, it was HDD storage, bandwidth, and especially latency. It was faster to sequentially read the complete binary, compared to reading individual memory pages as required. Nowadays disk space is very cheap, disks are often solid state with ridiculously high IOPS, but antivirus software became much worse and likely to mark a UPX compressed binary as malware.

oefrha 7y ago

<deleted> Edit: somehow read “doing both” as “doing either”. Just ignore this.

wongarsu 7y ago
That's already 30% off for free, now run that through upx [1] 1: https://upx.github.io/ https://upx.github.io/
robfig 7y ago
UPX compresses the binary and introduces a decompression step on startup, so you run exactly the input binary Stripped means that the compiler does not include debug symbols. They are completely different - you can use either or both.

amscanne 7y ago

UPX transforms demand-paged, reclaimable page cache memory into a blob of unreclaimable anonymous memory. It makes no sense for most use cases where I’ve seen it. It adds runtime costs both in terms of start-up and memory usage. Maybe it helps in terms of binary sizes for downloads — but those are often compressed anyways! E.g. Your docker images are compressed and UPX’ed binaries in a layer aren’t buying you anything (just adding runtime costs).

Carpetsmoker 7y ago
For a lot of applications the increase in startup size and memory usage are negligible. The startup time is increased by dozens or hundreds of ms, which is not a lot for a server application. I tried to measure the increase in memory, but wasn't really able to, so either it's a very subtle difference or it's very small. Not everything uses Docker.
- charleslmunger 7y ago
  UPXed binaries have their code pages mapped as dirty - this means the OS can't page them back to disk if needs or wants to. In some cases, that's an acceptable cost to pay - for low-latency servers you might want to mlock all your executable pages so there's no risk of a page fault and disk read killing your tail latency. Of course if you're doing that then you have to either pay the full cost of your binary size in memory, or you have to have some warm up phase and then hope that everything you need is loaded then. In the first case you suddenly care a lot about binary size, because memory is quite a bit more expensive than disk. But one valuable reason to use something like UPX is that you can attach a crappy and thus inexpensive disk to servers that you're not using for actual storage. Compression on disk lets you load from a slow disk faster, and of you weren't paging to disk anyway then UPX doesn't have much of a cost. But if you're on a traditional desktop operating system, UPX will increase your effective memory footprint, and force writing to swap instead of merely dropping pages. On Android, which doesn't swap, you'll significantly increase your memory footprint.
jniedrauer 7y ago
I have one system I target where, bizarrely, persistent storage is the bottleneck, not volatile memory or startup speed. In this one case, UPX makes a whole lot of sense.
ComputerGuru 7y ago
I used to UPX single-file binaries (think Windows freeware) because the download is runnable and does not need to be extracted (users get confused even today by zip files). I stopped because a number of big name corporate A.V. software simply flags all non-whitelisted UPX-compresses binaries as viruses.

barrkel 7y ago

UPX makes sense if you're trying to fit your executable onto fixed-size media from which it needs to execute (e.g. a floppy disk or USB drive), and almost nowhere else.

Carpetsmoker 7y ago
Reduced transfer times are good. Not everyone has a 100MB/s internet connection (in fact, most of the world doesn't).
- deleted 7y ago
  [deleted]
- TheDong 7y ago
  Okay, so transfer it using `rsync -z` to compress it over the wire, or gzip it first, and extract it on the other end. gzip probably gives you better compression ratios too.
- Const-me 7y ago
  People don’t often directly download and run software binaries. On Windows, they download .msi packages, or .exe installers with embedded MSIs. On Linux, people download .deb or .rpm packages. All of the above packaging formats are already compressed. Also, even if you publish raw binaries without an installer package, HTTP protocol supports compression. Usually quite easy to implement, couple lines in a web server config.
  - Carpetsmoker 7y ago
    Yes, there are many solutions. upx just gives you an easy, transparent, and convenient way to compress a binary without worrying about web server configurations and whatnot.
    - Const-me 7y ago
      I have used UPX on Windows some 15 years ago and liked it a lot these days. My primary motivation was not network transfer speed, it was HDD storage, bandwidth, and especially latency. It was faster to sequentially read the complete binary, compared to reading individual memory pages as required. Nowadays disk space is very cheap, disks are often solid state with ridiculously high IOPS, but antivirus software became much worse and likely to mark a UPX compressed binary as malware.