4 ms·

People who compress their go binaries (or any other binaries, really) - please be aware that thus doing, you stop the OS from being able to page out your execut

People who compress their go binaries (or any other binaries, really) - please be aware that thus doing, you stop the OS from being able to page out your executable (rarely a big loss), and also to be unable to share executable pages (not a huge loss for a 2MB executable, a huge loss for a 100MB executable). If there's only one copy of a program running, it won't matter - but if you are running hundreds of copies (even docerized and stuff), you are likely better off NOT upxing.

foobarchu 9y ago
I'm curious, why did you feel the need to call out go specifically? Does this apply more to go than other source languages?
- MichaelBurge 9y ago
  Many go programmers statically compile their code, which should make the paging problem worse.
- takeda 9y ago
  If you look at other responses large number of them actually mentions that UPX works really with go and docker, I believe it was response to that.
- beagle3 9y ago
  I wrote "or any other binaries, really". It's just that (a) go compiles static binaries which makes UPX especially effective, unlike e.g. common C and C++ projects with their hundreds of .so/.dlls ; Delphi/FPC and Nim or the other two ecosystems that share this trait, but neither is as common as go. (b) it's not good for non-static binaries, that is C#, Java, Python have no benefit from this. (c) At the time I posted it, there were already 3 or 4 posts extolling the virtues of compressing Go executables.
smegel 9y ago
> you stop the OS from being able to page out your executable (rarely a big loss) Why is this exactly?
- jlebar 9y ago
  I think what OP meant is, if you have a page that's backed by disk, then when you're low on RAM, the OS can simply drop that page -- it's sitting there on disk if you ever need it back. But if you have a page that's not backed by disk, then you have to write it out to disk before you can drop it.
- beagle3 9y ago
  Indeed, as jlebar notes - since the actual executable code (the thing that the CPU executes) does not exist directly in the executable file, the OS will have to write it out to the swap if it needs the memory (unlike uncompressed files, where it just reuses the memory and later reloads from the executable file). It is rarely a big loss, because executables that are in use tend to remain in memory if the program is actually active. If you have a 300MB demon that sleeps, though, you will likely notice a swap out to magnetic disk.
  - pgeorgi 9y ago
    Also if you have the same executable running several times on a machine, they share the RAM for the code (read-only pages, which should be most of them). That doesn't work for UPX because each execution decompresses anew, which makes it a "new executable" from the OS' point of view. The only thing that would help is kernel space merging, but that's really only activated for some virtual machines.
    - icebraining 9y ago
      that's really only activated for some virtual machines. And even there, it's a security threat, since it enables cross-VM timing and other attacks.
FooBarWidget 9y ago
> and also to be unable to share executable pages (not a huge loss for a 2MB executable, a huge loss for a 100MB executable). I don't think people care about that nowadays, seeing how popular Docker containers are. I think Docker containers already make it so that you cannot share executable memory between different containers because each one runs in its private namespace.
- beagle3 9y ago
  Do namespaces separate file system caches? That would surprise me.
  - viraptor 9y ago
    The only one that could matter here would be mnt. And since bind mounts normally wouldn't cache pages separately, I don't think that would happen for the namespaces either. Happy to be proven wrong though. More specifically I wouldn't expect "free -m" to produce different result depending on the namespace it's run in.
  - ptman 9y ago
    IIRC it depends on which storage backend for docker you use: https://web.archive.org/web/20170405122924/https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ https://web.archive.org/web/20170405122924/https://developer...

foobarchu 9y ago

I'm curious, why did you feel the need to call out go specifically? Does this apply more to go than other source languages?

MichaelBurge 9y ago
Many go programmers statically compile their code, which should make the paging problem worse.
takeda 9y ago
If you look at other responses large number of them actually mentions that UPX works really with go and docker, I believe it was response to that.
beagle3 9y ago
I wrote "or any other binaries, really". It's just that (a) go compiles static binaries which makes UPX especially effective, unlike e.g. common C and C++ projects with their hundreds of .so/.dlls ; Delphi/FPC and Nim or the other two ecosystems that share this trait, but neither is as common as go. (b) it's not good for non-static binaries, that is C#, Java, Python have no benefit from this. (c) At the time I posted it, there were already 3 or 4 posts extolling the virtues of compressing Go executables.

smegel 9y ago

> you stop the OS from being able to page out your executable (rarely a big loss) Why is this exactly?

jlebar 9y ago
I think what OP meant is, if you have a page that's backed by disk, then when you're low on RAM, the OS can simply drop that page -- it's sitting there on disk if you ever need it back. But if you have a page that's not backed by disk, then you have to write it out to disk before you can drop it.
beagle3 9y ago
Indeed, as jlebar notes - since the actual executable code (the thing that the CPU executes) does not exist directly in the executable file, the OS will have to write it out to the swap if it needs the memory (unlike uncompressed files, where it just reuses the memory and later reloads from the executable file). It is rarely a big loss, because executables that are in use tend to remain in memory if the program is actually active. If you have a 300MB demon that sleeps, though, you will likely notice a swap out to magnetic disk.
- pgeorgi 9y ago
  Also if you have the same executable running several times on a machine, they share the RAM for the code (read-only pages, which should be most of them). That doesn't work for UPX because each execution decompresses anew, which makes it a "new executable" from the OS' point of view. The only thing that would help is kernel space merging, but that's really only activated for some virtual machines.
  - icebraining 9y ago
    that's really only activated for some virtual machines. And even there, it's a security threat, since it enables cross-VM timing and other attacks.

FooBarWidget 9y ago

> and also to be unable to share executable pages (not a huge loss for a 2MB executable, a huge loss for a 100MB executable). I don't think people care about that nowadays, seeing how popular Docker containers are. I think Docker containers already make it so that you cannot share executable memory between different containers because each one runs in its private namespace.

beagle3 9y ago
Do namespaces separate file system caches? That would surprise me.
- viraptor 9y ago
  The only one that could matter here would be mnt. And since bind mounts normally wouldn't cache pages separately, I don't think that would happen for the namespaces either. Happy to be proven wrong though. More specifically I wouldn't expect "free -m" to produce different result depending on the namespace it's run in.
- ptman 9y ago
  IIRC it depends on which storage backend for docker you use: https://web.archive.org/web/20170405122924/https://developers.redhat.com/blog/2014/09/30/overview-storage-scalability-docker/ https://web.archive.org/web/20170405122924/https://developer...