7 ms·

Sometimes people ask me why I'm so skeptical about software delivery techniques involving bundling and static compilation. This is a thing that people sometimes

Sometimes people ask me why I'm so skeptical about software delivery techniques involving bundling and static compilation. This is a thing that people sometimes ask.

0xbadcafebee 4y ago
It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "replacing a jar in a zip file".
- skybrian 4y ago
  The situation for Go seems to have gotten a bit better with Go 1.18. Go binaries now include the versions of all the modules in them, and this can be listed with the "go version" command. That won't help you rebuild the binary, but at least you can find them.
  - jewel 4y ago
    That's fantastic. Since Go has gone all-in on static compiling, it'd be awesome if they made it possible to update a dependency without recompiling (or even having the source code for) the rest of the program. In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.
    - er4hn 4y ago
      Wouldn't that require the ABI, or at least the API, to be the same across versions of the affected library? That feels like it may be possible with enough tooling, but still very hard to enforce.
      - 0xbadcafebee 4y ago
        The library being patched needs to use the same ABI, yes. So you can usually only patch the same version of a library, not upgrade it. Afaik, the ELF format allows you to replace sections in a binary. The code (or data) is packed as "Sections" and mapped to memory locations. You use a tool to unpack the ELF sections and replace them. The application accesses those sections of memory and uses whatever ABI the application and library are designed to interact with. As long as you patch a section with the same ABI that an app/library used before, nobody will notice. And the ELF ABI never changes. But of course, not every system uses ELFs. But all this depends on the compiler/linker/etc and how they generate sections and store them in the binary (for example, is it all compiled into one blob-section, or does it store libraries as separate sections, which might be inefficient?). I somehow doubt Go does things in such a way to allow this kind of patching. They probably generate a giant blob full of random functions and record versions in a header without the ability to extricate a whole library.
    - kllrnohj 4y ago
      That doesn't seem possible. The benefits of static linking are that unused methods, symbols, etc... can be stripped as well as cross-library optimizations like in-lining. There's not necessarily going to be library lines left at all, at least not in any way that looks anything like upstream. And it's also going to vary from binary to binary, depending on what exactly they used from the dependency. What you're describing would be more akin to shared linkage just in a single blob. That seems like it'd be a worst-of-both-worlds result.
      - skybrian 4y ago
        Instead of "update without compiling", maybe it would be better to attach an archive of the source to the binary, so you can update by compiling? Or maybe attach some suitable intermediate format?
- kevincox 4y ago
  I also like now "single file deployment" is considered a feature. It makes it easier for people without proper packaging and distribution tools. Of course these people without packing and distribution tools will now have a hard time finding out everywhere they have copied these binaries to.
  - LaGrange 4y ago
    packageblabla-1.2.3.deb is a single file.
    - Beltalowda 4y ago
      That will only work on Debian. And that "single file" requires also the libfoo-1.5.deb file (must be >=1.5, not 1.4 and not higher than 2.0!) etc.
      - WesolyKubeczek 4y ago
        The thing about "single file deployment" is that it shouldn't have external requirements, otherwise it's not a single file deployment anymore.
        kelnos 4y ago
        ... which is why the parent is (IMO correctly) claiming that the grandparent's assertion is incorrect that a Debian package is a "single file deployment".
        WesolyKubeczek 4y ago
        I think both grandparent you’re referring to and its parent are lost in the woods.
        LaGrange 4y ago
        Oh no, no I'm not, and the point of the joke was that a deb is as much of a single file as your compiled go thing. It's... well, it's a single file, it's a single artifact. And yes, it has dependencies! So does your go file, I'm afraid.
- shagie 4y ago
  I'm not as familiar with Go... I do know that we use a dependency scan that looks at the docker images and recognizes vulnerable versions of various Java libraries. With Go, can you look at a compiled binary in an image on the artifact repo and identify what libraries it used and if those libraries have vulnerabilities? (and it appears that my question was answered in a sibling comment)
  - hedora 4y ago
    Java's "statically link then scan" approach doesn't really work though. (Some sibling threads explain why.) Even Linux distributions where rebuilding the world is easy (like Debian) insist on breaking dependencies out into shared libaries, since tracking ad hoc vendored dependencies is just too difficult. For example, sometimes people play dynamic loading and renaming tricks to allow multiple versions of the same Java library to coexist. That can usually be detected by vulnerability scanners, but not always.
    - shagie 4y ago
      I'm not sure what you mean by "statically link then scan" in the Java context. The fingerprint of the vulnerable class files are the same no matter where they are stored. They could be bundled together, or shoved in a fat jar, or exploded into all their classes -- but the class file always is distinct and identifiable.
      - hedora 4y ago
        What if the project recompiled them from source with some other version of javac, and prepended "My" to each class name?
        shagie 4y ago
        I am not familiar with any project that goes through such extremes to avoid vulnerability scanners. I am also not aware of any Java developers who build 3rd party libraries from source and avoid using maven (or gradle, or the odd one with ivy). Your "what if..." doesn't describe any workflow that I have ever experienced with Java in a professional (and even hobbyist) context.
- cesarb 4y ago
  > Just finding all the affected Go apps will be a nightmare This isn't even a new situation. We had the exact same problem some time ago with zlib: it was also embedded everywhere, and just finding all the affected apps was a nightmare. It's the reason most Linux distributions came to deeply dislike bundling libraries, instead of using a dynamically linked shared copy from a separate package. Unfortunately, it seems the lessons from that zlib incident are being forgotten, perhaps because too much time has passed (IIRC, in the order of decades) since then.
  - kelnos 4y ago
    > It's the reason most Linux distributions came to deeply dislike bundling libraries Is it, though? Seems like it would be trivial for a distribution maintainer to write a script to go through the dependency trees of every package, and then automatically rebuild any that depend on the vulnerable library. I don't think discovery is the problem. I suspect this is really a legacy of when disk space was not so cheap; dynamic linking means smaller binaries (usually). And, even today, many distros would probably prefer their users to only download a 2MB library update from them than have to download 500MB worth of updated programs. Bandwidth is still not as cheap as we'd hope. Edit: just realized you're probably talking about software packages vendoring in their own version of a library, not about a package statically linking with a system-provided copy of the library.
- zdragnar 4y ago
  Easy, just don't write bugs. /s That said, I think this missing the full picture- we have the same problem further down the stack, at the OS level, where old machines are running without receiving security updates on a regular basis. I don't know that there's really much of a good answer to the problem short of "don't leave your children unattended". Unlike real kids and the real world, there's plenty of bad actors looking for unattended services to stab.
- chrisseaton 4y ago
  > nearly every Go project in the world having to be patched and recompiled Is that a particular problem? What's the difficulty with recompiling?
  - 0xbadcafebee 4y ago
    Imagine you are a random person in charge of a computer. You hear there's this big vuln affecting Go programs. 1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go", pray that's enough) to identify them. 2. They don't list a website, so I have to google the name of every program and find their website. 3. How do I tell which ones are affected by the vuln? I can hope their websites tell me, but maybe they're no longer maintained, or they haven't updated their READMEs. 4. If they don't provide a patched version (most probably won't), I might have to upgrade, which may break things (probably will), assuming they even have an upgradeable version that isn't affected. 5. If I can't upgrade and have to patch, am I even a software developer and know how to do that? 6. If I luckily happen to be one, what system compiled this app 8 years ago, with what dependencies, with what version of Go, etc? Can I recreate that system and learn their build tools to finally patch & recompile? (I am not a Go developer and it has taken me days to figure out how to get different Go programs to compile, for reasons I don't understand) 7. Once I apply the patch how do I test it? (Log4j had multiple iterations of patches/mitigations, some didn't work) If I can't find the source code, can't upgrade it, & can't patch it, I am just stuck with a vulnerable version until I can somehow replace the program. For the programs I was able to identify as vulnerable, anyway. Log4j was "easy" compared to this, because you could just find all the jars on your system and unzip them, find a file, replace it, zip it back up. Log4j was actually a nightmare though, and Go apps will be worse.
    - chrisseaton 4y ago
      > How do I find out which programs on my system are Go programs? Well how are these programs getting onto your system? Presumably via some package manager, or an image builder? Don't those manage the update for you? If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues. But you had issues already if that's the predicament you're in. Some of this applies to Java too - obfuscation breaks the fixes you described using!
yjftsjthsd-h 4y ago
It's not a question of bundling / static builds, it's a question of dependencies and updates. If you build a static binary from maintained inputs (say, nix, Gentoo+portage, whatever) in CI and automatically distribute it, you're golden. If you completely separate a library out and... heck, make the user supply their own copy! but don't monitor it for updates, don't notify users, don't ship patches ever, then you're in a bad spot. Static/dynamic isn't the issue.
- WesolyKubeczek 4y ago
  The beauty of the unbundled approach is, though, that you don't need to rebuild anything except the vulnerable thing. It scales. Having to rebuild the whole world, not so much, even if you can do it. Maybe you have access to an infinite supply of zero-cost energy, but I don't, and neither does the world at large. When they find a critical bug in a Go module or a Rust crate, you usually end up with a litany of packages that had to be updated as well because everyone and their crab relies on that crate/module. Count in release managers (or whatever that's called at your shop) who pin all dependencies to the most exact minor version before shipping, because they have learned to do it like a mantra without actually understanding all the "whys" of it and just keep religiously applying that hammer everywhere. People keep complaining about how distributions work and how the traditional package management sucks all the time (of thousands complaining, I estimate that maybe 20 people actually know what they are talking about and thus are qualified to whine), but they somehow keep forgetting that those tools exist for a reason and that they are about to rediscover it the hard way. Chesterton's fence and all that.

Sometimes people ask me why I'm so skeptical about software delivery techniques involving bundling and static compilation. This is a thing that people sometimes ask.

0xbadcafebee 4y ago

It's funny how every Go project seems to statically compile in the same libraries. If just one of them has a serious vuln, we're talking nearly every Go project in the world having to be patched and recompiled, or upgraded with potentially breaking changes. And as we know from Log4j, that can be incredibly difficult. Just finding all the affected Go apps will be a nightmare, and patching will not be as simple as "replacing a jar in a zip file".

skybrian 4y ago
The situation for Go seems to have gotten a bit better with Go 1.18. Go binaries now include the versions of all the modules in them, and this can be listed with the "go version" command. That won't help you rebuild the binary, but at least you can find them.
- jewel 4y ago
  That's fantastic. Since Go has gone all-in on static compiling, it'd be awesome if they made it possible to update a dependency without recompiling (or even having the source code for) the rest of the program. In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.
  - er4hn 4y ago
    Wouldn't that require the ABI, or at least the API, to be the same across versions of the affected library? That feels like it may be possible with enough tooling, but still very hard to enforce.
    - 0xbadcafebee 4y ago
      The library being patched needs to use the same ABI, yes. So you can usually only patch the same version of a library, not upgrade it. Afaik, the ELF format allows you to replace sections in a binary. The code (or data) is packed as "Sections" and mapped to memory locations. You use a tool to unpack the ELF sections and replace them. The application accesses those sections of memory and uses whatever ABI the application and library are designed to interact with. As long as you patch a section with the same ABI that an app/library used before, nobody will notice. And the ELF ABI never changes. But of course, not every system uses ELFs. But all this depends on the compiler/linker/etc and how they generate sections and store them in the binary (for example, is it all compiled into one blob-section, or does it store libraries as separate sections, which might be inefficient?). I somehow doubt Go does things in such a way to allow this kind of patching. They probably generate a giant blob full of random functions and record versions in a header without the ability to extricate a whole library.
  - kllrnohj 4y ago
    That doesn't seem possible. The benefits of static linking are that unused methods, symbols, etc... can be stripped as well as cross-library optimizations like in-lining. There's not necessarily going to be library lines left at all, at least not in any way that looks anything like upstream. And it's also going to vary from binary to binary, depending on what exactly they used from the dependency. What you're describing would be more akin to shared linkage just in a single blob. That seems like it'd be a worst-of-both-worlds result.
    - skybrian 4y ago
      Instead of "update without compiling", maybe it would be better to attach an archive of the source to the binary, so you can update by compiling? Or maybe attach some suitable intermediate format?
kevincox 4y ago
I also like now "single file deployment" is considered a feature. It makes it easier for people without proper packaging and distribution tools. Of course these people without packing and distribution tools will now have a hard time finding out everywhere they have copied these binaries to.
- LaGrange 4y ago
  packageblabla-1.2.3.deb is a single file.
  - Beltalowda 4y ago
    That will only work on Debian. And that "single file" requires also the libfoo-1.5.deb file (must be >=1.5, not 1.4 and not higher than 2.0!) etc.
    - WesolyKubeczek 4y ago
      The thing about "single file deployment" is that it shouldn't have external requirements, otherwise it's not a single file deployment anymore.
      - kelnos 4y ago
        ... which is why the parent is (IMO correctly) claiming that the grandparent's assertion is incorrect that a Debian package is a "single file deployment".
        WesolyKubeczek 4y ago
        I think both grandparent you’re referring to and its parent are lost in the woods.
        LaGrange 4y ago
        Oh no, no I'm not, and the point of the joke was that a deb is as much of a single file as your compiled go thing. It's... well, it's a single file, it's a single artifact. And yes, it has dependencies! So does your go file, I'm afraid.
shagie 4y ago
I'm not as familiar with Go... I do know that we use a dependency scan that looks at the docker images and recognizes vulnerable versions of various Java libraries. With Go, can you look at a compiled binary in an image on the artifact repo and identify what libraries it used and if those libraries have vulnerabilities? (and it appears that my question was answered in a sibling comment)
- hedora 4y ago
  Java's "statically link then scan" approach doesn't really work though. (Some sibling threads explain why.) Even Linux distributions where rebuilding the world is easy (like Debian) insist on breaking dependencies out into shared libaries, since tracking ad hoc vendored dependencies is just too difficult. For example, sometimes people play dynamic loading and renaming tricks to allow multiple versions of the same Java library to coexist. That can usually be detected by vulnerability scanners, but not always.
  - shagie 4y ago
    I'm not sure what you mean by "statically link then scan" in the Java context. The fingerprint of the vulnerable class files are the same no matter where they are stored. They could be bundled together, or shoved in a fat jar, or exploded into all their classes -- but the class file always is distinct and identifiable.
    - hedora 4y ago
      What if the project recompiled them from source with some other version of javac, and prepended "My" to each class name?
      - shagie 4y ago
        I am not familiar with any project that goes through such extremes to avoid vulnerability scanners. I am also not aware of any Java developers who build 3rd party libraries from source and avoid using maven (or gradle, or the odd one with ivy). Your "what if..." doesn't describe any workflow that I have ever experienced with Java in a professional (and even hobbyist) context.
cesarb 4y ago
> Just finding all the affected Go apps will be a nightmare This isn't even a new situation. We had the exact same problem some time ago with zlib: it was also embedded everywhere, and just finding all the affected apps was a nightmare. It's the reason most Linux distributions came to deeply dislike bundling libraries, instead of using a dynamically linked shared copy from a separate package. Unfortunately, it seems the lessons from that zlib incident are being forgotten, perhaps because too much time has passed (IIRC, in the order of decades) since then.
- kelnos 4y ago
  > It's the reason most Linux distributions came to deeply dislike bundling libraries Is it, though? Seems like it would be trivial for a distribution maintainer to write a script to go through the dependency trees of every package, and then automatically rebuild any that depend on the vulnerable library. I don't think discovery is the problem. I suspect this is really a legacy of when disk space was not so cheap; dynamic linking means smaller binaries (usually). And, even today, many distros would probably prefer their users to only download a 2MB library update from them than have to download 500MB worth of updated programs. Bandwidth is still not as cheap as we'd hope. Edit: just realized you're probably talking about software packages vendoring in their own version of a library, not about a package statically linking with a system-provided copy of the library.
zdragnar 4y ago
Easy, just don't write bugs. /s That said, I think this missing the full picture- we have the same problem further down the stack, at the OS level, where old machines are running without receiving security updates on a regular basis. I don't know that there's really much of a good answer to the problem short of "don't leave your children unattended". Unlike real kids and the real world, there's plenty of bad actors looking for unattended services to stab.
chrisseaton 4y ago
> nearly every Go project in the world having to be patched and recompiled Is that a particular problem? What's the difficulty with recompiling?
- 0xbadcafebee 4y ago
  Imagine you are a random person in charge of a computer. You hear there's this big vuln affecting Go programs. 1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go", pray that's enough) to identify them. 2. They don't list a website, so I have to google the name of every program and find their website. 3. How do I tell which ones are affected by the vuln? I can hope their websites tell me, but maybe they're no longer maintained, or they haven't updated their READMEs. 4. If they don't provide a patched version (most probably won't), I might have to upgrade, which may break things (probably will), assuming they even have an upgradeable version that isn't affected. 5. If I can't upgrade and have to patch, am I even a software developer and know how to do that? 6. If I luckily happen to be one, what system compiled this app 8 years ago, with what dependencies, with what version of Go, etc? Can I recreate that system and learn their build tools to finally patch & recompile? (I am not a Go developer and it has taken me days to figure out how to get different Go programs to compile, for reasons I don't understand) 7. Once I apply the patch how do I test it? (Log4j had multiple iterations of patches/mitigations, some didn't work) If I can't find the source code, can't upgrade it, & can't patch it, I am just stuck with a vulnerable version until I can somehow replace the program. For the programs I was able to identify as vulnerable, anyway. Log4j was "easy" compared to this, because you could just find all the jars on your system and unzip them, find a file, replace it, zip it back up. Log4j was actually a nightmare though, and Go apps will be worse.
  - chrisseaton 4y ago
    > How do I find out which programs on my system are Go programs? Well how are these programs getting onto your system? Presumably via some package manager, or an image builder? Don't those manage the update for you? If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues. But you had issues already if that's the predicament you're in. Some of this applies to Java too - obfuscation breaks the fixes you described using!

yjftsjthsd-h 4y ago

It's not a question of bundling / static builds, it's a question of dependencies and updates. If you build a static binary from maintained inputs (say, nix, Gentoo+portage, whatever) in CI and automatically distribute it, you're golden. If you completely separate a library out and... heck, make the user supply their own copy! but don't monitor it for updates, don't notify users, don't ship patches ever, then you're in a bad spot. Static/dynamic isn't the issue.

WesolyKubeczek 4y ago
The beauty of the unbundled approach is, though, that you don't need to rebuild anything except the vulnerable thing. It scales. Having to rebuild the whole world, not so much, even if you can do it. Maybe you have access to an infinite supply of zero-cost energy, but I don't, and neither does the world at large. When they find a critical bug in a Go module or a Rust crate, you usually end up with a litany of packages that had to be updated as well because everyone and their crab relies on that crate/module. Count in release managers (or whatever that's called at your shop) who pin all dependencies to the most exact minor version before shipping, because they have learned to do it like a mantra without actually understanding all the "whys" of it and just keep religiously applying that hammer everywhere. People keep complaining about how distributions work and how the traditional package management sucks all the time (of thousands complaining, I estimate that maybe 20 people actually know what they are talking about and thus are qualified to whine), but they somehow keep forgetting that those tools exist for a reason and that they are about to rediscover it the hard way. Chesterton's fence and all that.