7 ms·

No shit, a mechanism to bake pinned shared libs into a static archive to counter version dependencies on Ubuntu/Debian vs RedHat Linuxen grows stale, fossilized

No shit, a mechanism to bake pinned shared libs into a static archive to counter version dependencies on Ubuntu/Debian vs RedHat Linuxen grows stale, fossilized images? Who would've thought! As much as I love Docker for development, going all-in on containers for prod was a huge mistake. Which is becoming clearer now that RedHat pulls free CentOS images, after taking the so-called Linux community further and further away from POSIX and other portability means with systemd, namespaces, oci, and whatnot.

oblio 6y ago
Docker images should be treated the same way as you'd treat any library you use or any package you create. Do you think Java .jars in Maven repos or .NET .dlls in Nuget are faring any better? Find solid dependencies, don't use too many of them, make sure you update them, use static analysis tools and dependency scanners, etc. Same drill as for anything.
- Frost1x 6y ago
  >Find solid dependencies, don't use too many of them, make sure you update them, use static analysis tools and dependency scanners, etc. Same drill as for anything. That's hilarious simply because most heavy Docker users I've interacted with basically use it as a way to build houses of cards which glue-sticked piles and piles of dependencies together rapidly. This is largely because application complexity in some domains and expectations of development pace have ballooned to insanity but there are plenty of heavy Docker users that do this even when these sort of pressures don't exist. I think as software progresses and more development techniques become easier to leverage and automated (correctly and incorrectly), we're getting to a point advanced statistical software packages have been at for quite some time: no matter what data you throw at the software for analysis and what analysis you choose, the software has so much experience baked into it that you'll get some result that makes assumptions what you're doing is valid that may look like reasonable outputs even if they aren't. If you do need to do some preprocessing, much of that is even automated so it's incredibly easy to do something very wrong if you dont know what you're doing but not realize it. Automation has made it much easier to fail late instead of fail early.
  - rhencke 6y ago
    This was all achingly true well before Docker, though. (That's why Docker got popular, after all!) Docker isn't the disease, its use in such manner is just a symptom.
  - evilduck 6y ago
    The trouble is finding balance between simple DIY implementations or cobbling together a house of cards. "Allow CRUD of names" ok, you'll have it next sprint. "Allow upload of their resumé docs" no problem, two more weeks. "Parse the docx and PDF files users upload and extract the contents and fill our db schema with relevant data". Well shit. The specs to understand either of those formats are gigantic and now I'm forced to cobble together something using more dependencies if you want it done in less than two year's time.
    - incrudible 6y ago
      Is it really that troublesome to strike a balance between implementing something yourself and using a library when the difference is two weeks versus two years? If, for example, a file format is really complex, it has most likely one implementation that almost everyone uses. If there is a problem with it, you will hear about it. Use that. On the other hand, if a file format is really simple or you only need a simple subset of it, rolling your own could avoid you getting 0-dayed when the most popular implementation has a vulnerability.
      - ComodoHacker 6y ago
        >you getting 0-dayed when the most popular implementation has a vulnerability vs getting hit when your vulnerable implementation becomes popular. Not an obvious choice indeed.
        incrudible 6y ago
        > vs getting hit when your vulnerable implementation becomes popular. Which is not going to happen, because there's no point in publishing something that is tailored to your specific purpose.
- pjmlp 6y ago
  They definitely are, because each jar/dll doesn't have the whole world inside itself. Docker is like shipping EAR + Websphere + OS in one package.
  - oblio 6y ago
    Yeah, I know, but you'd use the same mechanism for both. For jars/dlls you look up dependency scanners and hook them up to your CI/CD. For Docker images you look up image scanners and hook them up to your CI/CD.
  - thu2111 6y ago
    Realistically a lot of Java apps are shipped as fat JARs that do in fact include the (Java) world. Hot-patching apps to use newer versions of JARs because of security bugs, outside of the update stream of the actual app itself, is not really a good idea due to the risk of regressions. Docker has basically codified the failure of dynamic linking in the real world.
- marcosdumay 6y ago
  Hum... Docker automatically adds an entire OS as static dependency to anything. How does that goes with "not using too many dependencies" rule? Also, good luck doing static analysis on binary tools. There's an entire area of work people call "ops" that specializes on managing that dependency. But they can only do that if it's dynamic and loosely coupled.
  - oblio 6y ago
    Ummm... You make your own images, if you really care. You involve your "ops" people in that. They manage the base images that are then used by your dev team. They're responsible for updating the base images. Your dev team provides a CI/CD pipeline which the ops team can use to check that security updates don't break functionality. It's not rocket science.
  - cle 6y ago
    Docker doesn’t add anything as a dependency, you do. Use “FROM scratch” if you want to run a process with nothing but the Linux kernel. You won’t even get libc.
feinte 6y ago
The problem lies in baking entire distribution into an image. Using distroless images or "FROM scratch" with statically compiled app reduces the risks. You still have to watch for your app dependencies updates but that's less work than for an entire distribution.
- mattgreenrocks 6y ago
  Huh, I didn't know that was possible. So you could specify an app container that is then overlaid atop the OS container? I suppose installing the dependencies ends up being easier if you know they'll end up in, e.g. a ubuntu 18.04 image, hence that's what people do.
dralley 6y ago
I've never heard anything about RH pulling free CentOS images, it looks like they're still there to me? https://hub.docker.com/_/centos https://hub.docker.com/_/centos They also have free RHEL container images. https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image https://www.redhat.com/en/blog/introducing-red-hat-universal...
- walty8 6y ago
  I believe it's discussed here: https://news.ycombinator.com/item?id=25400249 https://news.ycombinator.com/item?id=25400249
  - dralley 6y ago
    I don't interpret "pulling free CentOS images" as being the same thing as "pulling investment in CentOS" in a thread specifically about docker images. Existing docker images are still there, future images will continue being pushed (but only for CentOS Stream post-2021), and UBI exists for minor releases of RHEL which is supposed to be identical to CentOS anyways. And considering the fact that Docker containers don't use the kernel provided by the client OS, the difference between CentOS and CentOS Stream is even smaller than it already was. The most significant changes between minor releases of RHEL / CentOS are in the kernel.
alisonkisk 6y ago
What does prod have to do with stale Docker images? The problem is not updating your images. Running old libs on bare metal has exactly the same problem.
- syshum 6y ago
  Exactly, I know a few places that have years old installs of linux that never get updates... same thing. nothing in docker prevents updates, just people in general are poor about doing maintenance, this is a story that is old as computers, and why MS forced updates on everyone because left to their own most people choose to "postpone" them indefinitely for fear of breaking things
  - Wowfunhappy 6y ago
    > everyone because left to their own most people choose to "postpone" them indefinitely for fear of breaking things It sure would be helpful if the updates didn't actually break things. It's usually not the security fixes that do that—stuff rarely breaks on LTSC, or in Debian point releases—it's all the other changes that get rolled in. There's just too much churn across the industry. Even if the changes are good, too little thought is given to the cost of change itself.
- tannhaeuser 6y ago
  > The problem is not updating your images. Running old libs on bare metal has exactly the same problem. Updating libs on bare metal has maintainers and distributions, though. And doesn't force you to roll your own security infra within containers.
  - acdha 6y ago
    You have all of that inside the container, too: apt-get upgrade doesn’t magically stop working because it’s in a Dockerfile. The difference is that you have a trivial way to confirm deployments and be confident that there isn’t other mutable state preventing an upgrade.
mfer 6y ago
I think it was Kelsey Hightower who said that Kubernetes is a platform for building platforms. Unfortunately, we've gotten very caught up with the boring infra parts (the containers and the orchestration of k8s). We too often forget about the other platform we should build that uses these tools. People don't complain about mass security holes in Heroku. That's long been container based. That's an example of building a platform that enables keeping on top of security issues. This isn't a problem of the tools we have. It's a problem because of how we use them and what we build with them.
- dimitrios1 6y ago
  > People don't complain about mass security holes in Heroku. That's because Heroku is mostly a closed platform that the average consumer does not have insight into. You don't know what outdated dep the software that executes your dynos runs. As the case in many areas of life, ignorance is bliss.
  - mfer 6y ago
    I just used Heroku as an example to show the architecture. FWIW, it looks like Heroku is using Kubernetes these days. It's a platform built on a platform. If you want open platforms we can talk about cloud foundry. If a base image or buildpack has a vulnerability there is a process to update it and rebuild the containers for the running application without and app dev ever needing to know about or deal with it. This is considered a feature. The idea is to look at the architecture, concerns, and how to handle different roles with different needs at different layers of the concerns. I don't mean to hold up Heroku or Cloud Foundry as some amazing thing. Just that you can have an architecture that solves for these problems with containers.
    - ethbr0 6y ago
      Docker's security architecture problems definitely feel like a "Closed: Working as intended" / not prioritized, rather than something they see as a core feature. And why not? They're shipping benefits (immediate productivity) and externalizing costs (eventual security breaches) onto the end user.
sam0x17 6y ago
It has it's uses though. For example I used to work at a startup where this super finicky NodeJS script was written by a junior dev that ended up being the main deployment script for client build-outs on one of their products. Fast forward a few years, the old CEO contacts me because he still has a few clients using the platform and has no way of running the script. He pays me to try to get it to run on his machine -- I get it working on Ubuntu but will not run on his Mac because the Mac is no longer supported by whatever weird packages the junior dev was using. Long story short, Docker to the rescue -- now he has a baked-in image that will run that script forever.
- ethbr0 6y ago
  This is how we end up with software archaeology as a profession. :D
  - diveanon 6y ago
    * cryptoarchaeology
- FpUser 6y ago
  For this particular case VM might have more chances for long term survival.
  - diveanon 6y ago
    True, but I've taught non tech staff to use docker desktop to run qa and demo environment.
  - sam0x17 6y ago
    I had worries about him updating it and/or changing the disk state to the point where it broke and not knowing how to use snapshots.
oxfordmale 6y ago
I only remember too well the pain of having to migrate from a legacy VM to a new one, having to figuring out all the software that was installed over the years. Docket is no magical bullet, but much better than managing VM. In our company we regularly force rebuilds of Dockets to ensure we pick up the latest changes
- shuntress 6y ago
  Using docker just forces you to solve the problem instead of continuing to put it off. You don't need docker to automate your fresh install process.
  - oxfordmale 6y ago
    There are indeed other ways to manage the fresh install process. Not sure they result in less critical vulnerabilities overall. It is all too easy to never update ancient versions of installations, especially as server maintenance is often handled by a separate team.
sedachv 6y ago
> As much as I love Docker for development, going all-in on containers for prod was a huge mistake. Containers are a great deployment mechanism for production. Unfortunately a lot of people seem to think that they are a substitute for package management. This was obviously (and I do mean very clearly obvious in advance from first principles¹, no hindsight needed) going to become a disaster. Most people just do not care - they are either completely unaware of the problem, or they know that in a few years' time they will either have changed jobs or be promoted (in no small part thanks to their "Docker thought leadership") and it will no longer be their problem (and they would still get to claim "Docker thought leadership" success on their résumés). ¹ Also from experience - VMware came out with "virtual appliances" in the mid-2000s that promised to do a lot of what Docker would later be used for.

oblio 6y ago

Docker images should be treated the same way as you'd treat any library you use or any package you create. Do you think Java .jars in Maven repos or .NET .dlls in Nuget are faring any better? Find solid dependencies, don't use too many of them, make sure you update them, use static analysis tools and dependency scanners, etc. Same drill as for anything.

Frost1x 6y ago
>Find solid dependencies, don't use too many of them, make sure you update them, use static analysis tools and dependency scanners, etc. Same drill as for anything. That's hilarious simply because most heavy Docker users I've interacted with basically use it as a way to build houses of cards which glue-sticked piles and piles of dependencies together rapidly. This is largely because application complexity in some domains and expectations of development pace have ballooned to insanity but there are plenty of heavy Docker users that do this even when these sort of pressures don't exist. I think as software progresses and more development techniques become easier to leverage and automated (correctly and incorrectly), we're getting to a point advanced statistical software packages have been at for quite some time: no matter what data you throw at the software for analysis and what analysis you choose, the software has so much experience baked into it that you'll get some result that makes assumptions what you're doing is valid that may look like reasonable outputs even if they aren't. If you do need to do some preprocessing, much of that is even automated so it's incredibly easy to do something very wrong if you dont know what you're doing but not realize it. Automation has made it much easier to fail late instead of fail early.
- rhencke 6y ago
  This was all achingly true well before Docker, though. (That's why Docker got popular, after all!) Docker isn't the disease, its use in such manner is just a symptom.
- evilduck 6y ago
  The trouble is finding balance between simple DIY implementations or cobbling together a house of cards. "Allow CRUD of names" ok, you'll have it next sprint. "Allow upload of their resumé docs" no problem, two more weeks. "Parse the docx and PDF files users upload and extract the contents and fill our db schema with relevant data". Well shit. The specs to understand either of those formats are gigantic and now I'm forced to cobble together something using more dependencies if you want it done in less than two year's time.
  - incrudible 6y ago
    Is it really that troublesome to strike a balance between implementing something yourself and using a library when the difference is two weeks versus two years? If, for example, a file format is really complex, it has most likely one implementation that almost everyone uses. If there is a problem with it, you will hear about it. Use that. On the other hand, if a file format is really simple or you only need a simple subset of it, rolling your own could avoid you getting 0-dayed when the most popular implementation has a vulnerability.
    - ComodoHacker 6y ago
      >you getting 0-dayed when the most popular implementation has a vulnerability vs getting hit when your vulnerable implementation becomes popular. Not an obvious choice indeed.
      - incrudible 6y ago
        > vs getting hit when your vulnerable implementation becomes popular. Which is not going to happen, because there's no point in publishing something that is tailored to your specific purpose.
pjmlp 6y ago
They definitely are, because each jar/dll doesn't have the whole world inside itself. Docker is like shipping EAR + Websphere + OS in one package.
- oblio 6y ago
  Yeah, I know, but you'd use the same mechanism for both. For jars/dlls you look up dependency scanners and hook them up to your CI/CD. For Docker images you look up image scanners and hook them up to your CI/CD.
- thu2111 6y ago
  Realistically a lot of Java apps are shipped as fat JARs that do in fact include the (Java) world. Hot-patching apps to use newer versions of JARs because of security bugs, outside of the update stream of the actual app itself, is not really a good idea due to the risk of regressions. Docker has basically codified the failure of dynamic linking in the real world.
marcosdumay 6y ago
Hum... Docker automatically adds an entire OS as static dependency to anything. How does that goes with "not using too many dependencies" rule? Also, good luck doing static analysis on binary tools. There's an entire area of work people call "ops" that specializes on managing that dependency. But they can only do that if it's dynamic and loosely coupled.
- oblio 6y ago
  Ummm... You make your own images, if you really care. You involve your "ops" people in that. They manage the base images that are then used by your dev team. They're responsible for updating the base images. Your dev team provides a CI/CD pipeline which the ops team can use to check that security updates don't break functionality. It's not rocket science.
- cle 6y ago
  Docker doesn’t add anything as a dependency, you do. Use “FROM scratch” if you want to run a process with nothing but the Linux kernel. You won’t even get libc.

feinte 6y ago

The problem lies in baking entire distribution into an image. Using distroless images or "FROM scratch" with statically compiled app reduces the risks. You still have to watch for your app dependencies updates but that's less work than for an entire distribution.

mattgreenrocks 6y ago
Huh, I didn't know that was possible. So you could specify an app container that is then overlaid atop the OS container? I suppose installing the dependencies ends up being easier if you know they'll end up in, e.g. a ubuntu 18.04 image, hence that's what people do.

dralley 6y ago

I've never heard anything about RH pulling free CentOS images, it looks like they're still there to me? https://hub.docker.com/_/centos https://hub.docker.com/_/centos They also have free RHEL container images. https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image https://www.redhat.com/en/blog/introducing-red-hat-universal...

walty8 6y ago
I believe it's discussed here: https://news.ycombinator.com/item?id=25400249 https://news.ycombinator.com/item?id=25400249
- dralley 6y ago
  I don't interpret "pulling free CentOS images" as being the same thing as "pulling investment in CentOS" in a thread specifically about docker images. Existing docker images are still there, future images will continue being pushed (but only for CentOS Stream post-2021), and UBI exists for minor releases of RHEL which is supposed to be identical to CentOS anyways. And considering the fact that Docker containers don't use the kernel provided by the client OS, the difference between CentOS and CentOS Stream is even smaller than it already was. The most significant changes between minor releases of RHEL / CentOS are in the kernel.

alisonkisk 6y ago

What does prod have to do with stale Docker images? The problem is not updating your images. Running old libs on bare metal has exactly the same problem.

syshum 6y ago
Exactly, I know a few places that have years old installs of linux that never get updates... same thing. nothing in docker prevents updates, just people in general are poor about doing maintenance, this is a story that is old as computers, and why MS forced updates on everyone because left to their own most people choose to "postpone" them indefinitely for fear of breaking things
- Wowfunhappy 6y ago
  > everyone because left to their own most people choose to "postpone" them indefinitely for fear of breaking things It sure would be helpful if the updates didn't actually break things. It's usually not the security fixes that do that—stuff rarely breaks on LTSC, or in Debian point releases—it's all the other changes that get rolled in. There's just too much churn across the industry. Even if the changes are good, too little thought is given to the cost of change itself.
tannhaeuser 6y ago
> The problem is not updating your images. Running old libs on bare metal has exactly the same problem. Updating libs on bare metal has maintainers and distributions, though. And doesn't force you to roll your own security infra within containers.
- acdha 6y ago
  You have all of that inside the container, too: apt-get upgrade doesn’t magically stop working because it’s in a Dockerfile. The difference is that you have a trivial way to confirm deployments and be confident that there isn’t other mutable state preventing an upgrade.

mfer 6y ago

I think it was Kelsey Hightower who said that Kubernetes is a platform for building platforms. Unfortunately, we've gotten very caught up with the boring infra parts (the containers and the orchestration of k8s). We too often forget about the other platform we should build that uses these tools. People don't complain about mass security holes in Heroku. That's long been container based. That's an example of building a platform that enables keeping on top of security issues. This isn't a problem of the tools we have. It's a problem because of how we use them and what we build with them.

dimitrios1 6y ago
> People don't complain about mass security holes in Heroku. That's because Heroku is mostly a closed platform that the average consumer does not have insight into. You don't know what outdated dep the software that executes your dynos runs. As the case in many areas of life, ignorance is bliss.
- mfer 6y ago
  I just used Heroku as an example to show the architecture. FWIW, it looks like Heroku is using Kubernetes these days. It's a platform built on a platform. If you want open platforms we can talk about cloud foundry. If a base image or buildpack has a vulnerability there is a process to update it and rebuild the containers for the running application without and app dev ever needing to know about or deal with it. This is considered a feature. The idea is to look at the architecture, concerns, and how to handle different roles with different needs at different layers of the concerns. I don't mean to hold up Heroku or Cloud Foundry as some amazing thing. Just that you can have an architecture that solves for these problems with containers.
  - ethbr0 6y ago
    Docker's security architecture problems definitely feel like a "Closed: Working as intended" / not prioritized, rather than something they see as a core feature. And why not? They're shipping benefits (immediate productivity) and externalizing costs (eventual security breaches) onto the end user.

sam0x17 6y ago

It has it's uses though. For example I used to work at a startup where this super finicky NodeJS script was written by a junior dev that ended up being the main deployment script for client build-outs on one of their products. Fast forward a few years, the old CEO contacts me because he still has a few clients using the platform and has no way of running the script. He pays me to try to get it to run on his machine -- I get it working on Ubuntu but will not run on his Mac because the Mac is no longer supported by whatever weird packages the junior dev was using. Long story short, Docker to the rescue -- now he has a baked-in image that will run that script forever.

ethbr0 6y ago
This is how we end up with software archaeology as a profession. :D
- diveanon 6y ago
  * cryptoarchaeology
FpUser 6y ago
For this particular case VM might have more chances for long term survival.
- diveanon 6y ago
  True, but I've taught non tech staff to use docker desktop to run qa and demo environment.
- sam0x17 6y ago
  I had worries about him updating it and/or changing the disk state to the point where it broke and not knowing how to use snapshots.

oxfordmale 6y ago

I only remember too well the pain of having to migrate from a legacy VM to a new one, having to figuring out all the software that was installed over the years. Docket is no magical bullet, but much better than managing VM. In our company we regularly force rebuilds of Dockets to ensure we pick up the latest changes

shuntress 6y ago
Using docker just forces you to solve the problem instead of continuing to put it off. You don't need docker to automate your fresh install process.
- oxfordmale 6y ago
  There are indeed other ways to manage the fresh install process. Not sure they result in less critical vulnerabilities overall. It is all too easy to never update ancient versions of installations, especially as server maintenance is often handled by a separate team.

sedachv 6y ago

> As much as I love Docker for development, going all-in on containers for prod was a huge mistake. Containers are a great deployment mechanism for production. Unfortunately a lot of people seem to think that they are a substitute for package management. This was obviously (and I do mean very clearly obvious in advance from first principles¹, no hindsight needed) going to become a disaster. Most people just do not care - they are either completely unaware of the problem, or they know that in a few years' time they will either have changed jobs or be promoted (in no small part thanks to their "Docker thought leadership") and it will no longer be their problem (and they would still get to claim "Docker thought leadership" success on their résumés). ¹ Also from experience - VMware came out with "virtual appliances" in the mid-2000s that promised to do a lot of what Docker would later be used for.