7 ms·

> but often forget that they need to add an OS update step. Is this standard practice? I've never done this but it makes a lot of sense. Is this basically just

> but often forget that they need to add an OS update step. Is this standard practice? I've never done this but it makes a lot of sense. Is this basically just adding `RUN apt-get update && apt-get -y upgrade` at the top of the Dockerfile? (Assuming a Debian image.)

dsr_ 5y ago
Technically, this is correct. In practice, it will eventually bite you. Someday some package that you depend on will have a security issue that causes a necessary config change that has a default you don't like. Also, note that you have just changed to a non-reproducible container, where running it on Monday is not guaranteed to be the same as running it next Friday.
- mrweasel 5y ago
  > Also, note that you have just changed to a non-reproducible container Precisely, and that may be what stops many from just updating their images. I think ideally we'd have centralised updates, perhaps just one a week, and then use tag for that week. You could do all this in-house, but it's a lot of work for small teams. Redundant work.
  - dsr_ 5y ago
    In the end, what gets deployed is the responsibility of the entity doing the deployment. Maybe the organization wants to deploy the latest trunk on every commit. There are probably some situations where that is reasonable. Those situations probably shouldn't involve people's money, privacy or safety. Setting up a two-stage local repository isn't very hard. The intake side gets updates from upstream, and the deployment side gets updates from the intake side when the packages have been reviewed and hopefully tested. Do this for everything with an external upstream -- Ruby gems, Python modules, JS libraries, whatever -- and you have insulated yourself against supply-chain attacks. As a bonus, if your leftpad function just goes missing upstream, you still have a full copy in your deployment repository, and will until you decide it's time to implement your own.
    - mrweasel 5y ago
      > In the end, what gets deployed is the responsibility of the entity doing the deployment. Yes and no, I don't disagree, but remember classic hosting, semi-managed infrastructure still exists. We often get container images delivered and are responsible for the operational side of things, but we have no control of what is actually inside the containers. Sure we can make requests, or inform a customer that we believe what they are doing isn't safe. Ideally we could reject a request to run a container, but in reality that's not really an option. That shifts the responsibility of security more in the direction of the developers and they often do not have regular patch management as a priority.
      - dsr_ 5y ago
        If your organization isn't in control of the software you are running, you are a service provider and you have to treat them as potentially hostile external entities. Your responsibility is for the infrastructure, and you must explicitly disclaim responsibility for the functionality and security of anything inside the containers. This is, incidentally, the antithesis of "devops".
  - jrochkind1 5y ago
    > You could do all this in-house, but it's a lot of work for small teams. Redundant work. What's the alternative?
    - WJW 5y ago
      Right now the alternative seems to be "ignore the problem". :)
      - jrochkind1 5y ago
        Interesting, thank you! I still don't use containers (working on very small teams), and as I've tried to get into it, this is one of the questions I had where I was sure I was missing something and couldn't figure out what everyone else did... this is making me feel a lot more competent! (Although questioning the reasonableness of our industry!).
- sneak 5y ago
  Containers aren't reproducible anyway. Skipping apt upgrade will not give you the same result on a rebuild. Builds on debian/ubuntu aren't deterministic. Additionally, if you don't "apt update", some of the packages you try to install from mirrors will 404. Unless you're using something with deterministic builds, reproducibility is a myth anyway. Update your OS and test the image and save the artifact somewhere with "docker save | zstd".
  - lbgdn 5y ago
    Just curious, can you go into a bit more detail on why / how "builds on debian/ubuntu are not deterministic"?
    - sneak 5y ago
      For one, the packages you install on Monday may not be available on the mirrors on Tuesday. The mirrors prune old packages not in the indices. You have to do an “apt update” to ensure that the packages you install will be fetchable, because the apt indices inside the image are out of date and “apt-get install -y whatever” may fail with a 404. That means you aren’t guaranteed to get the same version installed from an “apt-get install -y whatever” after Monday’s “apt-get update” as you would after Tuesday’s “apt-get update”, as the current version of “whatever” may have changed in the interim, even if you don’t run an “apt upgrade”. In any case, a lot of the files on disk are generated dynamically at install time for certain types of things, and include things derived from the state of the system (which frequently depends on remote network resources, as described above), so issuing the exact same dockerfile FROM+RUN+RUN+ADD etc lines will not result in an identical image result when run on different days: it’s nondeterministic. A deterministic build is one where the same build always results in a byte-for-byte identical build artifact. There is effort being put in to make the building of the backing .debs deterministic, but AFAIK no Debian or Debian-like is trying to make apt itself work in a deterministic manner when installing packages. There are still postinstall scripts, for example, that are system-state dependent. Really, you need to be saving your build artifacts when you do Docker builds. Saving the Dockerfile and expecting to be able to rebuild the image at any time later isn’t a good bet. You might, sometimes, be able to rebuild a mostly-compatible image, but there is no chance whatsoever you will be able to build a byte-for-byte compatible image, and it’s entirely possible that your build might just fail entirely (e.g. if you are pinning specific package versions that fall out of date and are no longer fetchable from the mirrors). Then, in the worst case scenario, you can always load your original working/saved image back in, replace/patch/modify specific files on disk to address issues (either with vendor tools or manually) via a new build that pulls FROM the saved artifact image, and make a derived one.
- jrochkind1 5y ago
  So... what is the conventional/best practice way of dealing with this? I ways always confused about this, thinking I didn't understand what people were doing here... but it turns out maybe there is no good solution and most people are ignoring it? Would you expect this to result in more attention at some point, after it results in more exploits?
  - squiggleblaz 5y ago
    > So... what is the conventional/best practice way of dealing with this? I don't know. I have been able to tolerate "apt-get dist-upgrade" on my legacy non-container system: upgrading from one Ubuntu LTS to the next is hard, but within an LTS we've never encountered a problem over 15 years (I think? more than a decade at least) this has been done. A newer project uses NixOS, but it is so new it has no users. NixOS allows me to mention the particular commit of nixpkgs - that means the versions of the packages in the package manager - into a lockfile. This means I can run daily updates, run my tests against them, and deploy them. And because the lockfile is checked in, if things suddenly start going wrong on the umpteenth of Octember, I can see what changes happened on that day: was it the file I commited? no? oh I see, libtwiddle was upgraded. Yes, it was libtwiddle that broke things. As I say, this is a brand new project. It may not work as perfectly as all that. And swallowing Nix requires a certain amount of koolaid: it is user friendly but it's very picky about who its friends are. > Would you expect this to result in more attention at some point, after it results in more exploits? I think that we will switch to paid platforms that offer managed runtimes. These will probably offer targeted, well communicated updates of dependencies (everyone will hear Microsoft Python is releasing ms-py-http-3: here's what you need to know), and have fewer libraries that offer more code. In a sense these will be switch back to distributions. We will say "why would you manage your own dependency?" And the cycle will repeat with a new flavor.
  - michael1999 5y ago
    Best practice would be a pipeline producing an up-to-date bare image with your major dependencies tested and a CVE check. Then use that base layer in your deployed system. Conventional would be to ignore it.
- pnutjam 5y ago
  If you control your update source, which is standard practice for old school linux admins; this is not true. I pull new updates on a set schedule and only push the ones that are tested in nonprod. There are any number of tools that do this, Uyani is a good free one.
Spivak 5y ago
The far more maintainable way to do this rather than every dev team at a company doing ad-hoc patching is to have your ops team maintain your base images that they're responsible patching and then every team derives from that. This gives you the ability to move the pieces independently from one another so when you release it's actually (sw_version, platform_version) and lets you track down bugs caused by platform updates easier.
- squiggleblaz 5y ago
  Any solution that works is going to have to work for a small team of seven developers at a company that has twenty employees, most of whom assume they're tech experts because they tweet a lot. The bigger companies run the most code, but it's the smaller companies that are must vulnerable and most in need of solutions.
stonemetal12 5y ago
Not sure what is standard practice but what seems to work for me is building it up in layers. Start with Alpine:latest install OS level packages tag that as baseimage:N. Use that image as the base of the development level Dockerfile. Then on a schedule build base image N+1, and run the tests. That way if something in the base changes and causes a regression you can pin to the last known good until you can fix the bug.

dsr_ 5y ago

Technically, this is correct. In practice, it will eventually bite you. Someday some package that you depend on will have a security issue that causes a necessary config change that has a default you don't like. Also, note that you have just changed to a non-reproducible container, where running it on Monday is not guaranteed to be the same as running it next Friday.

mrweasel 5y ago
> Also, note that you have just changed to a non-reproducible container Precisely, and that may be what stops many from just updating their images. I think ideally we'd have centralised updates, perhaps just one a week, and then use tag for that week. You could do all this in-house, but it's a lot of work for small teams. Redundant work.
- dsr_ 5y ago
  In the end, what gets deployed is the responsibility of the entity doing the deployment. Maybe the organization wants to deploy the latest trunk on every commit. There are probably some situations where that is reasonable. Those situations probably shouldn't involve people's money, privacy or safety. Setting up a two-stage local repository isn't very hard. The intake side gets updates from upstream, and the deployment side gets updates from the intake side when the packages have been reviewed and hopefully tested. Do this for everything with an external upstream -- Ruby gems, Python modules, JS libraries, whatever -- and you have insulated yourself against supply-chain attacks. As a bonus, if your leftpad function just goes missing upstream, you still have a full copy in your deployment repository, and will until you decide it's time to implement your own.
  - mrweasel 5y ago
    > In the end, what gets deployed is the responsibility of the entity doing the deployment. Yes and no, I don't disagree, but remember classic hosting, semi-managed infrastructure still exists. We often get container images delivered and are responsible for the operational side of things, but we have no control of what is actually inside the containers. Sure we can make requests, or inform a customer that we believe what they are doing isn't safe. Ideally we could reject a request to run a container, but in reality that's not really an option. That shifts the responsibility of security more in the direction of the developers and they often do not have regular patch management as a priority.
    - dsr_ 5y ago
      If your organization isn't in control of the software you are running, you are a service provider and you have to treat them as potentially hostile external entities. Your responsibility is for the infrastructure, and you must explicitly disclaim responsibility for the functionality and security of anything inside the containers. This is, incidentally, the antithesis of "devops".
- jrochkind1 5y ago
  > You could do all this in-house, but it's a lot of work for small teams. Redundant work. What's the alternative?
  - WJW 5y ago
    Right now the alternative seems to be "ignore the problem". :)
    - jrochkind1 5y ago
      Interesting, thank you! I still don't use containers (working on very small teams), and as I've tried to get into it, this is one of the questions I had where I was sure I was missing something and couldn't figure out what everyone else did... this is making me feel a lot more competent! (Although questioning the reasonableness of our industry!).
sneak 5y ago
Containers aren't reproducible anyway. Skipping apt upgrade will not give you the same result on a rebuild. Builds on debian/ubuntu aren't deterministic. Additionally, if you don't "apt update", some of the packages you try to install from mirrors will 404. Unless you're using something with deterministic builds, reproducibility is a myth anyway. Update your OS and test the image and save the artifact somewhere with "docker save | zstd".
- lbgdn 5y ago
  Just curious, can you go into a bit more detail on why / how "builds on debian/ubuntu are not deterministic"?
  - sneak 5y ago
    For one, the packages you install on Monday may not be available on the mirrors on Tuesday. The mirrors prune old packages not in the indices. You have to do an “apt update” to ensure that the packages you install will be fetchable, because the apt indices inside the image are out of date and “apt-get install -y whatever” may fail with a 404. That means you aren’t guaranteed to get the same version installed from an “apt-get install -y whatever” after Monday’s “apt-get update” as you would after Tuesday’s “apt-get update”, as the current version of “whatever” may have changed in the interim, even if you don’t run an “apt upgrade”. In any case, a lot of the files on disk are generated dynamically at install time for certain types of things, and include things derived from the state of the system (which frequently depends on remote network resources, as described above), so issuing the exact same dockerfile FROM+RUN+RUN+ADD etc lines will not result in an identical image result when run on different days: it’s nondeterministic. A deterministic build is one where the same build always results in a byte-for-byte identical build artifact. There is effort being put in to make the building of the backing .debs deterministic, but AFAIK no Debian or Debian-like is trying to make apt itself work in a deterministic manner when installing packages. There are still postinstall scripts, for example, that are system-state dependent. Really, you need to be saving your build artifacts when you do Docker builds. Saving the Dockerfile and expecting to be able to rebuild the image at any time later isn’t a good bet. You might, sometimes, be able to rebuild a mostly-compatible image, but there is no chance whatsoever you will be able to build a byte-for-byte compatible image, and it’s entirely possible that your build might just fail entirely (e.g. if you are pinning specific package versions that fall out of date and are no longer fetchable from the mirrors). Then, in the worst case scenario, you can always load your original working/saved image back in, replace/patch/modify specific files on disk to address issues (either with vendor tools or manually) via a new build that pulls FROM the saved artifact image, and make a derived one.
jrochkind1 5y ago
So... what is the conventional/best practice way of dealing with this? I ways always confused about this, thinking I didn't understand what people were doing here... but it turns out maybe there is no good solution and most people are ignoring it? Would you expect this to result in more attention at some point, after it results in more exploits?
- squiggleblaz 5y ago
  > So... what is the conventional/best practice way of dealing with this? I don't know. I have been able to tolerate "apt-get dist-upgrade" on my legacy non-container system: upgrading from one Ubuntu LTS to the next is hard, but within an LTS we've never encountered a problem over 15 years (I think? more than a decade at least) this has been done. A newer project uses NixOS, but it is so new it has no users. NixOS allows me to mention the particular commit of nixpkgs - that means the versions of the packages in the package manager - into a lockfile. This means I can run daily updates, run my tests against them, and deploy them. And because the lockfile is checked in, if things suddenly start going wrong on the umpteenth of Octember, I can see what changes happened on that day: was it the file I commited? no? oh I see, libtwiddle was upgraded. Yes, it was libtwiddle that broke things. As I say, this is a brand new project. It may not work as perfectly as all that. And swallowing Nix requires a certain amount of koolaid: it is user friendly but it's very picky about who its friends are. > Would you expect this to result in more attention at some point, after it results in more exploits? I think that we will switch to paid platforms that offer managed runtimes. These will probably offer targeted, well communicated updates of dependencies (everyone will hear Microsoft Python is releasing ms-py-http-3: here's what you need to know), and have fewer libraries that offer more code. In a sense these will be switch back to distributions. We will say "why would you manage your own dependency?" And the cycle will repeat with a new flavor.
- michael1999 5y ago
  Best practice would be a pipeline producing an up-to-date bare image with your major dependencies tested and a CVE check. Then use that base layer in your deployed system. Conventional would be to ignore it.
pnutjam 5y ago
If you control your update source, which is standard practice for old school linux admins; this is not true. I pull new updates on a set schedule and only push the ones that are tested in nonprod. There are any number of tools that do this, Uyani is a good free one.

Spivak 5y ago

The far more maintainable way to do this rather than every dev team at a company doing ad-hoc patching is to have your ops team maintain your base images that they're responsible patching and then every team derives from that. This gives you the ability to move the pieces independently from one another so when you release it's actually (sw_version, platform_version) and lets you track down bugs caused by platform updates easier.

squiggleblaz 5y ago
Any solution that works is going to have to work for a small team of seven developers at a company that has twenty employees, most of whom assume they're tech experts because they tweet a lot. The bigger companies run the most code, but it's the smaller companies that are must vulnerable and most in need of solutions.

stonemetal12 5y ago

Not sure what is standard practice but what seems to work for me is building it up in layers. Start with Alpine:latest install OS level packages tag that as baseimage:N. Use that image as the base of the development level Dockerfile. Then on a schedule build base image N+1, and run the tests. That way if something in the base changes and causes a regression you can pin to the last known good until you can fix the bug.