6 ms·

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vuln

This is always a balance. The moment you pin to a specific version you need to have a process in place to ensure you regularly upgrade to avoid introducing vulnerabilities in your production system. Throughout my career I have seen many cases where certain software still runs on ancient versions as the team originally maintaining it is no longer around (e.g. reorganisations or lay-offs). It is always hard to convince senior management to invest any resources to upgrade (If it works don't fix it) If I have project that has good unit test coverage, I prefer to use :latest, as this results in a gradual update over time. If something breaks due to a version discrepancy, it is a lot easier to convince management to fix this, as the breakage would only be noticed as part of a feature request, and often would only require a small amount of work.

pojzon 5y ago
Its hard to convince management that them adapting devops means they actually have to follow devops principles. Too many times Ive seen companies throwing this buzzword around without understanding how much work is actually required to have real devops.
- cube00 5y ago
  :s/devops/agile/g
yardstick 5y ago
Doesn’t the same logic apply to using :latest? If you are always pulling the latest, unvetted, code, shouldn’t you also have a vulnerability management process in place? You need vulnerability & risk management for any dependency, full stop.
- oxfordmale 5y ago
  Ideally you only use Docker official images,or their equivalent to avoid using unvetted code. It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one
  - yardstick 5y ago
    I wouldn’t describe Docker official images as more or less secure than anything else. Supply chain attacks can easily slip a vulnerability in.
  - laumars 5y ago
    > Ideally you only use Docker official images,or their equivalent to avoid using unvetted code. Docker images don’t ship every dependency in the average development project. They’re also not a security guarantee either. > It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one In general that’s the case but in practice that’s still the wrong mindset. You pin to a minor version and still get patch updates. Or run CVE checks to prove you’re dependencies have no reported vulnerabilities. And that will also mitigate the risk of you again accidentally updating to to new vulnerabilities (though you’d need to pin against patch version too if you want to be certain there)
    - oxfordmale 5y ago
      I have seen many cases where a something is pinned to a minor version, and then gets forgotten about for 2 or 3 years. In my current companies there is Production codebase that hasn't been touched for over two years. The original developers have long since moved on and no one really owns the code. That is the most common scenario leading to security breaches. This is about succession planning, with the realisation there might not be any succession if business priorities change.
      - yardstick 5y ago
        In that case, better the devil you know? But either way if you don’t have a vulnerability management plan you are at risk.
      - laumars 5y ago
        I’ve seen many cases were version pinning has caused issues at the worst possible time (like missed delivery dates because of a breaking change in dependencies being pushed by maintainers at an inconvenient time for us). Ultimately there’s no perfect answer here but given a choice I’d rather opt for a known risk than an unknown one.
bheadmaster 5y ago
In that case, it's better to just pin the major version of the container: FROM python:3 will work as long as Python3 doesn't make any backwards incompatible changes (note: Python3 occasionally deprecates then removes a feature that was part of the official API[0], so Python3 is not technically semver-correct). If, however, one wants automatic updates WITH reproducible builds, then a CI/CD pipeline that automatically updates the FROM line in a Dockerfile on every upstream release is an only solution. [0] https://docs.python.org/3.10/whatsnew/3.10.html#removed https://docs.python.org/3.10/whatsnew/3.10.html#removed
- thejosh 5y ago
  this works great until a library decides it only likes python 3.8 or 3.9 or 3.10 and won't work with 3.10.
- Kwpolska 5y ago
  Python never claimed to be semver-compliant. Also, if you’re using 3rd-party packages that require compiling stuff (eg. numpy), pre-built binary packages (aka wheels) might not be available on PyPI in the first few days (or weeks, or sometimes even months) after Python 3.x.0 is released, so you don’t want the version to be randomly changed under you. Moreover, I pulled python:3 a year ago, so I get 3.9. FROM python:3.10 should generally be fine though.
- oefrha 5y ago
  This is very bad advice. Lots of nontrivial things aren't automatically compatible with the latest Python minor version. PyTorch for instance doesn't support Python 3.10 and it's been five months since the stable release (ten months since the first beta). For anything nontrivial, you almost always want to specify a minor Python version.
  - deleted 5y ago
    [deleted]
  - paskozdilar 5y ago
    Why is this bad advice? The PyTorch scenario is an exception, not the rule, and the users of PyTorch should know not to use unsupported versions. > For anything nontrivial, you almost always want to specify a minor Python version. Depends on how you define "trivial".
ReactiveJelly 5y ago
It's better to only change one thing at a time, not to get your dependencies updated as a surprise when you're trying to update something else. Besides, if the dependencies aren't pinned in version control, how can you answer a query like "On this date last year, what version were we running?"
- oxfordmale 5y ago
  I am using AWS ECS as a Docker repo with releases using the image hash (as well as a server). If you want to rollback to a certain time, you just identify the appropriate Docker image tag, and restore the Docker you used at that time in your development environment. To give an example, we got stuck on Python Pandas 0.24.2 for two years when the Pandas 1.0 upgrade happened. At the time we pinned the version as one job was failing. With COVID priorities naturally changed and this was forgotten. When normality returned we suddenly had to upgrade 10 jobs to Pandas 1.2.x, a very painful migration that took one Engineer 4 weeks of testing. In hindsight it would have been far better to upgrade this one job, as it would have only taken a few hours. Again it is a trade off. Having unexpected dependency changes can be cumbersome when you are releasing a small change. However, getting senior management to agree of weeks of upgrade testing can be challenging as well. My natural preference is not to pin versions, unless I absolutely have to.
KronisLV 5y ago
I'd say that there are two things that you should have in place regardless of which approach you take: - having tests (or even manually test the system with scenarios, if for whatever reason you cannot automate) in place to catch things breaking, before shipping any changes - having security scanning in place, be it for containers, your dependencies, or anything else; ideally for everything Then, things should get a bit more clear: - you should be able to spot the publicly known vulnerabilities and adequately evaluate their impact to decide what must be done - when you update versions of your dependencies, you should then also be able to see whether anything breaks before shipping Admittedly, all of that will only be viable if you have the buy-in from the people involved, since otherwise you'll get "blamed" for things breaking as a result of your initiative for avoiding shipping vulnerable software, or you'll find it difficult to justify to people why you're spending so much time updating dependencies and doing refactoring. Not only that, but some systems out there are not really easily testable, e.g. those that have a large amount of functionality within the DB so your tests might as well end up mocking either too much of the system or the wrong parts of the system (i'm yet to see anyone mocking the low level queries that go to the DB and back, e.g. setting and validating the individual parameters within query abstractions, as opposed to just the data that's returned) - in many cases it simply won't be viable to test everything. Another thing is that in practice semver tends to lie to us and even minor updates or patches can sometimes have breaking changes within them, something that i wrote about in my blog article "Never update anything": https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything (albeit there i also touch upon the fact that software should largely be more stable and have fewer breaking updates in the first place, or even less new features in "stable" branches) I guess my argument is that there's a lot of complexity to be tackled here and that people should invest more time and effort into handling updates, refactoring and even testing, than they do now: i've seen teams where people all agree that tests are important, yet nobody wants to write any because if they tried, they'd have to mock large parts of the system OR try to do integration tests and handle the fact that nobody has invested the time and effort into bringing up reproducible environments and services, e.g. a new automatically migrated DB instance for the tests OR the fact that they'll need to do tests in a shared DB instance and clean up afterwards. Environments like that are just a downwards spiral that's bound to produce brittle software. So what's my practical advice? Use pinned versions, preferably starting with the latest and most boring one that you can get away with (e.g. LTS). Have a process in place for figuring out when you need to update, set aside time for doing just that (even if manually), assign someone to do this and someone else to make sure it's been done. Have testing and serious validations be a part of this process, to make sure that there's no breakage that'd slip past - full regression testing, all your unit tests, integration tests, feature tests etc. How often should you do it? Depends on the importance of the system - i've seen it done quarterly, i've seen it done monthly, some with plenty of resources out there could probably do it weekly or more often. Additionally, be able to do this ASAP when critical vulnerabilities become apparent, e.g. log4shell. Of course, it's the ultimate example of useful but exceedingly boring work that people don't really want to do, so i've no doubts about running into unmaintained software in the future.
peakaboo 5y ago
I've also seen the same. As soon as people start locking versions, that code is no longer updated and nobody will change it, because it's extra work to do so. I personally think running latest is the best thing to do. And if something fails, you downgrade it temporarily until the latest work again. It's pretty much opposite to what is recommended, and it's just the best solution in my opinion.
- bspammer 5y ago
  My company locks versions, but dependabot is configured on all of our repos. It automatically creates PRs to bump versions, and if CI passes for minor/patch bumps they get automatically merged. This takes a lot of the hassle out of the problem. For major bumps, a manual approval is required, but they happen infrequently enough that it's not a lot of work.
  - ipsocannibal 5y ago
    A similar tool to dependabot written by Salesforce: https://github.com/salesforce/dockerfile-image-update https://github.com/salesforce/dockerfile-image-update
speed_spread 5y ago
Covert maintenance is the name of the game. If the site goes down because of a security flaw you'll be blamed anyway. Might as well get fired for doing the right thing. I'm only half joking... The right thing to do with dummy management is to get everything in writing to make it clear that if anything happens it'll be their fault.
EdwardDiego 5y ago
I've used Maven version ranges to great effect... ...only projects that actually observe semver. (Jackson, take a bow for doing so with your patch releases. I wouldn't use it for your 'minor' releases though.)
- cuteboy19 5y ago
  Maven versions are not present in VCS if you use ranges. Wouldn't this cause issues like not know what Library version is being used in production and builds not being reproducible?
  - isbvhodnvemrwvn 5y ago
    Not to mention that Maven doesn't have any mechanism for locking in transitive versions, because typically everyone uses hardcoded patch versions and never update them.
    - EdwardDiego 5y ago
      How do you mean? You can override versions of transitive dependencies.
      - isbvhodnvemrwvn 5y ago
        In most projects JVM devs don't pin the versions of transitive dependencies, even though in many other languages it's recognized as a basic functionality of the build tool.
        EdwardDiego 5y ago
        Yeah, that's a fair point about JVM culture. I suggest that there's a few cultural reasons this occurs, based on my experience: 1) JVM libs generally tend to follow semver to the Y level of X.Y.Z. reasonably well. (Spring can FOADIAF in this regard though) 2) Maven and Gradle etc. have very good dependency resolution algorithms 3) When the dependency algorithm struggles, you can provide guidance (e.g., Maven's dependencyManagement element, or the ability to exclude a dependency's dependency so that your pinned version of that dependency is what's used) On that last point, you can use the Enforcer plugin to be very very draconian about transitive dependencies. I worked on a project in the past that would fail the build if you introduced a dependency that had dependencies, and didn't precisely specify the version of each new dependency at the project level. I've done a fair bit of work also with Python, Go, and JS, and doing so made me realise that the dependency management situation in JVM land is the best I've ever encountered (so far). Which surprised me, I had always thought it somewhat over-engineered, but now I think it's sufficiently engineered and those other languages need to steal some ideas. (Especially Python, I love Poetry, especially compared to pipenv, but I would kill for the ability to override a dependency's dependency like I can in Maven and Gradle).
jenscow 5y ago
Right. Let's assume you're going to need to upgrade sooner or later - the longer you leave it, the harder it gets. Best to upgrade and discover dependency issues during dev/staging, than try to fix many months worth of them when there's an active CVE on your live pinned version.

pojzon 5y ago

Its hard to convince management that them adapting devops means they actually have to follow devops principles. Too many times Ive seen companies throwing this buzzword around without understanding how much work is actually required to have real devops.

cube00 5y ago
:s/devops/agile/g

yardstick 5y ago

Doesn’t the same logic apply to using :latest? If you are always pulling the latest, unvetted, code, shouldn’t you also have a vulnerability management process in place? You need vulnerability & risk management for any dependency, full stop.

oxfordmale 5y ago
Ideally you only use Docker official images,or their equivalent to avoid using unvetted code. It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one
- yardstick 5y ago
  I wouldn’t describe Docker official images as more or less secure than anything else. Supply chain attacks can easily slip a vulnerability in.
- laumars 5y ago
  > Ideally you only use Docker official images,or their equivalent to avoid using unvetted code. Docker images don’t ship every dependency in the average development project. They’re also not a security guarantee either. > It is always a trade off, however it is far more likely that a hacker will use a ten year old well exploited CVE, rather than a recent one In general that’s the case but in practice that’s still the wrong mindset. You pin to a minor version and still get patch updates. Or run CVE checks to prove you’re dependencies have no reported vulnerabilities. And that will also mitigate the risk of you again accidentally updating to to new vulnerabilities (though you’d need to pin against patch version too if you want to be certain there)
  - oxfordmale 5y ago
    I have seen many cases where a something is pinned to a minor version, and then gets forgotten about for 2 or 3 years. In my current companies there is Production codebase that hasn't been touched for over two years. The original developers have long since moved on and no one really owns the code. That is the most common scenario leading to security breaches. This is about succession planning, with the realisation there might not be any succession if business priorities change.
    - yardstick 5y ago
      In that case, better the devil you know? But either way if you don’t have a vulnerability management plan you are at risk.
    - laumars 5y ago
      I’ve seen many cases were version pinning has caused issues at the worst possible time (like missed delivery dates because of a breaking change in dependencies being pushed by maintainers at an inconvenient time for us). Ultimately there’s no perfect answer here but given a choice I’d rather opt for a known risk than an unknown one.

bheadmaster 5y ago

In that case, it's better to just pin the major version of the container: FROM python:3 will work as long as Python3 doesn't make any backwards incompatible changes (note: Python3 occasionally deprecates then removes a feature that was part of the official API[0], so Python3 is not technically semver-correct). If, however, one wants automatic updates WITH reproducible builds, then a CI/CD pipeline that automatically updates the FROM line in a Dockerfile on every upstream release is an only solution. [0] https://docs.python.org/3.10/whatsnew/3.10.html#removed https://docs.python.org/3.10/whatsnew/3.10.html#removed

thejosh 5y ago
this works great until a library decides it only likes python 3.8 or 3.9 or 3.10 and won't work with 3.10.
Kwpolska 5y ago
Python never claimed to be semver-compliant. Also, if you’re using 3rd-party packages that require compiling stuff (eg. numpy), pre-built binary packages (aka wheels) might not be available on PyPI in the first few days (or weeks, or sometimes even months) after Python 3.x.0 is released, so you don’t want the version to be randomly changed under you. Moreover, I pulled python:3 a year ago, so I get 3.9. FROM python:3.10 should generally be fine though.
oefrha 5y ago
This is very bad advice. Lots of nontrivial things aren't automatically compatible with the latest Python minor version. PyTorch for instance doesn't support Python 3.10 and it's been five months since the stable release (ten months since the first beta). For anything nontrivial, you almost always want to specify a minor Python version.
- deleted 5y ago
  [deleted]
- paskozdilar 5y ago
  Why is this bad advice? The PyTorch scenario is an exception, not the rule, and the users of PyTorch should know not to use unsupported versions. > For anything nontrivial, you almost always want to specify a minor Python version. Depends on how you define "trivial".

ReactiveJelly 5y ago

It's better to only change one thing at a time, not to get your dependencies updated as a surprise when you're trying to update something else. Besides, if the dependencies aren't pinned in version control, how can you answer a query like "On this date last year, what version were we running?"

oxfordmale 5y ago
I am using AWS ECS as a Docker repo with releases using the image hash (as well as a server). If you want to rollback to a certain time, you just identify the appropriate Docker image tag, and restore the Docker you used at that time in your development environment. To give an example, we got stuck on Python Pandas 0.24.2 for two years when the Pandas 1.0 upgrade happened. At the time we pinned the version as one job was failing. With COVID priorities naturally changed and this was forgotten. When normality returned we suddenly had to upgrade 10 jobs to Pandas 1.2.x, a very painful migration that took one Engineer 4 weeks of testing. In hindsight it would have been far better to upgrade this one job, as it would have only taken a few hours. Again it is a trade off. Having unexpected dependency changes can be cumbersome when you are releasing a small change. However, getting senior management to agree of weeks of upgrade testing can be challenging as well. My natural preference is not to pin versions, unless I absolutely have to.

KronisLV 5y ago

I'd say that there are two things that you should have in place regardless of which approach you take: - having tests (or even manually test the system with scenarios, if for whatever reason you cannot automate) in place to catch things breaking, before shipping any changes - having security scanning in place, be it for containers, your dependencies, or anything else; ideally for everything Then, things should get a bit more clear: - you should be able to spot the publicly known vulnerabilities and adequately evaluate their impact to decide what must be done - when you update versions of your dependencies, you should then also be able to see whether anything breaks before shipping Admittedly, all of that will only be viable if you have the buy-in from the people involved, since otherwise you'll get "blamed" for things breaking as a result of your initiative for avoiding shipping vulnerable software, or you'll find it difficult to justify to people why you're spending so much time updating dependencies and doing refactoring. Not only that, but some systems out there are not really easily testable, e.g. those that have a large amount of functionality within the DB so your tests might as well end up mocking either too much of the system or the wrong parts of the system (i'm yet to see anyone mocking the low level queries that go to the DB and back, e.g. setting and validating the individual parameters within query abstractions, as opposed to just the data that's returned) - in many cases it simply won't be viable to test everything. Another thing is that in practice semver tends to lie to us and even minor updates or patches can sometimes have breaking changes within them, something that i wrote about in my blog article "Never update anything": https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything (albeit there i also touch upon the fact that software should largely be more stable and have fewer breaking updates in the first place, or even less new features in "stable" branches) I guess my argument is that there's a lot of complexity to be tackled here and that people should invest more time and effort into handling updates, refactoring and even testing, than they do now: i've seen teams where people all agree that tests are important, yet nobody wants to write any because if they tried, they'd have to mock large parts of the system OR try to do integration tests and handle the fact that nobody has invested the time and effort into bringing up reproducible environments and services, e.g. a new automatically migrated DB instance for the tests OR the fact that they'll need to do tests in a shared DB instance and clean up afterwards. Environments like that are just a downwards spiral that's bound to produce brittle software. So what's my practical advice? Use pinned versions, preferably starting with the latest and most boring one that you can get away with (e.g. LTS). Have a process in place for figuring out when you need to update, set aside time for doing just that (even if manually), assign someone to do this and someone else to make sure it's been done. Have testing and serious validations be a part of this process, to make sure that there's no breakage that'd slip past - full regression testing, all your unit tests, integration tests, feature tests etc. How often should you do it? Depends on the importance of the system - i've seen it done quarterly, i've seen it done monthly, some with plenty of resources out there could probably do it weekly or more often. Additionally, be able to do this ASAP when critical vulnerabilities become apparent, e.g. log4shell. Of course, it's the ultimate example of useful but exceedingly boring work that people don't really want to do, so i've no doubts about running into unmaintained software in the future.

peakaboo 5y ago

I've also seen the same. As soon as people start locking versions, that code is no longer updated and nobody will change it, because it's extra work to do so. I personally think running latest is the best thing to do. And if something fails, you downgrade it temporarily until the latest work again. It's pretty much opposite to what is recommended, and it's just the best solution in my opinion.

bspammer 5y ago
My company locks versions, but dependabot is configured on all of our repos. It automatically creates PRs to bump versions, and if CI passes for minor/patch bumps they get automatically merged. This takes a lot of the hassle out of the problem. For major bumps, a manual approval is required, but they happen infrequently enough that it's not a lot of work.
- ipsocannibal 5y ago
  A similar tool to dependabot written by Salesforce: https://github.com/salesforce/dockerfile-image-update https://github.com/salesforce/dockerfile-image-update

speed_spread 5y ago

Covert maintenance is the name of the game. If the site goes down because of a security flaw you'll be blamed anyway. Might as well get fired for doing the right thing. I'm only half joking... The right thing to do with dummy management is to get everything in writing to make it clear that if anything happens it'll be their fault.

EdwardDiego 5y ago

I've used Maven version ranges to great effect... ...only projects that actually observe semver. (Jackson, take a bow for doing so with your patch releases. I wouldn't use it for your 'minor' releases though.)

cuteboy19 5y ago
Maven versions are not present in VCS if you use ranges. Wouldn't this cause issues like not know what Library version is being used in production and builds not being reproducible?
- isbvhodnvemrwvn 5y ago
  Not to mention that Maven doesn't have any mechanism for locking in transitive versions, because typically everyone uses hardcoded patch versions and never update them.
  - EdwardDiego 5y ago
    How do you mean? You can override versions of transitive dependencies.
    - isbvhodnvemrwvn 5y ago
      In most projects JVM devs don't pin the versions of transitive dependencies, even though in many other languages it's recognized as a basic functionality of the build tool.
      - EdwardDiego 5y ago
        Yeah, that's a fair point about JVM culture. I suggest that there's a few cultural reasons this occurs, based on my experience: 1) JVM libs generally tend to follow semver to the Y level of X.Y.Z. reasonably well. (Spring can FOADIAF in this regard though) 2) Maven and Gradle etc. have very good dependency resolution algorithms 3) When the dependency algorithm struggles, you can provide guidance (e.g., Maven's dependencyManagement element, or the ability to exclude a dependency's dependency so that your pinned version of that dependency is what's used) On that last point, you can use the Enforcer plugin to be very very draconian about transitive dependencies. I worked on a project in the past that would fail the build if you introduced a dependency that had dependencies, and didn't precisely specify the version of each new dependency at the project level. I've done a fair bit of work also with Python, Go, and JS, and doing so made me realise that the dependency management situation in JVM land is the best I've ever encountered (so far). Which surprised me, I had always thought it somewhat over-engineered, but now I think it's sufficiently engineered and those other languages need to steal some ideas. (Especially Python, I love Poetry, especially compared to pipenv, but I would kill for the ability to override a dependency's dependency like I can in Maven and Gradle).

jenscow 5y ago

Right. Let's assume you're going to need to upgrade sooner or later - the longer you leave it, the harder it gets. Best to upgrade and discover dependency issues during dev/staging, than try to fix many months worth of them when there's an active CVE on your live pinned version.