11 ms·

We should all be using dependency cooldowns

• elevation 1y ago
  You could do a lot of this with CI if you scheduled a job to fetch the most recent packages once a month and record a manifest with the current versions, then, if no security issues are reported before the end of the cooldown period, run integration tests against the new manifest. If no tests fail, automatically merge this update into the project. For projects with hundreds or thousands of active dependencies, the feed of security issues would be a real fire hose. You’d want to use an LLM to filter the security lists for relevance before bringing them to the attention of a developer. It would be more efficient to centralize this capability as a service so that 5000 companies aren’t all paying for an LLM to analyze the same security reports. Perhaps it would be enough for someone to run a service like cooldown.pypi.org that served only the most vetted packages to everyone.
  • AndrewDucker 1y ago
    Why would you use an LLM rather than just sort by CVE score?
    • ok123456 1y ago
      CVE scores are junk. Most CVEs are noise.
• jayd16 1y ago
  Doesn't this mean you're leaving yourself open to known vulnerabilities during that "cool down" time?
  • jcalvinowens 1y ago
    Yep. Not only vulnerabilities, but just bugs in general, which usually matter more than than vulnerabilities IMHO.
    • bityard 1y ago
      Do you believe new releases don't introduce new bugs?
      • jcalvinowens 1y ago
        Obviously. Every release introduces bugs. There's an inevitable positive correlation between the amount of code we write and the number of bugs we introduce, we just try to minimize it. The probability of introducing bugs is a function of the amount of development being done. Releasing less often doesn't change that. In fact, under that assumption, delaying releases strictly increases the amount of time users are affected by the average bug. People who do this tell themselves the extra time allows them to catch more bugs. But in my experience that's a bedtime story, most bugs aren't noticed until after deployment anyway. That's completely orthogonal to slowly rolling out changes, btw.
    • woodruffw 1y ago
      To be clear, there's no reason why you can't update dependencies in advance of a cooldown period. The cooldown is an enforced policy that you can choose to override as needed. (This also doesn't apply to vulnerabilities per se, since known vulnerabilities typically aren't evaluated against cooldowns by tools like Dependabot.)
      • jcalvinowens 1y ago
        No you can't, the cooldown period is started by the new upstream release. So if you follow this "rule" you're guaranteed to be behind the latest upstream release.
        • woodruffw 1y ago
          I don't understand what you mean. The cooldown period is something you decide to enforce; you can always override it. It's your prerogative as a responsible engineer to decide the boundaries of policy enforcement.
  • swatcoder 1y ago
    No. A sane "cooldown" is just for automated version updates relying on semantic versioning rules, which is a pretty questionable practice in the first place, but is indeed made a lot more safe this way. You can still manually update your dependency versions when you learn that your code is exposed to some vulnerability that's purportedly been fixed. It's no different than manually updating your dependency version when you learn that there's some implementation bug or performance cliff that was fixed. You might even still use an automated system to identify these kinds of "critical" updates and bring them to your attention, so that you can review them and can appropriately assume accountability for the choice to incorporate them early, bypassing the cooldown, if you believe that's the right thing to do. Putting in that effort, having the expertise to do so, and assuming that accountability is kind of your "job" as a developer or maintainer. You can't just automate and delegate everything if you want people to be able to trust what you share with them.
    • jayd16 1y ago
      If you could understand the quality of updates you're pulling in, that solves the issue entirely. The point is that you can't. There's no reason to pretend we live in a world where everyone is manually combing through the source of every dependency update.
    • zahlman 1y ago
      > A sane "cooldown" is just for automated version updates relying on semantic versioning rules, which is a pretty questionable practice in the first place, but is indeed made a lot more safe this way. I'm reminded of how new Setuptools versions are able to cause problems for so many people, basically because install tools default to setting up isolated build environments using the latest version of whatever compatible build backend is specified (which in turn defaults to "any version of Setuptools"). qv. my LWN article https://lwn.net/Articles/1020576/ https://lwn.net/Articles/1020576/ .
  • astrobe_ 1y ago
    TFA shows that most vulnerabilities have a "window of opportunity" smaller than one day. Are you anxious going on week-end because Friday evening a zero-day or a major bug could be made public?
    • jayd16 1y ago
      Well then you agree that the answer is yes. At the end of the article a 14 day window is mentioned but not dismissed and does not mention the downsides.
• Havoc 1y ago
  > we should all Except if everyone does it chance of malicious things being spotted in source also drops by virtue of less eyeballs Still helps though in cases where maintainer spot it etc
  • deleted 1y ago
    [deleted]
  • smaudet 1y ago
    > also drops by virtue of less eyeballs I don't think the people automatically updating and getting hit with the supply chain attack are also scanning the code, I don't think this will impact them much. If instead, updates are explicitly put on cooldowns, with the option of manually updating sooner, then there would be more eyeballs, not fewer, as people are more likely to investigate patch notes, etc., possibly even test in isolation...
  • woodruffw 1y ago
    (Author of the post.) The underlying premise here is that supply chain security vendors are honest in their claims about proactively scanning (and effectively detecting + reporting) malicious and compromised packages. In other words, it's not about eyeballs (I don't think people who automatically apply Dependabot bumps are categorically reading the code anyways), but about rigorous scanning and reporting.
    • mkoubaa 1y ago
      Maybe it's more about mileage (attacks that actually happen) than eyeballs.
  • tjpnz 1y ago
    You might read the source if something breaks but in a successful supply chain attack that's unlikely to happen. You push to production, go home for the evening and maybe get pinged about it by some automation in a few weeks.
  • pico303 1y ago
    I agree. This "cooldown" approach seems antithetical to some basic tenants of security in the open source world, namely that more eyeballs makes for better code. If we all stop looking at or using the thing, are these security professionals really going to find the supply-chain problems for us in the thing, for free? Instead of a period where you don't use the new version, shouldn't we instead be promoting a best practice of not just blindly using a package or library in production? This "cooldown" should be a period of use in dev or QA environments while we take the time to investigate the libraries we use and their dependencies. I know this can be difficult in many languages and package managers, given the plethora of libraries and dependencies (I'm looking at you in particular JavaScript). But "it's hard" shouldn't really be a good excuse for our best efforts to maintain secure and stable applications.
• jcalvinowens 1y ago
  I hate this. Delaying real bugfixes to achieve some nebulous poorly defined security benefit is just bad engineering.
  • wrs 1y ago
    Do you upgrade all your dependencies every day? If not, then there’s no real difference in upgrading as if it were 7 days ago.
    • jerlam 1y ago
      Your CI/CD might be setup to upgrade all your dependencies on every build.
      • wrs 1y ago
        I’ve seen a lot of CI/CD setups and I’ve never seen that. If that were common practice, it would certainly simplify the package manager, since there would be no need for lockfiles!
        • jerlam 1y ago
          I didn't necessarily say they were good CI/CD practices.
        • chuckadams 1y ago
          I do see some CI running without lockfiles, and there's still a contingent that believes that libraries should never commit their lockfiles. It's a reasonably good idea to _test_ a configuration without the lockfile, since any user of your dependency is using _their_ lockfile that their local solver came up with, not yours, but this ought to be something you'd do alongside the tests using the lockfile. So locking down the CI environment is a good idea for that and many other reasons. Realistically, no one does full side-by-side tests with and without lockfiles, but it's a good idea to at least do a smoke test or two that way.
    • ktpsns 1y ago
• Will-Reppeto 1y ago
  [dead]
• 33a 1y ago
  A lot of security problems can be solved by moving slower.
  • solarist 1y ago
    All of security problems can be solved by not moving at all.
    • kykat 1y ago
      What about food security
      • solarist 1y ago
        If you don’t eat food you won’t get food poisoning.
      • ksherlock 1y ago
        The slow food movement encourages eating local foods and gardening, among other things, so it actually improves food security, for people who aren't food insecure.
• theoldgreybeard 1y ago
  jokes on them I already have 10 year dependency cooldowns on the app I work on at work!
  • cheschire 1y ago
    Retirement Driven Development
  • nitwit005 1y ago
    This did make me think of our app running on Java 8. Although, I suppose we've probably updated the patch version.
    • marcosdumay 1y ago
      There's always the one that requires Java 8, and the one that requires Java >= 11.
    • deleted 1y ago
      [deleted]
• layer8 1y ago
  People in this thread are worried that they are significantly vulnerable if they don't update right away. However, this is mostly not an issue in practice. A lot of software doesn't have continuous deployment, but instead has customer-side deployment of new releases, which follow a slower rhythm of several weeks or months, barring emergencies. They are fine. Most vulnerabilities that aren't supply-chain attacks are only exploitable under special circumstances anyway. The thing to do is to monitor your dependencies and their published vulnerabilities, and for critical vulnerabilities to assess whether your product is affect by it. Only then do you need to update that specific dependency right away.
  • silvestrov 1y ago
    I think the main question is: do your app get unknown input (i.e. controlled by other people). Browsers get a lot of unknown input, so they have to update often. A Weather app is likely to only get input from one specific site (controlled by the app developers), so it should be relatively safe.
  • duped 1y ago
    A million times this. You update a dependency when there are bug fixes or features that you need (and this includes patching vulnerabilities!). Those situations are rare. Otherwise you're just introducing risk into your system - and not that you're going to be caught in some dragnet supply chain attack, but that some dependency broke something you relied on by accident. Dependencies are good. Churn is bad.
    • latentsea 1y ago
      I'd rather have excellent quality test suites, and pipelines I can have a high degree of confidence in to catch these issues and mitigate the risk, than dependencies that basically never get updated and then become very difficult to do so, which just introduce different classes of risk.
    • jbkgujklgui 1y ago
      Vulnerabilities are not rare, the two most popular programming languages, javascript (node) and python, have 1000+ CVEs in their official docker images. I.e. in practice useless and shouldn't be used by anyone for anything.
• cosmic_cheese 1y ago
  I think there’s a much stronger argument for policies that both limit the number and complexity of dependencies. Don’t add it unless it’s highly focused (no “everything libraries” that pull in entire universes of their own) and carries a high level of value. A project’s entire dependency tree should be small and clean. Libraries themselves should perhaps also take a page from the book of Linux distributions and offer LTS (long term support) releases that are feature frozen and include only security patches, which are much easier to reason about and periodically audit.
  • pengaru 1y ago
    I'd be willing to pay $100 to upvote your comment 100x.
    • deleted 1y ago
      [deleted]
    • Y_Y 1y ago
      How do you think dang puts bread on the table?
  • tcfhgj 1y ago
    Won't using highly focused dependencies increase the amount of dependencies? Limiting the number of dependencies, but then rewriting them in your own code, will also increase the maintenance burden and compile times
    • skydhash 1y ago
      A lot of projects are using dependencies, but are only using a small part of. Or are using them in a single place for a single usecase. Like bringing in formik (npm), but you only have one single form. Or moment, because you want to format a single date.
  • buu700 1y ago
    I think AI nudges the economics more in this direction as well. Adding a non-core dependency has historically bought short-term velocity in exchange for different long-term maintenance costs. With AI, there are now many more cases where a first-party implementation becomes cheaper/easier/faster in both the short term and the long term. Of course it's up to developers to weigh the tradeoffs and make reasonable choices, but now we have a lot more optionality. Reaching for a dependency no longer needs to be the default choice of a developer on a tight timeline/budget.
• gr4vityWall 1y ago
  The Debian stable model of having a distro handle common dependencies with a full system upgrade every few years looks more and more sane as years pass. It's a shame some ecosystems move waaay too fast, or don't have a good story for having distro-specific packages. For example, I don't think there are Node.js libraries packaged for Debian that allow you to install them from apt and use it in projects. I might be wrong.
  • kykat 1y ago
    It is possible to work with rust, using debian repositories as the only source.
  • embedding-shape 1y ago
    > For example, I don't think there are Node.js libraries packaged for Debian that allow you to install them from apt and use it in projects Web search shows some: https://packages.debian.org/search?keywords=node&searchon=names&suite=trixie&section=all https://packages.debian.org/search?keywords=node&searchon=na... (but also shows "for optimizing reasons some results might have been suppressed" so might not be all) Although probably different from other distros, Arch for example seems to have none.
    • o11c 1y ago
      Locally, you can do: apt-cache showpkg 'node-*' | grep ^Package: which returns 4155 results, though 727 of them are type packages. Using these in commonjs code is trivial; they are automatically found by `require`. Unfortunately, system-installed packages are yet another casualty of the ESM transition ... there are ways to make it work but it's not automatic like it used to be.
      • 9dev 1y ago
        > Unfortunately, system-installed packages are yet another casualty of the ESM transition ... A small price to pay for the abundant benefits ESM brings.
• l9o 1y ago
  I wonder if LLMs can help here to some extent. I agree with others that cooldowns aren't helpful if everyone is doing it. I've been working on automatic updates for some of my [very overengineered] homelab infra and one thing that I've found particularly helpful is to generate PRs with reasonable summaries of the updates with an LLM. it basically works by having a script that spews out diffs of any locks that were updated in my repository, while also computing things like `nix store diff-closures` for the before/after derivations. once I have those diffs, I feed them into claude code in my CI job, which generates a pull request with a nicely formatted output. one thing I've been thinking is to lookup all of those dependencies that were upgraded and have the LLM review the commits. often claude already seems to lookup some of the commits itself and be able to give a high level summary of the changes, but only for small dependencies where the commit hash and repository were in the lock file. it would likely not help at all with the xz utils backdoor, as IIRC the backdoor wasn't even in the git repo, but on the release tarballs. but I wonder if anyone is exploring this yet?
  • robszumski 1y ago
    We built and launched this product about 2 months ago, HN thread here: https://news.ycombinator.com/item?id=45439721 https://news.ycombinator.com/item?id=45439721 Totally agree that AI is great for this, it will work harder and go deeper and never gets tired of reading code or release notes or migration guides. What you want instead of summaries is to find the breaking changes, figure out if they matter, then comment on _that_.
• perlgeek 1y ago
  Some scattered thoughts on that: * If everybody does it, it won't work so well * I've seen cases where folks pinned their dependencies, and then used "npm install" instead of "npm ci", so the pinning was worthless. Guess they are the accidental, free beta testers for the rest of us. * In some ecosystems, distributions (such as Debian) does both additional QA, and also apply a cooldown. Now we try to retrofit some of that into our package managers.
  • exasperaited 1y ago
    > * If everybody does it, it won't work so well Indeed, this is a complex problem to solve. And the "it won't work so well" of this is probably a general chilling effect on trying to fix things because people won't roll them out fast enough anyway. This may seem theoretical but for example in websites where there are suppliers and customers, there's quite a chilling effect on any mechanism that encourages people to wait until a supplier has positive feedback; there are fewer and fewer people with low enough stakes who are willing to be early adopters in that situation. What this means is that new suppliers often drop out too quickly, abandon platforms, work around those measures in a way that reduces the value of trust, and worse still there's a risk of bad reviews because of the reviewer's Dunning-Kruger etc. I think the mechanism is important for people who really must use it, but there will absolutely be side effects that are hard to qualify/correct.
  • layer8 1y ago
    "If everybody does it, <some outcome>" is rarely a good argument, because the premise rarely becomes reality.
    • dugidugout 1y ago
      If you're prescribing a public practice you intend to be taken seriously, you should contend with what happens at scale, especially when evaluating competing alternatives, where comparative scale effects become decisive. Given the article's push for ecosystem-wide support, there's good reason to probe this hypothetical.
    • rmunn 1y ago
• cxr 1y ago
  What everyone should all be doing is practicing the decades-old discipline of source control. Attacks of the form described in the post, where a known-good, uncompromised dependency is compromised at the "supply chain" level, can be 100% mitigated—not fractionally or probabilistically—by cutting out the vulnerable supply chain. The fact that people are still dragging their feet on this and resist basic source control is the only reason why this class of attack is even possible. That vendoring has so many other benefits and solves other problems is even more reason to do so. Stacking up more sub-par tooling is not going to solve anything. Fortunately this is a problem that doesn't even have to exist, and isn't one that anyone falls into naturally. It's a problem that you have to actively opt into by taking steps like adding things to .gitignore to exclude them from source control, downloading and using third-party tools in a way that introduces this and other problems, et cetera—which means you can avoid all of it by simply not taking those extra steps. (Fun fact: on a touch-based QWERTY keyboard, the gesture to input "vendoring" by swiping overlaps with the gesture for "benefitting".)
  • dbdr 1y ago
    Doesn't vendoring solve the supply chain issue in the same way as picking a dependency version and never upgrading would? (especially if your package manager includes a hash of the dependency in a lock file)
    • cxr 1y ago
      Mm, no. The practice I described is neither "never upgrading" your dependencies nor predicated upon that. And your comment ignores that there's an entire other class of problem I alluded to (reproducibility/failure-to-reproduce) that normal, basic source control ("vendoring") sidesteps almost entirely, if not entirely in the majority of cases. That, too, is without addressing the fact that what you're tacitly treating as bedrock is not, in fact, bedrock (even if it can be considered the status quo within some subculture). The problem of "supply chain attacks" (which, to reiterate, is coded language for "not exercising source control"—or, "pretending to, but not actually doing it") is a problem that people have to opt in to. It takes extra tooling. It takes extra steps. You have to actively do things to get yourself into the situation where you're susceptible to it—and then it takes more tooling and more effort to try and get out. In contrast, you can protect yourself from it +and+ solve the reproducibility problem all by simply doing nothing[1]—not taking the extra steps to edit .gitignore to exclude third-party code (and, sometimes, first-party code) from the SCM history, not employing lock files to undo the consequences of selectively excluding key parts of the input to the build process (and your app's behavior). All these lock file schemes that language package manager projects have belatedly and sheepishly began to offer as a solution are inept attempts at doing what your base-level RCS already does for free (if only all the user guides for these package managers weren't all instructing people to subvert source control to begin with). Every package manager lock file format (requirements file, etc.) is an inferior, ad hoc, formally-specified, error-prone, incompatible re-implementation of half of Git. 1. <https://news.ycombinator.com/item?id=25623388 https://news.ycombinator.com/item?id=25623388>
• callamdelaney 1y ago
  Anything to save us from not being able to apply because of a CVE which is only relevant if you're doing something which we don't.
• pimlottc 1y ago
  This assumes that most exploits are discovered by pro-active third-party security vendors, instead of being noticed in deployed projects. Is this actually true?
  • woodruffw 1y ago
    > Is this actually true? I don't know, but it's the claimed truth from a lot of vendors! The value proposition for a lot of supply chain security products is a lot weaker if their proactive detection isn't as strong as claimed.
• nicoburns 1y ago
  I would like to see a variant of this that is based on a manual review/audit process rather than a time-based cooldown. Something like, upgrade once there are N independent positive reviews AND less than M negative reviews (where you can configure which people are organisations you trust to audit). And of course you would be able to audit dependencies yourself (and make your review available for others).
  • testplzignore 1y ago
    I've been wanting something like this for years. It's simply impossible for millions of companies to individually review the thousands of updates made to their thousands of dependencies every day. Imagine a world where every software update has hundreds of signoffs from companies across the industry. That is achievable if we work together. For only a few minutes a day, you too can save a CVSS 10.0 vulnerability from going unpatched :)
• DrScientist 1y ago
  The think I find most odd about the constant pressure to update to the most recent and implied best version is that there is some implicit belief that software get's uniformly better with each release. Bottom line those security bugs are not all from version 1.0 , and when you update you may well just be swapping known bugs for unknown bugs. As has been said elsewhere - sure monitor published issues and patch if needed but don't just blindly update.
  • switchbak 1y ago
    I remember this used to actually be the case, but that was many moons ago when you'd often wait a long time between releases. Or maybe the quality bar was lower generally, and it was just easier to raise it? These days it seems most software just changes mostly around the margins, and doesn't necessarily get a whole lot better. Perhaps this is also a sign I'm using boring and very stable software which is mostly "done"?
    • DrScientist 1y ago
      The question is do you take control of your update cycle - or outsource it. That's balancing the effort and risk of managing it yourself versus the risk and busy work generated from dependency churn. What evergreen update policies have done is increase development velocity ( much easier to make breaking changes if you assume everyone is upto date everywhere ) - but also increases churn.
  • cesarb 1y ago
    > Bottom line those security bugs are not all from version 1.0, and when you update you may well just be swapping known bugs for unknown bugs. One great example of that is log4shell. If you were still using version 1.0 (log4j 1.x), you were not vulnerable, since the bug was introduced in version 2.0 (log4j 2.x). There were some known vulnerabilities in log4j 1.x, but the most common configuration (logging only to a local file or to the console, no remote logging or other exotic stuff) was not affected by any of them.
    • 1y ago
• edoceo 1y ago
  I do this with my pull-thru-proxy. When new stuff shows up, I know, makes it easy to identify if urgent update is needed. I usually lag a bit. Make a dependency update a dedicated process. Update a few at a time, test, etc. slower but stable.
• programmertote 1y ago
  I know it's impossible in some software stack and ecosystem. But I live mostly in the data world, so I usually could get away from such issues by aggressively keeping my upstream dependency list lean. P.S. When I was working at Amazon, I remember that a good number of on-call tickets were about fixing dependencies (in most of them are about updating the outdated Scala Spark framework--I believe it was 2.1.x or older) and patching/updating OS'es in our clusters. What the team should have done (I mentioned this to my manager) is to create clusters dynamically (do not allow long-live clusters even if the end users prefer it that way), and upgrading the Spark library. Of course, we had a bunch of other annual and quarterly OKRs (and KPIs) to meet, so updating Spark got the lowest of priorities...
• jhatemyjob 1y ago
  This is a decent approach, another approach is to vendor your dependencies and don't update them unless absolutely necessary. The only thing you'd need to react to is major security vulnerabilities.
  • deleted 1y ago
    [deleted]
• compumike 1y ago
  There's a tradeoff and the assumption here (which I think is solid) is that there's more benefit from avoiding a supply chain attack by blindly (by default) using a dependency cooldown vs. avoiding a zero-day by blindly (by default) staying on the bleeding edge of new releases. It's comparing the likelihood of an update introducing a new vulnerability to the likelihood of it fixing a vulnerability. While the article frames this problem in terms of deliberate, intentional supply chain attacks, I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development. On the unintentional bug/vulnerability side, I think there's a similar argument to be made. Maybe even SemVer can help as a heuristic: a patch version increment is likely safer (less likely to introduce new bugs/regressions/vulnerabilities) than a minor version increment, so a patch version increment could have a shorter cooldown. If I'm currently running version 2.3.4, and there's a new release 2.4.0, then (unless there's a feature or bugfix I need ASAP), I'm probably better off waiting N days, or until 2.4.1 comes out and fixes the new bugs introduced by 2.4.0!
  • woodruffw 1y ago
    Yep, that's definitely the assumption. However, I think it's also worth noting that zero-days, once disclosed, do typically receive advisories. Those advisories then (at least in Dependabot) bypass any cooldown controls, since the thinking is that a known vulnerability is more important to remediate than the open-ended risk of a compromised update. > I'm sure the majority of bugs and vulnerabilities were never supply chain attacks: they were just ordinary bugs introduced unintentionally in the normal course of software development. Yes, absolutely! The overwhelming majority of vulnerabilities stem from normal accidental bug introduction -- what makes these kinds of dependency compromises uniquely interesting is how immediately dangerous they are versus, say, a DoS somewhere in my network stack (where I'm not even sure it affects me).
    • mik3y 1y ago
      Could a supply chain attacker simulate an advisory-remediating release somehow, i.e., abuse this feature to bypass cooldowns?
• OhMeadhbh 1y ago
  I'm not arguing that cooldowns are a bad idea. But I will argue that the article presents a simplified version of user behaviour. One of the reasons people upgrade their dependencies is to get bug fixes and feature enhancements. So there may be significant pressure to upgrade as soon as the fix is available, cooldowns be damned! If you tell people that cooldowns are a type of test and that until the package exits the testing period, it's not "certified" [*] for production use, that might help with some organizations. Or rather, would give developers an excuse for why they didn't apply the tip of a dependency's dev tree to their PROD. So... not complaining about cooldowns, just suggesting some verbiage around them to help contextualize the suitability of packages in the cooldown state for use in production. There are, unfortunately, several mid-level managers who are under pressure to close Jira tickets IN THIS SPRINT and will lean on the devs to cut whichever corners need to be cut to make it happen. [*] for some suitable definition of the word "CERTIFIED."
  • nine_k 1y ago
    There is a difference between doing $ npm i 78 packages upgraded and upgrading just one dependency from 3.7.2_1 to 3.7.2_2, after carefully looking at the code of the bugfix. The cooldown approach makes the automatic upgrades of the former kind much safer, while allowing for the latter approach when (hopefully rarely) you actually need a fix ASAP.
• andrewla 1y ago
  I'm not convinced that this added latency will help, especially if everyone uses it. It may protect you as long as nobody else uses a cooldown period, but once everyone uses one then the window between the vulnerability being introduced and it being detected will expand by a similar amount, because without exposure it is less likely to be found.
• cmckn 1y ago
  I don’t think it’s bad advice, it really just depends on the project, its dependencies, and your attack surface. I so badly want this era of mindlessly ticking boxes to end. Security is a contact sport! “Best practices” won’t get you to the promised land, you have to actually think critically about the details day to day.
• abalone 1y ago
  Sure, but there's an obvious tradeoff: You're also delaying the uptake of fixes for zero-day vulnerabilities. The article does not discuss this tradeoff.
  • woodruffw 1y ago
    The article assumes that engineers have the technical wherewithal to know when they should manually upgrade their dependencies! Clearly I should have mentioned that Dependabot (and probably others) don't consider cooldown when suggesting security upgrades. That's documented here[1]. [1]: https://docs.github.com/en/code-security/dependabot/working-with-dependabot/dependabot-options-reference#cooldown- https://docs.github.com/en/code-security/dependabot/working-...
  • awesome_dude 1y ago
    Ye Olde "Cache Invalidation" problems really Instead of updating the cache of dependencies you have immediately, the suggestion is to use the cooldown to wait.... As you point out, this means that you have a stale cache member has a critical fix applied. Next week's solution - have a dependency management tool that alerts you when critical fixes are created upstream for dependencies you have Followed by - now the zero day authors are publishing their stuff as critical fixes... Hilarity ensues
• jimbob45 1y ago
  Isn't this more or less what the Microsoft Long-Term Support (LTS) versus Short-Term Support (STS) is meant to do? LTS only receives critical updates but eschews all experimental/beta features. STS gets everything for the people that couldn't care less if their app gets hacked (e.g. apps like calculators, sandboxed internal tools, etc). I know Ubuntu and others do the same but I don't know what they call their STS equivalent.
  • Marsymars 1y ago
    Depends which Microsoft products you're referring to. If you're talking about .NET versions, MS explicitly says "The only difference is the length of support."
• xg15 1y ago
  It's a good idea, but not without weak points, I think. One of the classic scammer techniques is to introduce artificial urgency to prevent the victim from thinking clearly about a proposal. I think this would be a weakness here as well: If enough projects adopt a "cooldown" policy, the focus of attackers would shift to manipulate projects into making an exception for "their" dependency and install it before the regular cooldown period elapsed. How to do that? By playing the security angle once again: An attacker could make a lot of noise how a new critical vulnerability was discovered in their project and every dependant should upgrade to the emergency release as quickly as possible, or else - with the "emergency release" then being the actually compromised version. I think a lot of projects would could come under pressure to upgrade, if the perceived vulnerability seems imminent and the only point for not upgrading is some generic cooldown policy.
  • __MatrixMan__ 1y ago
    Along those lines: If you're packaging an exploit, it's probably best to fix a bug while you're at it. That way people who want to remove their ugly workarounds will be motivated to violate the dependency cooldown.
  • mewpmewp2 1y ago
    How would they create that noise?
    • xg15 1y ago
      Depends on the level of infiltration I guess. If the attacker managed to get themselves into a trusted position, as with the XZ backdoor, they could use the official communication channels of the project and possibility even file a CVE. If it's "only" technical access, it would probably be harder.
      • andix 1y ago
        If they file a CVE, they will draw a lot of attention from experts to the project. Even from people who never heard from this package before.
• gbin 1y ago
  Feels like the tragedy of the commons: I don't want to look at the change, I don't want to take responsibility, somebody else will take care or it, I just have to wait. Ok if this is an amazing advice and the entire ecosystem does that: just wait .... then what? We wait even more to be sure someone else is affected first? Every time I see people saying you need to wait to upgrade it is like you are accumulating tech debt: the more you wait, the more painful the upgrade will be, just upgrade incrementally and be sure you have mitigations like 0 trust or monitoring to cut early any weird behavior.
  • tempestn 1y ago
    You're not taking on any meaningful tech debt by waiting a week after a new version goes public to adopt it. As the OP says, there are services that scan popular open source tools for vulnerabilities as soon as they are released; even if a large percentage of the user base is waiting a week to update, many will still be caught in that period. And for various reasons some will still upgrade immediately.
  • bongodongobob 1y ago
    This is just completely wrong. If you are talking about a sizeable number of devices, you're not getting anything updated immediately even if you wanted to. You roll out to groups over a period of time because you don't want to break everything if there are unintended consequences. Your personal device? Sure whatever, but any fleet of devices absolutely does not get immediate updates across the board.
  • catlifeonmars 1y ago
    You’re implicitly assuming that it’s exposure to downstream consumers that causes the malicious packages to be discovered, but we haven’t actually seen that in the last couple of major supply chain attacks. Instead it just buys time for the maintainers to undo the damage.
  • andix 1y ago
    Even if less consumers will notice a compromise and report it, it still gives additional time for security researchers to analyze the packages, and for maintainers to notice themselves they got compromised There are a lot of companies out there, that's scan packages and analyze them. Maintainers might notice a compromise, because a new release was published they didn't authorize. Or just during development, by getting all their bitcoin stolen ;)
• BrenBarn 1y ago
  The culture of constant updates is a cancer on the software world.
• ChrisMarshallNY 1y ago
  I'm an advocate for safe dependency usage. I can't in good conscience, say "Don't use dependencies," which solves a lot of problems, but I can say "Let's be careful, out there," to quote Michael Conrad. I strongly suspect that a lot of dependencies get picked because they have a sexy Website, lots of GH stars and buzz, and cool swag. I tend to research the few dependencies that I use. I don't depend lightly. I'm also fortunate to be in the position where I don't need too many. I am quite aware that, for many stacks, there's no choice.
• jrowen 1y ago
  A “cooldown” is exactly what it sounds like: a window of time between when a dependency is published and when it’s considered suitable for use. My understanding of a cooldown, from video games, is a period of time after using an ability where you can't use it again. When you're firing a gun and it gets too hot, you have to wait while it cools down. I was trying to apply this to the concept in the article and it wasn't quite making sense. I don't think "cooldown" is really used when taking a pie out of the oven, for example. I would call this more a trial period or verification window or something?
  • bauble 1y ago
    You're correct; it's a misuse of the term. Better terms would be embargo, quarantine, or just delay.
  • 333c 1y ago
    The dependency has just come out of the oven, so it's too hot and must cool down ;)
    • jrowen 1y ago
      I addressed this though lol. Nobody says "my pie is on cooldown."
• largbae 1y ago
  Wouldn't this be a great use case of any agentic AI coding solution? A background agent scanning code/repo/etc and making suggestions both for and against dependency updates? Copilot seems well placed with its GitHub integration here, it could review dependency suggestions, CVEs, etc and make pull requests.
• andix 1y ago
  One reason for cooldowns is not mentioned: maintainers often notice by themselves they got compromised. The attacker will try to figure out when they are the least available: during national holidays, when they sleep, during conferences they attend, when they are on sick leave, personal time off, ... Many projects have only a few or even only a single person, that's going to notice. They are often from the same country (time zone) or even work in the same company (they might all attend the same conference or company retreat weekend).
• kazinator 1y ago
  Cooldowns won't do anything if it takes wide deployment in order for the problem to be discovered. If the code just sits there for a week without anyone looking at it, and is then considered cooled down just due to the passage of time, then the cool down hasn't done anything beneficial. A form of cooldown that could would in terms of mitigating problems would be a gradual rollout. The idea is that the published change is somehow not visible to all downstreams at the same time. Every downstream consumer declares a delay factor. If your delay factor is 15 days, then you see all new change publications 15 days later. If your delay factor is 0 days, you see everything as it is published, immediately. Risk-averse organizations configure longer delay factors. This works because the risk-takers get hit with the problem, which then becomes known, protecting the risk-averse from being affected. Bad updates are scrubbed from the queue so those who have not yet received them due to their delay factor wlll not see those updates.
• deleted 1y ago
  [deleted]
• alphazard 1y ago
  For some reason everyone wants to talk about all the solutions to supply chain attacks except designing languages to avoid them in the first place. Austral[0] gets this right. I'm not a user, just memeing a good idea when I see it. Most languages could be changed to be similarly secure. No global mutable state, no system calls without capabilities, no manual crafting of pointers. All the capabilities come as tokens or objects passed in to the main, and they can be given out down the call tree as needed. It is such an easy thing to do at the language level, and it doesn't require any new syntax, just a new parameter in main, and the removal of a few bad ideas. [0] https://austral-lang.org/ https://austral-lang.org/
  • nextaccountic 1y ago
    How does Austral avoids supply chain attacks?
    • alphazard 1y ago
      The typical attack tries to get the victim to load a module which accesses information on the filesystem or in another module's memory, then exfiltrate that information using network access. Other attacks may install backdoors used for infiltration once the service is deployed. If the module provides some small helper function like `appendSmileEmoji(x: String) -> String` then it shouldn't be accessing the network or the filesystem, and the only memory it should have access to is the string passed into the function. The programmer already provided enough information to explain to the compiler that the function should not do these things but poor language design allows them to happen anyways. Components which do have access to the network should be vetted thoroughly because they can facilitate infiltration and exfiltration. Most supply chain attacks aren't introduced directly in a high profile project like a web framework, they are introduced deeper in the dependency chain in the long tail of transitively imported modules. In Austral (or any language using these good ideas) a small poorly vetted function like `appendSmileEmoji` wouldn't be called with access to the filesystem or network. Without access to any resources, the worst it can do is append a frowning face or a null character or something annoying. You can tell by its function signature above, it doesn't take a capability for the network or filesystem, just a String. So trying to ship a change that secretly accesses the filesystem would produce a "variable not found" error. To even get the new version to compile, you would have to add an argument to take a filesystem capability. `appendSmileEmoji(x: String, fs: FS) -> String`. That breaks any dependent modules because they have the wrong number of arguments, and looks very suspicious.
• kachapopopow 1y ago
  pick your poison: - you are vulnerable for 7 days because of a now public update - you are vulnerable for x (hours/days) because of a supply chain attack I think the answer is rather simple: subscribe to a vulnerability feed, evaluate & update. The amount of times automatic updates are necessary is near zero as someone who has ran libraries that are at times 5 to 6 years out of date exposed to the internet without a single event of compromise and it's not like these were random services, they were viewed by hundreds of thousands of unique addresses. There was only 3 times in the last 4 years where I had to perform updates due to a publically exposed service where these vulnerabilities affected me. Okay, the never being compromised part is a lie because of php, it's always PHP (monero-miner I am sure everyone is familiar with). The solution for that was to stop using PHP and assosiated software. Another one I had problems with was CveLab (GitLab if you couldn't tell), there has been so many critical updates pointing to highly exploitable CVE's that I had decided to simply migrate off it. In conclusion avoiding bad software is just as important as updates from my experience lowering the need for quick and automated actions.
• torginus 1y ago
  I just had a thought about dependency chain attacks - has there been any example where instead of overtly compromising the code, they injected an exploitable bug? Though I guess it'd be hard to prove intent in this case.
  • Jweb_Guru 1y ago
    These are just backdoors as a vulnerability class. It does happen occasionally even in high quality widely used libraries, but it's very rare and usually quite sophisticated (e.g. https://en.wikipedia.org/wiki/XZ_Utils_backdoor https://en.wikipedia.org/wiki/XZ_Utils_backdoor in 2024). A lot of these attacks, even when highly sophisticated, only really work when the attack payload can be hidden in a binary--it's very very hard to insert a useful backdoor into the source code of an actively maintained project that receives regular code reviews without anyone noticing (which is not to say it's impossible).
    • chuckadams 1y ago
      It's also worth noting that the xz exploit didn't even live in the xz source, it was added to the source by the attacker before uploading it to the package build farms. With most of the current package repositories, there's not even an expectation that the bundles you get are derived from a particular tag or commit hash, let alone a detailed chain of custody. I remember RPM in the old days made a big deal out of "pristine sources" to which patches would be applied, but it still has no way to prove or enforce that claim.
• nrhrjrjrjtntbt 1y ago
  Didnt know this had a name i.e. cooldowns but I totally agree. You want tooling to generate upgrade PRs but on a safe cadence. For urgent patches, i.e. zero days, a security team (internal or external can judge)
• weinzierl 1y ago
  The article assumes that the duration of the attack window is fixed and independent of the cooldown period. It's not. Once everyone waits to update the time until the vulnerability is found increases and the attack window will grow.
  • presentation 1y ago
    What if cooldowns were implemented by a package manager somewhat randomized, so that it’s more of a gradual rollout instead of a fixed cooldown period?
• xmddmx 1y ago
  Why not do an empirical A/B test: Set up two honeypots (or perhaps 2000 for statistical significance). A gets zero updates, B gets all updates immediately. See which ones get pwned faster.
• presentation 1y ago
  If everyone uses cooldowns would the discovery time potentially get pushed back?
  • pigbearpig 1y ago
    Probably, because researchers/vendors/maintainers aren't going to catch everything, but you have less exposure too.
• yyyk 1y ago
  This assumes the direct mode of exploitation. The indirect mode (where an attacker inserts a vulnerability) does not necessarily have the 'short time span' issue. So not a complete solution by any means. Also, there should a way to distinguish between security updates and normal updates for this. If there is, a cooldown is a useful idea in general for normal updates, since (presumably) the current version works and the new version may introduce bugs.
  • acdha 1y ago
    > This assumes the direct mode of exploitation. The indirect mode (where an attacker inserts a vulnerability) does not necessarily have the 'short time span' issue. So not a complete solution by any means. The short time span isn’t just because exploits get attention: it’s to allow the groups which do automated analysis time to respond. Significantly increasing the challenge level for an attacker to introduce a vulnerability is a meaningful improvement even if it doesn’t prevent that class of attack entirely.
• bandrami 1y ago
  I always remind my team that users of Debian Stable missed heartbleed entirely because their OpenSSL was too old. A lot of people and tools have the attitude that regressions never happen.
• joshdavham 1y ago
  Also interesting to note that the author is involved with the uv python package manager, homebrew and is a member of pypa (Python Packaging Authority). Very impressive!
  • ivh 1y ago
    He mentions uv in a footnote. I will try to remember to add --exclude-newer $(date -I -d '2 weeks ago') the next time I use uv lock --upgrade.
• deleted 1y ago
  [deleted]
• darig 1y ago
  [dead]
• illusive4080 1y ago
  The Noir iPhone extension won’t darken this white page because the author has explicitly told Noir to not darken the page. Why not just make a dark mode?
• unsungNovelty 1y ago
  I always liked semver. Never have I liked just churning version numbers. It tells you exactly what a release is. Dont even need to look at release notes (which some doesnt have). It helps a lot with these dev cycles. Recently, I saw even EmberJS moved away from it. I still think they could've done it without leaving semver. Maybe I'm wrong.
• josephcsible 1y ago
  Isn't this one of those things that can only work if most people don't do it? If everyone did it, wouldn't something like https://xkcd.com/989/ https://xkcd.com/989/ result?
• OakNinja 1y ago
  On dependencies, a friend of mine once said ”The early bird might get the worm but its mouse number two that get’s the cheese”.
• m000 1y ago
  Isn't the title kind of self-conflicting? Dependency cooldowns work because not everybody is using them. The more people start using dependency cooldowns, the less effective they become.
  • macintux 1y ago
    There are comments elsewhere in the discussion indicating that these attacks tend to be discovered by automated scanners and the maintainers themselves, not by customers negatively impacted.
    • teaganga 1y ago
      i had the same thing in mind, though, is not that part of the process until the release of available? should not the release already have the maintainers input?
• jappgar 1y ago
  If we all did it, it wouldn't work.
  • Quizzical4230 1y ago
    Could you please elaborate on this?
    • alkallio 1y ago
      Because nobody would test the updates.
• morshu9001 1y ago
  Heh, my stuff is always pinned to old versions just cause I don't want to deal with breakages, so... way ahead of you. Or behind. Also whoever keeps changing the entire API of node-redis, please stop.
• snowwrestler 1y ago
  When I admin’ed a bunch of Wordpress and Drupal websites (open source CMSes), I was very quick to install security patches, and relatively slow to install normal updates—for this reason. It’s one of the main reasons I used popular open source software: so I could ride the coattails of the rest of the community. Basically everyone else could be my beta tester. A regular update was an input to the community security practice, so I would let it settle for a while. A security patch was an output of the community security practice, so I would install ASAP, even if it meant breaking a feature temporarily. I also manually managed dependencies as commits to the main codebase, meaning my entire site was one deployable object from a single Git repo. The “modern” practice today seems to instead favor a minimal repo and resolving and pulling dependencies at deploy time. Personally I think this is a bad idea that has amplified the risk of supply chain attacks. And now the idea is apparently back to: give it a little while. Tell the automatic dependency puller to chill out and wait.
• cryptonector 1y ago
  If every dependent implements a dependency cooldown, then the cooldown might as well not exist.
• jbkgujklgui 1y ago
  Also protects against when a library developer suddenly goes rogue and enshittifies the library, to make a buck or two.
• coastalpuma 1y ago
  I really wish homebrew supported this feature. There are a bunch of terminal tools I have installed through it (and wouldn't make sense to sandbox) that are basically one-person projects, and I imagine subject to supply chain compromises. I don't urgently need the very latest versions for any of them, as they are unlikely to have exploitable security vulnerabilities on my machine (due to not being involved with any network facing service).
• Seattle3503 1y ago
  > vendors are still incentivized to discover and report attacks quickly, but are not as incentivized to emit volumes of blogspam about “critical” attacks on largely underfunded open source ecosystems. How?
• TZubiri 1y ago
  Nice. Additionally we should be using dependency limits and dependency moderation. Half of my job when reviewing work is telling junior devs to try to do something without the dependency. Usually they just learn about a very basic skill. "Ok so you downloaded a thing called apscheduler, what does that do?" "So it's like cron?" "Can you use threads, or even better just use a separate process?" "Cool" My (least) favourite example is the amount of juniors that download a dependency from a random indian dev to store and read credentials from .env . Just open() and read() the security critical file my dude.
• ElevenLathe 1y ago
  Maybe the repositories should do this by default?
• mattwilsonn888 1y ago
  The issue with this model in the most general sense is that it is zero-sum, and at the limit it doesn't provide hardly any security. I delay the use of updated software by a week, and anyone that doesn't takes the risk. Therefore I, the user of the cooldown, enjoys reduced risk at the expense of everyone not implementing a cooldown. If everyone simply delays their updates, then there is nobody to suffer an attack which notifies users of the cooldown (in this case, everybody). The blog post makes the argument that the vendors are incentivized to discover these attacks in this time, but that's an entirely different argument and if that were true, they would already be doing that. In fact, auditing updates for vulnerabilities is the general solution. The whole appeal of the cooldowns is that you don't have to do that - the cost is that it's a zero-sum game reliant on the suffering of those less wise.
• time4tea 1y ago
  Been saying this for years! All the security theatre! False urgency is red flag when it comes to emails and sms.. I dont see why people don't see the same when it comes to dependency alerts. The vast vast majority of alerts are totally useless, when it comes to your actual deployed system, unexploitable and irrelevant. Rushing to update causes make-work, churn and loss of focus on the real. Don't get me wrong, I love secure systems, but dependency theatre doesn't make it. Very very occasionally, you'll get a high priority signal, and then yes act on it immediately. Your deployment systems should be designed so that this is just another release. No expedited short cuts, just another day. Reducing dependencies, defense in depth, validation, type-safe, null-safe languages etc all good.. endless dependabot PRs with no understanding of the reason or impact, bad.