5 ms·

I've always thought that dependabot was busy-work, a waste of time. This article makes a good point that drives it home: Alarams that aren't real make all alarm

I've always thought that dependabot was busy-work, a waste of time. This article makes a good point that drives it home: Alarams that aren't real make all alarms useless. Dependabot is especially painful in non-typed languages (Python, Ruby, and especially Javascript) where "upgrading" a library can break things that there's no way to know until production. Maybe the constant work, extra build time (and cash for all that), and risk of breaking production, is worth it for the 0.01% of the time there's a real vulnerability? It seems like a high price to pay though. When there are major software vulnerabilities (like log4j), the whole industry usually swarms around it, and the alarm has high value. I just realized how much CircleCI probably loves Dependabot. I wonder what hit % their margins would take if we moved off it collectively as an industry.

danenania 4y ago
If you use vendoring, it's also worth considering that there's always some inherent security risk in upgrading dependencies. If an attacker takes control of a package somewhere in your dependency tree, you don't get compromised until you actually install a new version of that package. This risk can often outweigh the risk of very minor/dev-facing CVEs.
- feross 4y ago
  Shameless plug: This is what I’m building Socket.dev to solve. Socket watches for changes to “package manifest” files such as package.json, package-lock.json, and yarn.lock. Whenever a new dependency is added in a pull request, Socket analyzes the package's behavior and leaves a comment if it is a security risk. You can see some real-world examples here: https://socket.dev/blog/socket-for-github-1.0 https://socket.dev/blog/socket-for-github-1.0
  - e1g 4y ago
    We use Socket and my favorite feature is when you highlight new dependencies with a post-install hook. It’s not always a problem, but almost always a smell. One feature request: please allow me to “suppress” warnings for a specific package+version combo. This is useful for activist libs that take a political stance - I know it happens, but often cannot remove them, and don’t want to continue flagging the same problem at every sec review.
bawolff 4y ago
I kind of feel like dependabot alerts should be treated like a coding convention error - that extra whitespace isnt actually causing a problem but we fix it right away. Otherwise you have to start analyzing the alerts, and good luck with that. The low severity ones are marked critical and the scary ones are marked low. Suddenly you have 200 unfixed alerts and its impossible to know if somewhere in that haystack is an important one.
- jefftk 4y ago
  Impossible? The article we're commenting on describesa tool that does this.
smcleod 4y ago
IMO Dependabot is really dreadful at its job. Try Renovate - it's really brilliant, fast, flexible, supports properly binding PRs/MRs.
mfer 4y ago
> When there are major software vulnerabilities (like log4j), the whole industry usually swarms around it, and the alarm has high value. You're leaving me with the impression that you think we should only patch major software vulnerabilities. This I would disagree with. Minor vulnerabilities can be used, especially in groups, to do things we don't anticipate. It's not just about a single vulnerability but about how an attacker can leverage multiple different vulnerabilities together.
- deleted 4y ago
  [deleted]
rpep 4y ago
> can break things that there's no way to know until production I would argue that any production system should have enough tests that upgrading a dependency that breaks compatibility should cause failure of the test suite in some way
- wizofaus 4y ago
  You can argue that but I've seen plenty of production systems relying on dynamically typed languages where breaking third-party dependency changes were indeed undetectable until a new version was deployed and particular features stopped working at runtime. It's certainly not unheard of for the same to occur with statically typed languages but at least there's a whole class of potential errors that are caught automatically by compilers/linkers with no extra work on behalf of the programmers. And even when errors do occur at runtime due to JIT compilation etc. they're more likely to have helpful information attached to them. Either way, all the automated tests in the world aren't likely to catch a problem that's related to incorrect library versions being loaded from some global cache that you have limited control over (having moved back from SaaS to on-prem based software recently I've been very quickly reminded of that reality!).
- stevebmark 4y ago
  This is a very good argument for why Ruby should never be used for production systems, and neither should Javascript without using Typescript.
  - jaredsohn 4y ago
    Why specifically Ruby here? How about python, php, etc? Also, many dynamic languages have static type checkers if you want to use them (i.e. Ruby has Sorbet).
tardismechanic 4y ago
> where "upgrading" a library can break things that there's no way to know until production. unless you have automated testing

danenania 4y ago

If you use vendoring, it's also worth considering that there's always some inherent security risk in upgrading dependencies. If an attacker takes control of a package somewhere in your dependency tree, you don't get compromised until you actually install a new version of that package. This risk can often outweigh the risk of very minor/dev-facing CVEs.

feross 4y ago
Shameless plug: This is what I’m building Socket.dev to solve. Socket watches for changes to “package manifest” files such as package.json, package-lock.json, and yarn.lock. Whenever a new dependency is added in a pull request, Socket analyzes the package's behavior and leaves a comment if it is a security risk. You can see some real-world examples here: https://socket.dev/blog/socket-for-github-1.0 https://socket.dev/blog/socket-for-github-1.0
- e1g 4y ago
  We use Socket and my favorite feature is when you highlight new dependencies with a post-install hook. It’s not always a problem, but almost always a smell. One feature request: please allow me to “suppress” warnings for a specific package+version combo. This is useful for activist libs that take a political stance - I know it happens, but often cannot remove them, and don’t want to continue flagging the same problem at every sec review.

bawolff 4y ago

I kind of feel like dependabot alerts should be treated like a coding convention error - that extra whitespace isnt actually causing a problem but we fix it right away. Otherwise you have to start analyzing the alerts, and good luck with that. The low severity ones are marked critical and the scary ones are marked low. Suddenly you have 200 unfixed alerts and its impossible to know if somewhere in that haystack is an important one.

jefftk 4y ago
Impossible? The article we're commenting on describesa tool that does this.

smcleod 4y ago

IMO Dependabot is really dreadful at its job. Try Renovate - it's really brilliant, fast, flexible, supports properly binding PRs/MRs.

mfer 4y ago

> When there are major software vulnerabilities (like log4j), the whole industry usually swarms around it, and the alarm has high value. You're leaving me with the impression that you think we should only patch major software vulnerabilities. This I would disagree with. Minor vulnerabilities can be used, especially in groups, to do things we don't anticipate. It's not just about a single vulnerability but about how an attacker can leverage multiple different vulnerabilities together.

deleted 4y ago
[deleted]

rpep 4y ago

> can break things that there's no way to know until production I would argue that any production system should have enough tests that upgrading a dependency that breaks compatibility should cause failure of the test suite in some way

wizofaus 4y ago
You can argue that but I've seen plenty of production systems relying on dynamically typed languages where breaking third-party dependency changes were indeed undetectable until a new version was deployed and particular features stopped working at runtime. It's certainly not unheard of for the same to occur with statically typed languages but at least there's a whole class of potential errors that are caught automatically by compilers/linkers with no extra work on behalf of the programmers. And even when errors do occur at runtime due to JIT compilation etc. they're more likely to have helpful information attached to them. Either way, all the automated tests in the world aren't likely to catch a problem that's related to incorrect library versions being loaded from some global cache that you have limited control over (having moved back from SaaS to on-prem based software recently I've been very quickly reminded of that reality!).
stevebmark 4y ago
This is a very good argument for why Ruby should never be used for production systems, and neither should Javascript without using Typescript.
- jaredsohn 4y ago
  Why specifically Ruby here? How about python, php, etc? Also, many dynamic languages have static type checkers if you want to use them (i.e. Ruby has Sorbet).

tardismechanic 4y ago

> where "upgrading" a library can break things that there's no way to know until production. unless you have automated testing