3 ms·

You will also not get very far in engineering if you have a production incident, someone proposes a thing that will mitigate it partially but significantly, and

by jerf 2mo ago

You will also not get very far in engineering if you have a production incident, someone proposes a thing that will mitigate it partially but significantly, and you insist that we can't deploy that mitigation because we need to do the multi-year project that will actually fix it instead. Even if we still need that project, we also need that mitigation. Sure, solve the problem of "auditing and trusting codebases". Go for it. Shouldn't take you until, oh, let me be generous and give you until next Monday. No sweat.

rcxdude 2mo ago
Feel free to turn it off yourself, just don't be surprised when the attackers switch tactics. And don't make life harder for everyone else by pushing it as mandatory. (if you want a better mitigation: don't automatically update dependencies in CI. At a minimum have a cooling-off period that you only bypass on manual review. This is not hard to implement and at least gives some time for alarm bells to be sounded before you're pwned. Probably while doing the updates you can also just do a quick sanity check on the changes, that'll also catch the ones using a blatant install hook)
- esseph 2mo ago
  > if you want a better mitigation: don't automatically update dependencies in CI. At a minimum have a cooling-off period that you only bypass on manual review. This is not hard to implement and at least gives some time for alarm bells to be sounded before you're pwned. One of the problems a lot of companies are having right now is trying to determine a proper window size between CVE release and patching. On one hand, you have LLM created 0days and attacks are happening almost as fast as CVEs can be posted, and CVEs are being posted at lightning speed. You maybe had a month or a few weeks to patch before, now the window may be down to days or even hours. On the other hand, you want to prevent these repo bombing attacks. There's a tension there but I think we're going to end up measuring that window in hours within the year, if we're not already there.
  - rcxdude 2mo ago
    If you need quick responses to such things, you're gonna need some intelligence in the loop regardless, even if it's just deciding when it's worth pushing a new release to prod.
    - esseph 2mo ago
      What if it needs to get patched and pushed while you and everyone else in the org are asleep? I mean that's a big window of time now.
      - rcxdude 2mo ago
        Always has been, arguably. Either way I don't think most people have their CI configured to automatically re-run when a dependency updates (imagine the thundering herd!), the compromises are more incidental to runs happening for other reasons, and that configuration neither guarantees a quick response nor protects against supply chain attacks. (If you find you do need updates when you're sleeping, you probably need to delegate that process or at least the process of checking the update to someone you trust, and you're probably paying for the service. This blog post is in large part an ad for just such a service).
- jerf 2mo ago
  I mostly use languages where the feature doesn't even exist. We don't generally miss it. I also blocked a proposed mandatory dependabot at work a few months back because I didn't like the way it created a pipeline for any hacker to push a hacked dependency straight into someone's codebase. I'm lifting my objection now that dependabot is defaulting to a 3-day cooldown, though the code bases I'm managing I'm setting to 7 days. (Not to be behind everyone else; I'd be fine if everyone joined me at 7 days. I don't really accept the freeloading objection, there's plenty of entities scanning things now no matter what cooldown you set. I just think 3 is a little tight to expect the full discovery and remediation to take place.) So, whatever sort of "but it's really hard! I bet you don't do it yourself" implications you may be trying to draw fall very flat. Or whatever you may have been trying to imply about trusting this step but not trusting others... no, I do defense in depth. Giving up on defense in depth because one step isn't enough... well, I think I've probably played the "you won't get very far in engineering" card enough, but hey, here's one last time.
  - rcxdude 2mo ago
    Out of curiosity, which languages? The equivalent operations for C and C++ do generally involve a lot of arbitrary code execution to do the build, for example, they just don't have one central package manager and repository. I can't think of any package managers off the top of my head that don't have some similar feature, but I am obviously not familiar with all of them. You can see my other responses in this thread about what I think of this in the context of defence in depth. I think disabling install hooks is a very weak defence with substantial downsides and defence in depth does not mean just taking every option that might foil some attack. There are much better means to get a robust extra layer of defence in this context, which is mainly about protecting your publishing pipelines from become a vector for a worm.
BigTTYGothGF 2mo ago
> You will also not get very far in engineering if Man, I wish.

by jerf 2mo ago

rcxdude 2mo ago

Feel free to turn it off yourself, just don't be surprised when the attackers switch tactics. And don't make life harder for everyone else by pushing it as mandatory. (if you want a better mitigation: don't automatically update dependencies in CI. At a minimum have a cooling-off period that you only bypass on manual review. This is not hard to implement and at least gives some time for alarm bells to be sounded before you're pwned. Probably while doing the updates you can also just do a quick sanity check on the changes, that'll also catch the ones using a blatant install hook)

esseph 2mo ago
> if you want a better mitigation: don't automatically update dependencies in CI. At a minimum have a cooling-off period that you only bypass on manual review. This is not hard to implement and at least gives some time for alarm bells to be sounded before you're pwned. One of the problems a lot of companies are having right now is trying to determine a proper window size between CVE release and patching. On one hand, you have LLM created 0days and attacks are happening almost as fast as CVEs can be posted, and CVEs are being posted at lightning speed. You maybe had a month or a few weeks to patch before, now the window may be down to days or even hours. On the other hand, you want to prevent these repo bombing attacks. There's a tension there but I think we're going to end up measuring that window in hours within the year, if we're not already there.
- rcxdude 2mo ago
  If you need quick responses to such things, you're gonna need some intelligence in the loop regardless, even if it's just deciding when it's worth pushing a new release to prod.
  - esseph 2mo ago
    What if it needs to get patched and pushed while you and everyone else in the org are asleep? I mean that's a big window of time now.
    - rcxdude 2mo ago
      Always has been, arguably. Either way I don't think most people have their CI configured to automatically re-run when a dependency updates (imagine the thundering herd!), the compromises are more incidental to runs happening for other reasons, and that configuration neither guarantees a quick response nor protects against supply chain attacks. (If you find you do need updates when you're sleeping, you probably need to delegate that process or at least the process of checking the update to someone you trust, and you're probably paying for the service. This blog post is in large part an ad for just such a service).
jerf 2mo ago
I mostly use languages where the feature doesn't even exist. We don't generally miss it. I also blocked a proposed mandatory dependabot at work a few months back because I didn't like the way it created a pipeline for any hacker to push a hacked dependency straight into someone's codebase. I'm lifting my objection now that dependabot is defaulting to a 3-day cooldown, though the code bases I'm managing I'm setting to 7 days. (Not to be behind everyone else; I'd be fine if everyone joined me at 7 days. I don't really accept the freeloading objection, there's plenty of entities scanning things now no matter what cooldown you set. I just think 3 is a little tight to expect the full discovery and remediation to take place.) So, whatever sort of "but it's really hard! I bet you don't do it yourself" implications you may be trying to draw fall very flat. Or whatever you may have been trying to imply about trusting this step but not trusting others... no, I do defense in depth. Giving up on defense in depth because one step isn't enough... well, I think I've probably played the "you won't get very far in engineering" card enough, but hey, here's one last time.
- rcxdude 2mo ago
  Out of curiosity, which languages? The equivalent operations for C and C++ do generally involve a lot of arbitrary code execution to do the build, for example, they just don't have one central package manager and repository. I can't think of any package managers off the top of my head that don't have some similar feature, but I am obviously not familiar with all of them. You can see my other responses in this thread about what I think of this in the context of defence in depth. I think disabling install hooks is a very weak defence with substantial downsides and defence in depth does not mean just taking every option that might foil some attack. There are much better means to get a robust extra layer of defence in this context, which is mainly about protecting your publishing pipelines from become a vector for a worm.

BigTTYGothGF 2mo ago

> You will also not get very far in engineering if Man, I wish.