4 ms·

That's fine as long as HEAD doesn't break things. I can no longer count the number of times we had an issue with a "supposedly" minor release that ended up bre

by bb88 4y ago

That's fine as long as HEAD doesn't break things. I can no longer count the number of times we had an issue with a "supposedly" minor release that ended up breaking major things in our stack. Most of them were things that could have been detected using unit tests or some kind of basic regression testing. If you have a 1000 dependency packages, and at any point in time 0.1% of them are broken, then odds are you will always have something broken.

lbrandy 4y ago
We should be clear... in these large organizations... HEAD is always broken. But it has the advantage of being broken for everyone, tested by everyone, fixed by everyone, and thus fixed for everyone. And this usually makes it far better than the alternatives. Having 1000 dependencies with versions pinned means you are living alone and will run into fewer issues, but when they do come, they will be absolute nightmares that no one else is dealing with and no one can help with. And one day you'll have to do the game of begging someone else to upgrade their version of a downstream thing to fix the issue, and they won't, so you'll try to get the other group to backport the fix in their thing to the version you can't upgrade off. And they won't. etc. etc. Full versioning is the worst of all approaches, IMO, for large complex interconnected codebases (especially ones that are many-to-many from libraries to output binaries) but it absolutely is sometimes the only viable one (for example, the entire open-source-ecosystem is a giant(er) version of this problem, and in that space, versioning is the only thing that I can imagine working).
- galangalalgol 4y ago
  Supply chain attacks are worsened if everyone lives at the head. Staying far enough behind that some brave (and hopefully small) project discovers the compromise of a repo for some dependency five layers deep before you re-pin to a new version is probably the best mitigation short of some permissions based model like Austral is working on.
  - VHRanger 4y ago
    Supply chain attacks only matter for libraries that can make their own network call, or libraries that directly touch unsanitized web input however?
    - hamandcheese 4y ago
      How does one restrict network access for a library?
      - galangalalgol 4y ago
        That was my thinking. In Austral that is a thing, but in c++ not so much. And if you are building the dependency from source, which is pretty common, the common build systems are all Turing complete themselves so they can take over your pipeline and do bad things.
      - lanstin 4y ago
        Run your CI in an allow listed only network and only allow access to either your private, security scanned, mirror or else keep well trusted things. Even if a bitcoin miner gets into the stack it can’t send the results to the source so it is less dangerous.
        galangalalgol 4y ago
        But it can insert itself in your pipeline so that anyone who depends on you is also infected. We have already seen CI worms for thevlack of a better term. One was briefly spreading in crates.io early last year as part of the "crate depression" thing.
    - bawolff 4y ago
      Supply chain attacks matter anytime you need to trust the code being run. Which is usually always. Most libraries have network access, but even if they didn't, supply chain attacks could be relavent (but probably less generic)
  - 4y ago
- mihaaly 4y ago
  > it has the advantage of being broken for everyone, tested by everyone, fixed by everyone The correct word here is not advantage but risk. While making a product you rely on something, some other product/package. Thousand of them if you are unlucky. If you have 1000 packages to collaborate in the development/testing/QA of them you will do nothing else! Also try building a house on a concrete that is still maturing, with tools not ready yet, I dare you! Careful developers rely on reliable things. It is a shame in this industry this is not available. Either released(!) things are not ready yet, or are in demise already, the sweet spot is tiny.
  - HelloNurse 4y ago
    It depends on the context: in a React Native project you are almost guaranteed to realize the hard way that you depend on abandonware that won't ever support the latest React Native version or the latest smartphone operating systems, while in a Python project you are almost guaranteed to attempt to use something that is hard to install or compile, or just doesn't work on your slightly divergent platform (e.g. Windows).
- humanrebar 4y ago
  > We should be clear... in these large organizations... HEAD is always broken. I don't know about all of them, but a lot of them have CI set up such that HEAD is never broken for some definition of broken. Practically speaking it's basically impossible to fuzz test everything, but typically builds and reasonably fast and reliable tests are run before HEAD includes new changes. Being efficient about supporting this workflow is more or less what monorepo build systems like bazel, buck, and pants were designed for.
UncleMeat 4y ago
You don’t just say “okay C++20 is released, swap the compiler flags and break everyone.” But you do say “okay, C++20 is released, let’s fix all the build errors and deploy it company-wide.”
JohnFen 4y ago
This. Years of experience has taught me to avoid "living at HEAD". HEAD has undiscovered problems. Better be be just a bit behind that.

by bb88 4y ago

lbrandy 4y ago

We should be clear... in these large organizations... HEAD is always broken. But it has the advantage of being broken for everyone, tested by everyone, fixed by everyone, and thus fixed for everyone. And this usually makes it far better than the alternatives. Having 1000 dependencies with versions pinned means you are living alone and will run into fewer issues, but when they do come, they will be absolute nightmares that no one else is dealing with and no one can help with. And one day you'll have to do the game of begging someone else to upgrade their version of a downstream thing to fix the issue, and they won't, so you'll try to get the other group to backport the fix in their thing to the version you can't upgrade off. And they won't. etc. etc. Full versioning is the worst of all approaches, IMO, for large complex interconnected codebases (especially ones that are many-to-many from libraries to output binaries) but it absolutely is sometimes the only viable one (for example, the entire open-source-ecosystem is a giant(er) version of this problem, and in that space, versioning is the only thing that I can imagine working).

galangalalgol 4y ago
Supply chain attacks are worsened if everyone lives at the head. Staying far enough behind that some brave (and hopefully small) project discovers the compromise of a repo for some dependency five layers deep before you re-pin to a new version is probably the best mitigation short of some permissions based model like Austral is working on.
- VHRanger 4y ago
  Supply chain attacks only matter for libraries that can make their own network call, or libraries that directly touch unsanitized web input however?
  - hamandcheese 4y ago
    How does one restrict network access for a library?
    - galangalalgol 4y ago
      That was my thinking. In Austral that is a thing, but in c++ not so much. And if you are building the dependency from source, which is pretty common, the common build systems are all Turing complete themselves so they can take over your pipeline and do bad things.
    - lanstin 4y ago
      Run your CI in an allow listed only network and only allow access to either your private, security scanned, mirror or else keep well trusted things. Even if a bitcoin miner gets into the stack it can’t send the results to the source so it is less dangerous.
      - galangalalgol 4y ago
        But it can insert itself in your pipeline so that anyone who depends on you is also infected. We have already seen CI worms for thevlack of a better term. One was briefly spreading in crates.io early last year as part of the "crate depression" thing.
  - bawolff 4y ago
    Supply chain attacks matter anytime you need to trust the code being run. Which is usually always. Most libraries have network access, but even if they didn't, supply chain attacks could be relavent (but probably less generic)
- 4y ago
mihaaly 4y ago
> it has the advantage of being broken for everyone, tested by everyone, fixed by everyone The correct word here is not advantage but risk. While making a product you rely on something, some other product/package. Thousand of them if you are unlucky. If you have 1000 packages to collaborate in the development/testing/QA of them you will do nothing else! Also try building a house on a concrete that is still maturing, with tools not ready yet, I dare you! Careful developers rely on reliable things. It is a shame in this industry this is not available. Either released(!) things are not ready yet, or are in demise already, the sweet spot is tiny.
- HelloNurse 4y ago
  It depends on the context: in a React Native project you are almost guaranteed to realize the hard way that you depend on abandonware that won't ever support the latest React Native version or the latest smartphone operating systems, while in a Python project you are almost guaranteed to attempt to use something that is hard to install or compile, or just doesn't work on your slightly divergent platform (e.g. Windows).
humanrebar 4y ago
> We should be clear... in these large organizations... HEAD is always broken. I don't know about all of them, but a lot of them have CI set up such that HEAD is never broken for some definition of broken. Practically speaking it's basically impossible to fuzz test everything, but typically builds and reasonably fast and reliable tests are run before HEAD includes new changes. Being efficient about supporting this workflow is more or less what monorepo build systems like bazel, buck, and pants were designed for.

UncleMeat 4y ago

You don’t just say “okay C++20 is released, swap the compiler flags and break everyone.” But you do say “okay, C++20 is released, let’s fix all the build errors and deploy it company-wide.”

JohnFen 4y ago

This. Years of experience has taught me to avoid "living at HEAD". HEAD has undiscovered problems. Better be be just a bit behind that.