4 ms·

I agree that breaking a production user's experience is as verboten as it gets for agile software development. That's why we have massive test suites and CI in

I agree that breaking a production user's experience is as verboten as it gets for agile software development. That's why we have massive test suites and CI infrastructure. However, you do want to avoid "CI handcuffs", where either because CI is too brittle or because you're simply integrating too many independent projects which, in order to push a change requiring coordinating changes elsewhere, requires massively outsized developer effort in order to push things through with zero CI breakage. This is more of a problem in CI systems with binary metrics like PASS/FAIL where every change that doesn't PASS is rejected. It's too easy to "wedge" a system like this, where project X's change can't go through without project Y being able to handle the change, and you end up introducing multiple code paths in both projects on a solely temporary basis just in order to keep CI green and happy. Rather than having a zero-tolerance CI failure policy, developers should be allowed to break CI temporarily, so long as they fix it in a timely manner (within an hour or two). Per-developer breakage metrics, to the extent they are needed, should not be in terms of breakage counts but instead breakage durations. That is, outside of production, it's fine to break stuff and quickly fix it, so long as you don't leave it broken. The big problems are where domain-siloed developers break a zero-tolerance policy because it was necessary to relax it "temporarily", and things stay broken because the policy stays relaxed and cannot be reinstated without sirens going off. Then restoring the CI policy is blocked for everyone by the one guy who knows the AIX quirks for debootstrap or whatever. Instead, breaking changes should be "allowed" where there is a window to fix the error and still move forward. Only when the window closes without a fix should the breaking change be rolled back (automatically). This line of thinking lends itself to formal, automated policy, but this depends first on judgment and cultural approval. As it happens, the best CI system I'm aware of for truly distributed, multiple project integration is OpenStack's Zuul: https://github.com/openstack-infra/zuul https://github.com/openstack-infra/zuul I'm not sure if it accommodates my prescription above, but if not, they probably have a better idea.

skybrian 12y ago
If both projects share a source control system, you can commit a change and update its downstream usages all at once, with no breakage even temporarily. However, this requires source control that scales company-wide. (Git doesn't qualify.)
- RickHull 12y ago
  Yeah, git-repo-per-project was an unspoken assumption. Also, "upstream and downstream" may not suffice analytically, as many interdependencies in this model do not satisfy DAG, even if they ~should~. Agile projects aren't often built so much as grown.
  - jaggederest 12y ago
    I think the problem you're assuming here is that dependencies are automatically updated to use the latest version. That sounds like pretty strong coupling in a way that means other problems are lurking somewhere. Team A should be able to develop without having to consult Team B constantly. That means you have to be mature about deprecating things before you just remove them, but I think that's what the whole article is about.
    - skybrian 12y ago
      Yes, there are all the usual ways of deprecating things, migrating the callers, and then finally removing the deprecated methods. This is rather tedious and people tend not to do it for small cleanups like renames. If you can do it in one commit, you can work faster and hopefully get cleaner code. Of course even if you have atomic source control commits, you don't have atomic deployment so there's still migration to take care of, but it works fine for in-process API's.
  - Raphael 12y ago
    Of course, Facebook is supposedly one 50GB repo.
- jaggederest 12y ago
  Not sure why you're bashing on Git here, it certainly does scale to and handle this use case pretty effectively. This is precisely the problem that submodules solve, much maligned though they are. It's essentially a dependency problem, and we have good answers for those. Edit: Also I am fairly certain that the kernal experiences these kinds of issues, and is the flagship user of Git.
antimagic 12y ago
I, uh, don't understand. No sane environment commits straight to prod. You stage to an integration environment. The integration environment can break, no problems. But you only migrate staged dev from the integration environment to prod when all the test pass. Is that not how everyone does things???
- zimpenfish 12y ago
  I know at least one person working somewhere that has committing to TRUNK allegedly going straight to prod. Thankfully I've managed to avoid such mad places (more by luck than judgement, mind.)
- gknoy 12y ago
  Github does it differently. They have "prod" stuff that everyone sees, and "candidate" things which are ALSO deployed from the same codebase (with different enabling flags), and then can do long-term tests (and smaller pull requests) to verify that the New Way (candidate) behaves identically to the Old Way. Slide 35 [0] of this presentation actually starts the discussion of this exact thing, though Zach has talked about it before. Later, he shows a chart showing the differences as the code in the parallel branch changed over about five hours. (Wow, that's some fast iteration.) 0: https://speakerdeck.com/holman/move-fast-and-break-nothing?slide=35 https://speakerdeck.com/holman/move-fast-and-break-nothing?s...
- RickHull 12y ago
  Yes, what you describe is not at all at odds with what I described. If it wasn't clear, I was describing only developer interaction with integration testing per a CI system, with nothing to say about deploying to production other than the token "don't break users" in the first sentence.

skybrian 12y ago

If both projects share a source control system, you can commit a change and update its downstream usages all at once, with no breakage even temporarily. However, this requires source control that scales company-wide. (Git doesn't qualify.)

RickHull 12y ago
Yeah, git-repo-per-project was an unspoken assumption. Also, "upstream and downstream" may not suffice analytically, as many interdependencies in this model do not satisfy DAG, even if they ~should~. Agile projects aren't often built so much as grown.
- jaggederest 12y ago
  I think the problem you're assuming here is that dependencies are automatically updated to use the latest version. That sounds like pretty strong coupling in a way that means other problems are lurking somewhere. Team A should be able to develop without having to consult Team B constantly. That means you have to be mature about deprecating things before you just remove them, but I think that's what the whole article is about.
  - skybrian 12y ago
    Yes, there are all the usual ways of deprecating things, migrating the callers, and then finally removing the deprecated methods. This is rather tedious and people tend not to do it for small cleanups like renames. If you can do it in one commit, you can work faster and hopefully get cleaner code. Of course even if you have atomic source control commits, you don't have atomic deployment so there's still migration to take care of, but it works fine for in-process API's.
- Raphael 12y ago
  Of course, Facebook is supposedly one 50GB repo.
jaggederest 12y ago
Not sure why you're bashing on Git here, it certainly does scale to and handle this use case pretty effectively. This is precisely the problem that submodules solve, much maligned though they are. It's essentially a dependency problem, and we have good answers for those. Edit: Also I am fairly certain that the kernal experiences these kinds of issues, and is the flagship user of Git.

antimagic 12y ago

I, uh, don't understand. No sane environment commits straight to prod. You stage to an integration environment. The integration environment can break, no problems. But you only migrate staged dev from the integration environment to prod when all the test pass. Is that not how everyone does things???

zimpenfish 12y ago
I know at least one person working somewhere that has committing to TRUNK allegedly going straight to prod. Thankfully I've managed to avoid such mad places (more by luck than judgement, mind.)
gknoy 12y ago
Github does it differently. They have "prod" stuff that everyone sees, and "candidate" things which are ALSO deployed from the same codebase (with different enabling flags), and then can do long-term tests (and smaller pull requests) to verify that the New Way (candidate) behaves identically to the Old Way. Slide 35 [0] of this presentation actually starts the discussion of this exact thing, though Zach has talked about it before. Later, he shows a chart showing the differences as the code in the parallel branch changed over about five hours. (Wow, that's some fast iteration.) 0: https://speakerdeck.com/holman/move-fast-and-break-nothing?slide=35 https://speakerdeck.com/holman/move-fast-and-break-nothing?s...
RickHull 12y ago
Yes, what you describe is not at all at odds with what I described. If it wasn't clear, I was describing only developer interaction with integration testing per a CI system, with nothing to say about deploying to production other than the token "don't break users" in the first sentence.