3 ms·

I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting r

• yuliyp 4y ago
  Developer 1 fetches code, changes flag A. Rebuilds config. Developer 2 fetches code, changes flag B. Rebuilds config. Developer 1 deploys built config. Developer 2 deploys built config, inadvertently reverts developer 1's changes.
  • wstuartcl 4y ago
    also can happen when your deploy process has two flows for revert a forward movement revert (where new bits and head are committed fixing the items that needed to be reverted) and a "previous head" revert which just goes back one revision in the rcs (or tagged version). Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits). Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue. I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)
  • crvdgc 4y ago
    Sounds like a racing condition. A lock (algorithmical or just through communication) should have been used.
    • trhway 4y ago
      Lock is another huge failure mode.