4 ms·

How would CI help in this case? It isn't even software bug, it's a process issue - they had old code running on one of out of 8 servers. The monitoring was trig

How would CI help in this case? It isn't even software bug, it's a process issue - they had old code running on one of out of 8 servers. The monitoring was triggered, but no action was taken.

malux85 8y ago
Maybe I was using the wrong terminology, but when I said "CI" I meant that to include automated, atomic deployments. In the last 16 years I've worked in software the last 10 or so of that has not included manually copying files to production
- unreal37 8y ago
  Those must be new businesses / startups. The world is full of companies more than 20 years old.
UweSchmidt 8y ago
The bug report describes how they reuse a flag for new code that triggered old code. "CI" would generally help keeping things tidy and transparent. It would be easier to find out what branches or flags or parameters already exist, which ones are not relevant any more; someone could rename them with a prefix "OLD_" or otherwise clean up, so that you typically pick from a list of options instead of setting a flag by accidentially copying it from one script to the other. It is also easier to have error visualization plugins, or to look around previous deployments or test runs and get a pretty good idea if any new errors have shown up, even if you haven't analyzed the old ones.
werbel 8y ago
I disagree that it isn't a software bug. "The new RLP code also repurposed a flag" - this is the moment when terrible software development idea was executed that resulted in all of the mess. Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything. What I observe more often is something like this though: 1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag. 2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now. 3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B. 4. Turns out for use case A we need to do something more but things should stay the same for B. 5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B. If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases. edit: formatting
- reificator 8y ago
  > If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. A hard lesson to learn, and a hard rule to push for with others who have not yet learned. Imagine what our species could do if experience were directly and easily transferable...
  - werbel 8y ago
    Hah exactly :) Same goes for functions, classes, React components, DB tables and everything else. Just model it as close as possible to the real world. The world doesn't really change that often. What does is how we interpret and behave within it (logic/behaviour/appearance on top). If you have a Label and Subheader in your app, create separate components for them. It doesn't matter that they look exactly the same now. Those are two separate things and I guarantee you more likely than not at some point they will differ. My rule of thumb is: If it's something I can somehow name as an entity (especially in product and not tech talk) it deserves to be its own entity.
    - reificator 8y ago
      It's funny though, because my experience has led me to the exact opposite approach. Modeling based on real world understanding has been very fragile and error prone, and instead modeling as data and systems that operate on that data has been very fruitful.
- unseenbastiat 8y ago
  The urge to use the same flag for both B and A is probably motivated by a desire either to clean up the code by not using redundant flags. Or perhaps it was implicitly part of a forward-thinking (in theory, well intentioned) design to avoid separate flag B in the future. Either way, it shows some attempt at longer-term thinking undone by short term implementation done improperly. That’s a microcosm of this story as a whole which gives this incident a fractal quality.
- 8y ago
bluGill 8y ago
One of the tests in my CI pipeline is verifying that my new code works with previously released code. It gets painful when these tests fail, but it is a valid configuration so I have to work around bugs in the old code from time to time.

How would CI help in this case? It isn't even software bug, it's a process issue - they had old code running on one of out of 8 servers. The monitoring was triggered, but no action was taken.

malux85 8y ago

Maybe I was using the wrong terminology, but when I said "CI" I meant that to include automated, atomic deployments. In the last 16 years I've worked in software the last 10 or so of that has not included manually copying files to production

unreal37 8y ago
Those must be new businesses / startups. The world is full of companies more than 20 years old.

UweSchmidt 8y ago

The bug report describes how they reuse a flag for new code that triggered old code. "CI" would generally help keeping things tidy and transparent. It would be easier to find out what branches or flags or parameters already exist, which ones are not relevant any more; someone could rename them with a prefix "OLD_" or otherwise clean up, so that you typically pick from a list of options instead of setting a flag by accidentially copying it from one script to the other. It is also easier to have error visualization plugins, or to look around previous deployments or test runs and get a pretty good idea if any new errors have shown up, even if you haven't analyzed the old ones.

werbel 8y ago

I disagree that it isn't a software bug. "The new RLP code also repurposed a flag" - this is the moment when terrible software development idea was executed that resulted in all of the mess. Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything. What I observe more often is something like this though: 1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag. 2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now. 3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B. 4. Turns out for use case A we need to do something more but things should stay the same for B. 5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B. If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases. edit: formatting

reificator 8y ago
> If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. A hard lesson to learn, and a hard rule to push for with others who have not yet learned. Imagine what our species could do if experience were directly and easily transferable...
- werbel 8y ago
  Hah exactly :) Same goes for functions, classes, React components, DB tables and everything else. Just model it as close as possible to the real world. The world doesn't really change that often. What does is how we interpret and behave within it (logic/behaviour/appearance on top). If you have a Label and Subheader in your app, create separate components for them. It doesn't matter that they look exactly the same now. Those are two separate things and I guarantee you more likely than not at some point they will differ. My rule of thumb is: If it's something I can somehow name as an entity (especially in product and not tech talk) it deserves to be its own entity.
  - reificator 8y ago
    It's funny though, because my experience has led me to the exact opposite approach. Modeling based on real world understanding has been very fragile and error prone, and instead modeling as data and systems that operate on that data has been very fruitful.
unseenbastiat 8y ago
The urge to use the same flag for both B and A is probably motivated by a desire either to clean up the code by not using redundant flags. Or perhaps it was implicitly part of a forward-thinking (in theory, well intentioned) design to avoid separate flag B in the future. Either way, it shows some attempt at longer-term thinking undone by short term implementation done improperly. That’s a microcosm of this story as a whole which gives this incident a fractal quality.
8y ago

bluGill 8y ago

One of the tests in my CI pipeline is verifying that my new code works with previously released code. It gets painful when these tests fail, but it is a valid configuration so I have to work around bugs in the old code from time to time.