6 ms·

What baffles me is how they got his far into operations with such dreadful practices, 100-200k could have got them a really solid CI pipeline with rollbacks, mo

What baffles me is how they got his far into operations with such dreadful practices, 100-200k could have got them a really solid CI pipeline with rollbacks, monitoring, testing etc, But spend 200,000 on managing 460,000,000? No way!

free652 8y ago
How would CI help in this case? It isn't even software bug, it's a process issue - they had old code running on one of out of 8 servers. The monitoring was triggered, but no action was taken.
- malux85 8y ago
  Maybe I was using the wrong terminology, but when I said "CI" I meant that to include automated, atomic deployments. In the last 16 years I've worked in software the last 10 or so of that has not included manually copying files to production
  - unreal37 8y ago
    Those must be new businesses / startups. The world is full of companies more than 20 years old.
- UweSchmidt 8y ago
  The bug report describes how they reuse a flag for new code that triggered old code. "CI" would generally help keeping things tidy and transparent. It would be easier to find out what branches or flags or parameters already exist, which ones are not relevant any more; someone could rename them with a prefix "OLD_" or otherwise clean up, so that you typically pick from a list of options instead of setting a flag by accidentially copying it from one script to the other. It is also easier to have error visualization plugins, or to look around previous deployments or test runs and get a pretty good idea if any new errors have shown up, even if you haven't analyzed the old ones.
- werbel 8y ago
  I disagree that it isn't a software bug. "The new RLP code also repurposed a flag" - this is the moment when terrible software development idea was executed that resulted in all of the mess. Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything. What I observe more often is something like this though: 1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag. 2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now. 3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B. 4. Turns out for use case A we need to do something more but things should stay the same for B. 5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B. If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases. edit: formatting
  - reificator 8y ago
    > If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. A hard lesson to learn, and a hard rule to push for with others who have not yet learned. Imagine what our species could do if experience were directly and easily transferable...
    - werbel 8y ago
      Hah exactly :) Same goes for functions, classes, React components, DB tables and everything else. Just model it as close as possible to the real world. The world doesn't really change that often. What does is how we interpret and behave within it (logic/behaviour/appearance on top). If you have a Label and Subheader in your app, create separate components for them. It doesn't matter that they look exactly the same now. Those are two separate things and I guarantee you more likely than not at some point they will differ. My rule of thumb is: If it's something I can somehow name as an entity (especially in product and not tech talk) it deserves to be its own entity.
- bluGill 8y ago
  One of the tests in my CI pipeline is verifying that my new code works with previously released code. It gets painful when these tests fail, but it is a valid configuration so I have to work around bugs in the old code from time to time.
xkcd-sucks 8y ago
One problem is that business-focused people seem to think of tech debt as an easily determined quantity, when it behaves more like a complex derivative than a simple financial instruments like a loan
- hotsauceror 8y ago
  That's assuming that business-focused people have any interest in determining the amount of technical debt at all. Generally they're content to let other teams pay the debt, and many of them are quite happy to complain when it doesn't get paid fast enough.
zarkov99 8y ago
Firms like this have spent decades driven by a short term, "get it done now, future entropy be damned" mentality, partially because market opportunities are fleeting and partially because the principals usually come from a trading background and have very little respect or understanding for technology. The norm are gigantic systems accruing decades of technical debt,very large swats of dead code and a refusal (and sometimes plain inability) to address software engineering issues. The firms that survive are the ones that realize (and staff accordingly) they are tech firms that happen to trade rather then other way around.
bsder 8y ago
"You're not taking that $200K out of my budget." While something may be a rounding error to the company, you can't get someone at the correct political level where it's a rounding error to pay attention.

free652 8y ago

How would CI help in this case? It isn't even software bug, it's a process issue - they had old code running on one of out of 8 servers. The monitoring was triggered, but no action was taken.

malux85 8y ago
Maybe I was using the wrong terminology, but when I said "CI" I meant that to include automated, atomic deployments. In the last 16 years I've worked in software the last 10 or so of that has not included manually copying files to production
- unreal37 8y ago
  Those must be new businesses / startups. The world is full of companies more than 20 years old.
UweSchmidt 8y ago
The bug report describes how they reuse a flag for new code that triggered old code. "CI" would generally help keeping things tidy and transparent. It would be easier to find out what branches or flags or parameters already exist, which ones are not relevant any more; someone could rename them with a prefix "OLD_" or otherwise clean up, so that you typically pick from a list of options instead of setting a flag by accidentially copying it from one script to the other. It is also easier to have error visualization plugins, or to look around previous deployments or test runs and get a pretty good idea if any new errors have shown up, even if you haven't analyzed the old ones.
werbel 8y ago
I disagree that it isn't a software bug. "The new RLP code also repurposed a flag" - this is the moment when terrible software development idea was executed that resulted in all of the mess. Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything. What I observe more often is something like this though: 1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag. 2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now. 3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B. 4. Turns out for use case A we need to do something more but things should stay the same for B. 5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B. If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases. edit: formatting
- reificator 8y ago
  > If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now. A hard lesson to learn, and a hard rule to push for with others who have not yet learned. Imagine what our species could do if experience were directly and easily transferable...
  - werbel 8y ago
    Hah exactly :) Same goes for functions, classes, React components, DB tables and everything else. Just model it as close as possible to the real world. The world doesn't really change that often. What does is how we interpret and behave within it (logic/behaviour/appearance on top). If you have a Label and Subheader in your app, create separate components for them. It doesn't matter that they look exactly the same now. Those are two separate things and I guarantee you more likely than not at some point they will differ. My rule of thumb is: If it's something I can somehow name as an entity (especially in product and not tech talk) it deserves to be its own entity.
bluGill 8y ago
One of the tests in my CI pipeline is verifying that my new code works with previously released code. It gets painful when these tests fail, but it is a valid configuration so I have to work around bugs in the old code from time to time.

xkcd-sucks 8y ago

One problem is that business-focused people seem to think of tech debt as an easily determined quantity, when it behaves more like a complex derivative than a simple financial instruments like a loan

hotsauceror 8y ago
That's assuming that business-focused people have any interest in determining the amount of technical debt at all. Generally they're content to let other teams pay the debt, and many of them are quite happy to complain when it doesn't get paid fast enough.

zarkov99 8y ago

Firms like this have spent decades driven by a short term, "get it done now, future entropy be damned" mentality, partially because market opportunities are fleeting and partially because the principals usually come from a trading background and have very little respect or understanding for technology. The norm are gigantic systems accruing decades of technical debt,very large swats of dead code and a refusal (and sometimes plain inability) to address software engineering issues. The firms that survive are the ones that realize (and staff accordingly) they are tech firms that happen to trade rather then other way around.

bsder 8y ago

"You're not taking that $200K out of my budget." While something may be a rounding error to the company, you can't get someone at the correct political level where it's a rounding error to pay attention.