5 ms·

In people' experience(s) how likely is it that a company can salvage a situation like this? I'm starting to experience this at work and I've been thinking about

by hacknat 13y ago

In people' experience(s) how likely is it that a company can salvage a situation like this? I'm starting to experience this at work and I've been thinking about leaving (for other reasons than just this), and I wonder if anyone has actually seen a turnaround in an engineering team's culture.

bdehaaff 13y ago
I have seen a turnaround, but it usually takes significant change. And typically that means management change. How long have you been experiencing the problem? The longer it goes on the harder it is to escape.
- JimboOmega 13y ago
  I was at a company where my entire (too large) reporting change turned over, and yet we were still displaying all the symptoms. It was horribly demotivating to always be blocked, and to have even the simplest of fixes wait months to get deployed. In our case, the biggest problem seemed to be siloing. Developers were not allowed to touch ANY systems, even the QA and Dev environments. If we wanted to deploy a fix for QA to verify, and the SysAdmin team was busy, it simply didn't happen - often for weeks. The same was true of database problems - if there was even a small flaw in the SQL, I had to sit around and wait for them to come up with something new, put it through our system, and then I could copy and paste it. The problem was it had been going on for far too long, and was too entrenched. Any new manager who got hired just slid into the current "way we do things" - there was too much momentum. The various fixes that were attempted were all either counterproductive or pointless. The most common was to "slow down" the process to reduce the number of bugs that got pushed out, which was maddening. There would be other fads (A Kanban board nobody used was one of them)... we also tried to switch to estimating time using fibonacci numbers for no reason anybody could figure out. We also had sprints that the manager-du-jour would try to revive. They were never meaningful. Instead, when the sprint was over, we'd go through the tasks assigned to us and bump them to the next sprint in Jira. We went through a couple different managers (and VPs of Engineering/CTOs/etc). The biggest practical difference between them was how much they cared about the trappings of work - that is, how much they cared that developers were at their desks at time X, how angry they got when they saw a non-work website open when they walked by, etc. None seemed interested in fixing the deeper problems. None seemed able to change the fundamental cultural preference for silos, gates, and an ever slowing pace. None seemed incredibly concerned that a developer who spots a bug and wants to fix it would be embarking on a journey that would involve almost a dozen people, countless "gates", and probably months of time. (First QA needs to verify the bug, then requirements needs to specify what fix they need, then DB team needs to make any SQL changes... and of course managers need to specify priority...). The company was ultimately sold off at a steep discount.
  - bdehaaff 13y ago
    I love the honesty of this story and I wish it were told more often. I just kept nodding. You did touch on one concept that I almost added to the piece... "we always do it like that". If this post continues to grow in popularity I think I will add an "after thought" with additional insights like what you shared. Thanks.
    - JimboOmega 13y ago
      Another problem (symptom or cause? I'm not sure) was how we prioritized things. When I started, there was P1, which was only used if production was effectively down, P2, which meant it had to go in the next release, and P3 and on down. Of course everybody had issues they really wanted fixed - more than we could do. People realized that P3 bugs stood a good chance of missing the release - and might not get fixed for the better part of a year if they did. So almost all bugs started becoming P2. This obviously didn't work, so to supplement that, we had a system of "Shadow priority" - that is, there'd be P2 tickets that "important" managers really cared about and thus had to get done. This got a bit tiring because of the number of managers and their shifting priorities. I remember a spreadsheet where the rows highlighted in green were "true" P2s. Thus we created a new priority field, "relative importance", because everything was P2... And guess what, the process repeated; everything became high relative importance. By the time I left - and I did send an email complaining about this to my boss - I think we were up to 5 different places to look for priority, including a spreadsheet that would be shown at meetings but not circulated. The logic for figuring out priority was incredibly complicated, since certain types of priority overrode others. There might be P3s that were more important than P2s because they were the pet issue of a certain manager, for instance. There was a separate notion of "QA Priority" and "Manager Priority" (manager priority, on the spreadsheet, even had P0) Of course, what developers thought never was a big consideration. For the longest time, I just thought it was the inevitable result of bureaucracy, and yet, here I sit at a job where, for almost 5 months, we've had a backlog that we all are involved in creating; the priorities and rationale are explained to us... and best of all, the most important things are at the top! That's all I need to do, look at the top of the backlog. And somehow I spend almost no time waiting on requirements, and none waiting on "database" (which isn't even a team). Fixes I make get reviewed a week later at the absolute latest. I'm not sure why, exactly, this is. Clearly our situation is more desirable - so why did the old one happen at all? I don't feel like any of the teams at my old company were incompetent, evil, or stubborn. It was just the way we did things. (:With the possible exception of the sworn enemies of the dev team, ProdOps née SOC AKA the SysAdmins. But then, the inter-team rivalry was more of a symptom than an actual cause.)
Xylakant 13y ago
From what I've seen it depends on how long the situation has been lingering on. Teams have ups and downs, phases where things just hum along and phases where matters are more difficult. If it's just a phase and a fairly recent development it's possible to turn things around, especially if there's still people around that are motivated and want to change things. A clear diagnosis of what's wrong would be the first step. However, it gets harder to change things the longer the problem has been around - everybody retired to his own little world, blames the rest of the team for failures, too much blood shed between the individuals. It's very hard to climb back up from this hole.
- bdehaaff 13y ago
  Well said. I totally agree. Sometimes it's just a blip and sometimes there is distrust and structural dysfunction that is going to need a lot of work to fix.
RogerL 13y ago
I have seen a turnaround, a dramatic one. I forget the number, but this was for something in the range of a $60MM contract. I will not identify the company or product for various reasons. How was the turnaround accomplished? By firing everyone in charge, and replacing them with people that could actually make reasonable decisions. Some lower level people were also let go; we had some people without the chops to do the job. But that was not the main problem. The problem was consistent management focused entirely on 'process' without regard to what we were trying to accomplish. In the form of "MIL-STD-1234 says we have to produce documentation in the form of X. Therefore, produce X". Documentation does not result in a great product. Documentation can only, wait for it, document what was done. That was just one sliver of it, but entirely representative. Methodology X, process Y, 5 levels of sign offs all focused on what color ink you used or if your document's headers use the right capitalization. Just shuffling requirements along an endless document chain, without actually trying to figure out how to meet those requirements. A really smart systems engineer who had the bosses ear was able to say 'they will never make it', he eventually listened, she was put in charge, our best engineer was finally put in charge of the SW, and from there we recovered. In the interim we hired a bunch of outside consultants to come in. I don't see that they did anything of value. Lot's of interviewing, lots of 'do x, y, z'; meanwhile the people that had successfully produced this sort of project before were still being ignored. That is one story. The problem was mid-level management, and only once they were all fired (actually, there were several rounds of getting rid of person A, and replacing them with an exact clone, and of course that person had identical results) did we recover. This probably doesn't apply to any place where the problem is not middle management (say, a team full of cowboys, or a PM that sends you off on wild goose chases, or whatever).
- bdehaaff 13y ago
  Exactly. Put the folks who can get the work done in charge of actually getting the work done. Give them the responsibility and allow their courage to flourish 9and remove barriers when needed). Celebrate their accomplishments.
  - zobzu 13y ago
    yep - full management replacement is often the only thing that can help, and boards generally dont have that level understanding. plus the management wanna stay in place and advance personal careers/salary without effort and thys deflect issues and pressure the board. it even happens in non profits.
programminggeek 13y ago
It seems like from what people are saying you either have to get the people in charge to change or change the people in charge.
- bdehaaff 13y ago
  Yep. Big change is needed one way or another because the ground is approaching faster than most teams in the death spiral think.

by hacknat 13y ago

bdehaaff 13y ago

I have seen a turnaround, but it usually takes significant change. And typically that means management change. How long have you been experiencing the problem? The longer it goes on the harder it is to escape.

JimboOmega 13y ago
I was at a company where my entire (too large) reporting change turned over, and yet we were still displaying all the symptoms. It was horribly demotivating to always be blocked, and to have even the simplest of fixes wait months to get deployed. In our case, the biggest problem seemed to be siloing. Developers were not allowed to touch ANY systems, even the QA and Dev environments. If we wanted to deploy a fix for QA to verify, and the SysAdmin team was busy, it simply didn't happen - often for weeks. The same was true of database problems - if there was even a small flaw in the SQL, I had to sit around and wait for them to come up with something new, put it through our system, and then I could copy and paste it. The problem was it had been going on for far too long, and was too entrenched. Any new manager who got hired just slid into the current "way we do things" - there was too much momentum. The various fixes that were attempted were all either counterproductive or pointless. The most common was to "slow down" the process to reduce the number of bugs that got pushed out, which was maddening. There would be other fads (A Kanban board nobody used was one of them)... we also tried to switch to estimating time using fibonacci numbers for no reason anybody could figure out. We also had sprints that the manager-du-jour would try to revive. They were never meaningful. Instead, when the sprint was over, we'd go through the tasks assigned to us and bump them to the next sprint in Jira. We went through a couple different managers (and VPs of Engineering/CTOs/etc). The biggest practical difference between them was how much they cared about the trappings of work - that is, how much they cared that developers were at their desks at time X, how angry they got when they saw a non-work website open when they walked by, etc. None seemed interested in fixing the deeper problems. None seemed able to change the fundamental cultural preference for silos, gates, and an ever slowing pace. None seemed incredibly concerned that a developer who spots a bug and wants to fix it would be embarking on a journey that would involve almost a dozen people, countless "gates", and probably months of time. (First QA needs to verify the bug, then requirements needs to specify what fix they need, then DB team needs to make any SQL changes... and of course managers need to specify priority...). The company was ultimately sold off at a steep discount.
- bdehaaff 13y ago
  I love the honesty of this story and I wish it were told more often. I just kept nodding. You did touch on one concept that I almost added to the piece... "we always do it like that". If this post continues to grow in popularity I think I will add an "after thought" with additional insights like what you shared. Thanks.
  - JimboOmega 13y ago
    Another problem (symptom or cause? I'm not sure) was how we prioritized things. When I started, there was P1, which was only used if production was effectively down, P2, which meant it had to go in the next release, and P3 and on down. Of course everybody had issues they really wanted fixed - more than we could do. People realized that P3 bugs stood a good chance of missing the release - and might not get fixed for the better part of a year if they did. So almost all bugs started becoming P2. This obviously didn't work, so to supplement that, we had a system of "Shadow priority" - that is, there'd be P2 tickets that "important" managers really cared about and thus had to get done. This got a bit tiring because of the number of managers and their shifting priorities. I remember a spreadsheet where the rows highlighted in green were "true" P2s. Thus we created a new priority field, "relative importance", because everything was P2... And guess what, the process repeated; everything became high relative importance. By the time I left - and I did send an email complaining about this to my boss - I think we were up to 5 different places to look for priority, including a spreadsheet that would be shown at meetings but not circulated. The logic for figuring out priority was incredibly complicated, since certain types of priority overrode others. There might be P3s that were more important than P2s because they were the pet issue of a certain manager, for instance. There was a separate notion of "QA Priority" and "Manager Priority" (manager priority, on the spreadsheet, even had P0) Of course, what developers thought never was a big consideration. For the longest time, I just thought it was the inevitable result of bureaucracy, and yet, here I sit at a job where, for almost 5 months, we've had a backlog that we all are involved in creating; the priorities and rationale are explained to us... and best of all, the most important things are at the top! That's all I need to do, look at the top of the backlog. And somehow I spend almost no time waiting on requirements, and none waiting on "database" (which isn't even a team). Fixes I make get reviewed a week later at the absolute latest. I'm not sure why, exactly, this is. Clearly our situation is more desirable - so why did the old one happen at all? I don't feel like any of the teams at my old company were incompetent, evil, or stubborn. It was just the way we did things. (:With the possible exception of the sworn enemies of the dev team, ProdOps née SOC AKA the SysAdmins. But then, the inter-team rivalry was more of a symptom than an actual cause.)

Xylakant 13y ago

From what I've seen it depends on how long the situation has been lingering on. Teams have ups and downs, phases where things just hum along and phases where matters are more difficult. If it's just a phase and a fairly recent development it's possible to turn things around, especially if there's still people around that are motivated and want to change things. A clear diagnosis of what's wrong would be the first step. However, it gets harder to change things the longer the problem has been around - everybody retired to his own little world, blames the rest of the team for failures, too much blood shed between the individuals. It's very hard to climb back up from this hole.

bdehaaff 13y ago
Well said. I totally agree. Sometimes it's just a blip and sometimes there is distrust and structural dysfunction that is going to need a lot of work to fix.

RogerL 13y ago

I have seen a turnaround, a dramatic one. I forget the number, but this was for something in the range of a $60MM contract. I will not identify the company or product for various reasons. How was the turnaround accomplished? By firing everyone in charge, and replacing them with people that could actually make reasonable decisions. Some lower level people were also let go; we had some people without the chops to do the job. But that was not the main problem. The problem was consistent management focused entirely on 'process' without regard to what we were trying to accomplish. In the form of "MIL-STD-1234 says we have to produce documentation in the form of X. Therefore, produce X". Documentation does not result in a great product. Documentation can only, wait for it, document what was done. That was just one sliver of it, but entirely representative. Methodology X, process Y, 5 levels of sign offs all focused on what color ink you used or if your document's headers use the right capitalization. Just shuffling requirements along an endless document chain, without actually trying to figure out how to meet those requirements. A really smart systems engineer who had the bosses ear was able to say 'they will never make it', he eventually listened, she was put in charge, our best engineer was finally put in charge of the SW, and from there we recovered. In the interim we hired a bunch of outside consultants to come in. I don't see that they did anything of value. Lot's of interviewing, lots of 'do x, y, z'; meanwhile the people that had successfully produced this sort of project before were still being ignored. That is one story. The problem was mid-level management, and only once they were all fired (actually, there were several rounds of getting rid of person A, and replacing them with an exact clone, and of course that person had identical results) did we recover. This probably doesn't apply to any place where the problem is not middle management (say, a team full of cowboys, or a PM that sends you off on wild goose chases, or whatever).

bdehaaff 13y ago
Exactly. Put the folks who can get the work done in charge of actually getting the work done. Give them the responsibility and allow their courage to flourish 9and remove barriers when needed). Celebrate their accomplishments.
- zobzu 13y ago
  yep - full management replacement is often the only thing that can help, and boards generally dont have that level understanding. plus the management wanna stay in place and advance personal careers/salary without effort and thys deflect issues and pressure the board. it even happens in non profits.

programminggeek 13y ago

It seems like from what people are saying you either have to get the people in charge to change or change the people in charge.

bdehaaff 13y ago
Yep. Big change is needed one way or another because the ground is approaching faster than most teams in the death spiral think.