8 ms·
Kanban can not save you from the engineering death spiral
- hacknat 13y agoIn people' experience(s) how likely is it that a company can salvage a situation like this? I'm starting to experience this at work and I've been thinking about leaving (for other reasons than just this), and I wonder if anyone has actually seen a turnaround in an engineering team's culture.
- bdehaaff 13y agoI have seen a turnaround, but it usually takes significant change. And typically that means management change. How long have you been experiencing the problem? The longer it goes on the harder it is to escape.
- JimboOmega 13y agoI was at a company where my entire (too large) reporting change turned over, and yet we were still displaying all the symptoms. It was horribly demotivating to always be blocked, and to have even the simplest of fixes wait months to get deployed. In our case, the biggest problem seemed to be siloing. Developers were not allowed to touch ANY systems, even the QA and Dev environments. If we wanted to deploy a fix for QA to verify, and the SysAdmin team was busy, it simply didn't happen - often for weeks. The same was true of database problems - if there was even a small flaw in the SQL, I had to sit around and wait for them to come up with something new, put it through our system, and then I could copy and paste it. The problem was it had been going on for far too long, and was too entrenched. Any new manager who got hired just slid into the current "way we do things" - there was too much momentum. The various fixes that were attempted were all either counterproductive or pointless. The most common was to "slow down" the process to reduce the number of bugs that got pushed out, which was maddening. There would be other fads (A Kanban board nobody used was one of them)... we also tried to switch to estimating time using fibonacci numbers for no reason anybody could figure out. We also had sprints that the manager-du-jour would try to revive. They were never meaningful. Instead, when the sprint was over, we'd go through the tasks assigned to us and bump them to the next sprint in Jira. We went through a couple different managers (and VPs of Engineering/CTOs/etc). The biggest practical difference between them was how much they cared about the trappings of work - that is, how much they cared that developers were at their desks at time X, how angry they got when they saw a non-work website open when they walked by, etc. None seemed interested in fixing the deeper problems. None seemed able to change the fundamental cultural preference for silos, gates, and an ever slowing pace. None seemed incredibly concerned that a developer who spots a bug and wants to fix it would be embarking on a journey that would involve almost a dozen people, countless "gates", and probably months of time. (First QA needs to verify the bug, then requirements needs to specify what fix they need, then DB team needs to make any SQL changes... and of course managers need to specify priority...). The company was ultimately sold off at a steep discount.
- bdehaaff 13y agoI love the honesty of this story and I wish it were told more often. I just kept nodding. You did touch on one concept that I almost added to the piece... "we always do it like that". If this post continues to grow in popularity I think I will add an "after thought" with additional insights like what you shared. Thanks.
- JimboOmega 13y agoAnother problem (symptom or cause? I'm not sure) was how we prioritized things. When I started, there was P1, which was only used if production was effectively down, P2, which meant it had to go in the next release, and P3 and on down. Of course everybody had issues they really wanted fixed - more than we could do. People realized that P3 bugs stood a good chance of missing the release - and might not get fixed for the better part of a year if they did. So almost all bugs started becoming P2. This obviously didn't work, so to supplement that, we had a system of "Shadow priority" - that is, there'd be P2 tickets that "important" managers really cared about and thus had to get done. This got a bit tiring because of the number of managers and their shifting priorities. I remember a spreadsheet where the rows highlighted in green were "true" P2s. Thus we created a new priority field, "relative importance", because everything was P2... And guess what, the process repeated; everything became high relative importance. By the time I left - and I did send an email complaining about this to my boss - I think we were up to 5 different places to look for priority, including a spreadsheet that would be shown at meetings but not circulated. The logic for figuring out priority was incredibly complicated, since certain types of priority overrode others. There might be P3s that were more important than P2s because they were the pet issue of a certain manager, for instance. There was a separate notion of "QA Priority" and "Manager Priority" (manager priority, on the spreadsheet, even had P0) Of course, what developers thought never was a big consideration. For the longest time, I just thought it was the inevitable result of bureaucracy, and yet, here I sit at a job where, for almost 5 months, we've had a backlog that we all are involved in creating; the priorities and rationale are explained to us... and best of all, the most important things are at the top! That's all I need to do, look at the top of the backlog. And somehow I spend almost no time waiting on requirements, and none waiting on "database" (which isn't even a team). Fixes I make get reviewed a week later at the absolute latest. I'm not sure why, exactly, this is. Clearly our situation is more desirable - so why did the old one happen at all? I don't feel like any of the teams at my old company were incompetent, evil, or stubborn. It was just the way we did things. (:With the possible exception of the sworn enemies of the dev team, ProdOps née SOC AKA the SysAdmins. But then, the inter-team rivalry was more of a symptom than an actual cause.)
- Xylakant 13y agoFrom what I've seen it depends on how long the situation has been lingering on. Teams have ups and downs, phases where things just hum along and phases where matters are more difficult. If it's just a phase and a fairly recent development it's possible to turn things around, especially if there's still people around that are motivated and want to change things. A clear diagnosis of what's wrong would be the first step. However, it gets harder to change things the longer the problem has been around - everybody retired to his own little world, blames the rest of the team for failures, too much blood shed between the individuals. It's very hard to climb back up from this hole.
- bdehaaff 13y agoWell said. I totally agree. Sometimes it's just a blip and sometimes there is distrust and structural dysfunction that is going to need a lot of work to fix.
- RogerL 13y agoI have seen a turnaround, a dramatic one. I forget the number, but this was for something in the range of a $60MM contract. I will not identify the company or product for various reasons. How was the turnaround accomplished? By firing everyone in charge, and replacing them with people that could actually make reasonable decisions. Some lower level people were also let go; we had some people without the chops to do the job. But that was not the main problem. The problem was consistent management focused entirely on 'process' without regard to what we were trying to accomplish. In the form of "MIL-STD-1234 says we have to produce documentation in the form of X. Therefore, produce X". Documentation does not result in a great product. Documentation can only, wait for it, document what was done. That was just one sliver of it, but entirely representative. Methodology X, process Y, 5 levels of sign offs all focused on what color ink you used or if your document's headers use the right capitalization. Just shuffling requirements along an endless document chain, without actually trying to figure out how to meet those requirements. A really smart systems engineer who had the bosses ear was able to say 'they will never make it', he eventually listened, she was put in charge, our best engineer was finally put in charge of the SW, and from there we recovered. In the interim we hired a bunch of outside consultants to come in. I don't see that they did anything of value. Lot's of interviewing, lots of 'do x, y, z'; meanwhile the people that had successfully produced this sort of project before were still being ignored. That is one story. The problem was mid-level management, and only once they were all fired (actually, there were several rounds of getting rid of person A, and replacing them with an exact clone, and of course that person had identical results) did we recover. This probably doesn't apply to any place where the problem is not middle management (say, a team full of cowboys, or a PM that sends you off on wild goose chases, or whatever).
- bdehaaff 13y agoExactly. Put the folks who can get the work done in charge of actually getting the work done. Give them the responsibility and allow their courage to flourish 9and remove barriers when needed). Celebrate their accomplishments.
- zobzu 13y agoyep - full management replacement is often the only thing that can help, and boards generally dont have that level understanding. plus the management wanna stay in place and advance personal careers/salary without effort and thys deflect issues and pressure the board. it even happens in non profits.
- programminggeek 13y agoIt seems like from what people are saying you either have to get the people in charge to change or change the people in charge.
- bdehaaff 13y agoYep. Big change is needed one way or another because the ground is approaching faster than most teams in the death spiral think.
- pkananen 13y agoThe article has some good points, but I'm not really sure what it has to do with Kanban.
- bdehaaff 13y agoThanks. The point is that applying simple tools like Kanban or trying to "shock" the system by "becoming" agile typically does not work because these tools do not address the underlying issues.
- cjensen 13y agoKanban is the silver bullet of the day which sinking enterprises reach for. I've been a software engineer since 1989, and the entire time has been filled with process/method fads "which will fix everything." In 1989 it was Yourdon's institutionalized bureaucracy. Today it's Kanban. Doesn't make Kanban broken; it just means that it is the non-solution for today's desperate souls.
- bdehaaff 13y agoVery true. There is always some new "fix it." The problem is often that the engineering tools do not improve product management in the least and if the product strategy is bad and customer understanding is missing, no engineering team or tool can solve that.
- RogerL 13y agoBut "Aha" does solve it?
- bayesianhorse 13y agoOne problem with this article is the absence of any reasoning process. How do you know there is such a thing as a death spiral? How do you know these symptoms apply to the "death spiral" condition. How do you know your solutions actually alleviate the symptoms or abandon the condition?
- bdehaaff 13y agoPersonal experience after being an early employee or founder of six software companies and knowing or speaking directly with folks in hundreds of others. Simple experience and pattern recognition. Not science.
- VLM 13y agoWhatever metric you use for "working" goes up, and whatever metric you use for "accomplishing" goes down. Perhaps if you fix the problem those trends will reverse. Doesn't need to be hyper game-able beancounter make peoples pay depend on fudging the numbers. IN fact that will create negative creativity. You get better results if you write down and graph the departmental ratios of currently open bugs vs lines of git commits and for god sakes don't let anyone know or the gameification is on...
- bdehaaff 13y agoYes, I think you are saying that the focus needs to be on results. Not effort. I agree, but is that what you are highlighting with your comments?
- seiji 13y agoDeath spiral: When you keep creating products with no demand in the real world [1]. You create things, release things, nobody uses it, but by then, you're working on the next version nobody wants either. Death spiral: Working on components of a larger project with no hope of ever integrating all pieces into a coherent final release. Everything gets incrementally improved in isolation, but nobody can figure out how to cut a product out of all the pieces. [1]: Plenty of demand exists in the delusional minds of executives, PMs, and clueless employees though. It just doesn't match the real world.
- 3pt14159 13y agoThe article's own link to the authors product is broken.
- ecesena 13y agoA friend of mine, CTO at Elfo [1] (a 60-ppl Italian software basically-unknown company) is successfully using Kanban for software development since almost 2 years. Each team/project (6 to 10 developers) has it's own Kanban board, each person has no more than 1-2 tasks/phase and they focus on keeping the phases balanced, i.e. with the same amount of tasks. They're happy since its a light weight way to have a full picture of what's going on. His original notes (in Italian, sorry) [2]. [1] http://www.elfo.net http://www.elfo.net [2] http://www.robertocappelletti.com/2012/01/kanban/ http://www.robertocappelletti.com/2012/01/kanban/
- bdehaaff 13y agoGreat. Thanks for sharing this success story. It would be interesting to know if the engineering team is motivated and understands the why they are building what they are.
- ecesena 13y agoAs far as I got a lot. The main reason is that, with limited effort, you see your progress (tasks move). Moreover the management is really enthusiastic of the approach, so they're often showing to guests (like I was) and promoting the activity as something unique of "their guys", so I guess this also makes people proud of what they're doing...
- lmm 13y agoHuh? I'm pretty sure Kanban helps with all of these. Why are we building this feature? With a strict limit on work in progress, someone must have explicitly decided that it was more important than all the other features in your backlog. Sure, you can still disagree with their weighting, but at least it forces them to prioritize, rather than delaying the product with a kitchen sink of everyone's ideas. We can't release until quality is better? If you keep cards in progress until they're released (and you should), you very quickly reach a point where you have to release, or it's simply impossible to do any more work. In fact this is pretty much the core problem Kanban was meant to solve. I'm blocked? A blocked card is filling up a valuable work-in-progress slot, and a manager should notice very quickly. If particular issues keep blocking several cards, that means the manager is seeing those same issues every day; sooner or later they should take notice.
- bdehaaff 13y agoKanban does nothing to solve the product strategy and market fit questions and is often applied as a fix-all for much deeper rooted problems. I would also argue that Kanban has the tendency to turn engineers into widget builders and is demotivating. Here, take this tiny piece of work and when you finish it I will reward you with another. Keep your head down and just trust me.
- mprovost 13y agoI would say Kanban is the opposite of keeping your head down - you have to come up for air every time you finish something and priorities can change each time. Waterfall is the classic head-down approach, work on something for months/years and then release it and see if there is still a market for it. I think there are strong tendencies towards waterfall in many tech personalities - "I'm just going to disappear and come back with something brilliant at some unspecified point in the future." People don't want to feel that they're building widgets when they could be making a cathedral but that's often what is needed. Agile and Kanban fight this tendency and are therefore not always popular.
- bdehaaff 13y ago
- coryfklein 13y agoWarning: title not reflective of content.
- bdehaaff 13y agoThanks for the comment. The point is that using Kanban or implementing "agile" is not a savior from the engineering death spiral. The post includes that exact point in the first paragraph.
- karlkfi 13y agoThe real silver bullet behind Kanban has nothing to do with Kanban. It's the desire for continual improvement and willingness to start where you are and improve from there. Anytime you attempt to make a huge process shift you are tempting fate and risking failure. The short term drawbacks to huge change can often stifle the process before it reaches maturity. The trick is to work through the rationalization with the team based on current observable problems and lead them towards self-realization and analysis of their own problems. Most people have ideas of how to make things better, but if you lead them towards the desire for slack, optimizing for the bottleneck, rather than maxing out everyone as if they're all the bottleneck, then you can easily get from there to Kanban-like WIP limits. From there you can adopt other solutions as the problems they tackle are realized. Smart people frequently arrive at similar approaches whenf aced with the same problems, but not always. Your team may find some novel approach that works better for your context, or they may go looking for options and find Kanban. But the goal should begin with being able to communicate process problems, ask how they can be improved, test the potential solutions, and adopt the best ones. Unfortunately, the biggest hinderance to gradual process improvement (once your team is on board) is that software tools tend to offer a "whole solution" and don't have the ability to grow with your team's maturity, unless they're writing it themselves.
- bdehaaff 13y agoThanks for the detailed thoughts. I am not sure though that Kanban or an other agile tool can solve the problem when it is one of leadership or direction. Strategy and management do not rise out of the incremental planning bits.
- karlkfi 13y agoOne of management's jobs is to enable and empower the workers. One of the principals that enables Kanban development is that management should not dictate worker process. If management doesn't know that they may be tempted to fix the development problems with more process, which actually disempowers the workers. Adding more dictated process may work in the short term, but it's a tactical fix, not a strategic one. A well run business keeps tactics and strategy in balance, and hopefully delegates the tactics as low as possible, to retain empowerment. Good management acts as a multiplier to worker productivity. Bad management cripples what it's trying to manage. So yes, you need good management, but good management alone does not guarantee success. Management cannot be the source of the solution to every problem. The other bit to take into account is that management can benefit just as much from its own cycle of metrics, feedback and retrospective action. All levels of an organization need to be able to improve, not just the developer teams. However, that's usually a completely different set of tools and skill sets.
- jonathandart 13y agoCan anyone here recommend some references for someone wanting to learn about Kanban?
- karlkfi 13y agohttp://www.amazon.com/Kanban-ebook/dp/B0057H2M70/ http://www.amazon.com/Kanban-ebook/dp/B0057H2M70/ Note the subtitle: Successful Evolutionary Change. Don't try to change everything all in one go. And if you already know Scrum, here's some differences: http://www.infoq.com/news/2009/05/kniberg-kanban-v-scrum http://www.infoq.com/news/2009/05/kniberg-kanban-v-scrum
- bdehaaff 13y agoThere is a good overview here for Kanban for development: http://en.wikipedia.org/wiki/Kanban_%28development%29 http://en.wikipedia.org/wiki/Kanban_%28development%29
- msaspence 13y agoThe key word (that is admittedly missed from the title) is "blindly": 'blindly applying "agile shock therapy" or introducing Kanban will only suck you further into the abyss'. Which is a bit of truism: do anything blindly and your chances of succeeding are based entirely on luck. Depending on the specific circumstances Kanban, or any other agile methodology, may or may not be part of the solution to the so called "death spiral".