17 ms·

I've been doing Scrum-driven development for a few years now after working much more independently for most of my career. I understand why managers like it beca

I've been doing Scrum-driven development for a few years now after working much more independently for most of my career. I understand why managers like it because it at least lends some structure and predictability to what is an inherently unpredictable enterprise. But the author's criticisms of the incentives of Scrum are on point I think. Because the stories are always articulated in terms of user facing features they encourage developers to hack things together in the most expedient way possible and completely fail to capture the need to address cross cutting concerns, serious consideration of architecture, and refactoring. This is how you can get two years into a project and have managers and clients that think that things are going well when the actual code is an increasingly unmaintainable rats' nest. Good devs confronted with this kind of mess will eventually burn out on sticking their necks out defending necessary but opaque refactoring tasks and move on to greener pastures.

maxxxxx 10y ago
In my view the only way to handle this is to make constant refactoring part of the work without telling management. If you ask for permission for refactoring you will almost always get a "No".
- humanrebar 10y ago
  This is mentioned in the article as possible only for small changes. When the refactoring involves more systemic problems, it becomes impossible to make the change in the course of making other client-facing changes. I'll add that some subsystems are a mess but stable. It's difficult to make refactorings with other changes when there aren't other changes to make. I see this in particular with old but stable subsystems that will need to be ported to new environments someday, but it's never a good day to lay the groundwork for that inevitable change.
  - lmm 10y ago
    > I see this in particular with old but stable subsystems that will need to be ported to new environments someday, but it's never a good day to lay the groundwork for that inevitable change. I think it's correct to defer that work until you need it. You may end up never porting that subsystem, in which case refactoring the currently-stable implementation is wasted effort. As and when porting becomes an actual business requirement it can be prioritized appropriately.
    - humanrebar 10y ago
      > You may end up never porting that subsystem, in which case refactoring the currently-stable implementation is wasted effort. I understand this approach, and it works many times. But when the porting is delayed until the last possible minute, it's more likely that hacks are put in because the requirement turned into a hard deadline. Instead of defining a sensible OS abstraction layer, the developers might find and replace "Windows XP" with "Windows 7".
      - lmm 10y ago
        > I understand this approach, and it works many times. But when the porting is delayed until the last possible minute, it's more likely that hacks are put in because the requirement turned into a hard deadline. Instead of defining a sensible OS abstraction layer, the developers might find and replace "Windows XP" with "Windows 7". Which may well be the right choice for the business at that point. More generally, it's not like doing it now makes it faster than doing it later: you have to put the same amount of total work in either way. In fact you have to put more work in if you do it now, because you will have to ensure that any subsequent changes don't break the porting. If you have other tasks that are a higher priority than porting, you should do them first, almost by definition. Of course if the port is your highest priority then that is what you should be working on (again almost by definition). Porting should be left as late as possible, but no later.
        humanrebar 10y ago
        You misunderstand. If more than a small minority of your code cares what OS it's running on (beyond obvious excepted), you have a large amount of technical debt. Cleaning that up, either by refactoring or replacing the offending code, is in the business's best interest because it keeps costs down and keeps more options available to the business. The 'business people' are not qualified by themselves to say that the cost-benefit analysis of the refactoring is worth the effort. Because while they may have a good idea of the customer benefits, they generally have little insight into what the future costs of technical debt are. You see this obviously demonstrated by 'business people' who are shocked that there is work to do on five-year-old code that has been working fine for years. Any reasonably experienced developer, any many junior developers besides, can tell you that code rots if it's not maintained properly.
- blub 10y ago
  Why is management getting involved? The PO should try to balance this with feature impementation and the team + SM should make it clear when it's needed. This seems to work fine in an environment based on collaboration between PO,SM and dev team, but will probably fail in an adversarial one.
  - JustSomeNobody 10y ago
    Because managers have to report to other managers and they don't want to report that the devs are working on something that already has been implemented and "works".
  - maxxxxx 10y ago
    I have rarely seen a PO as the single contact to the "business". Most of the time there are multiple project managers, product mangers and line managers who have a say and put pressure on the dev team.
- cel1ne 10y ago
  Indeed. That's sometimes the case with other stuff like playability too. I've added simple animations which delighted users from time to time. If I had asked beforehand to invest time for this, the response would be "let's postpone this (forever)".
- JustSomeNobody 10y ago
  Can't do that for deep refactoring. You'll need QA involved and the boss will never sign off on that because, well, he's a boss not a proper manager.
- s_kilk 10y ago
  This is how we ended up with the "Shadow Sprint" at a previous workplace. A whole bunch of utterly essential engineering work was left out of the sprint process, so many of us would just go ahead and do what needed to be done anyway while working on whatever tickets we'd picked up frome the "real" sprint. Utterly dysfunctional of course, but if it's do-or-die, I'd prefer to 'do'.
  - bmj 10y ago
    How did you handle regression testing of the refactored functionality? That's our biggest hurdle. QC is swamped with "regular" sprint work, so there's no way they manage any additional load during the sprint.
    - s_kilk 10y ago
      We had pretty good (but not amazing) unit and integration tests in many of those systems, and a willingness to manually test the systems where we didn't have tests. If we had to lean on a QA department to get this stuff done, I'm not convinced we would have been able to actually deliver that project.
    - ktRolster 10y ago
      Only refactor things in sections of code that you're changing anyway. That way QA already needs to test those sections of code, and gets no extra work.
- matthewmacleod 10y ago
  We tend to bundle that work up and include it in tickets with the full acknowledgement of the rest of the team. "Oh, since we're adding new email features, we'll need to clean up some of the old email code. That will add additional complexity to this task, so we will estimate it higher." It works, allows us to still track the impact on velocity, keeps everyone informed, and makes technical debt clear and trackable.
  - xwkd 10y ago
    I believe this is the correct approach. You're not hiding the complexity of the work, and you're also not compromising on quality. This leaves only the task of saying "no" when asked if you, "for now, could just..."
  - maxxxxx 10y ago
    This is how it should work in my view. You just need managers that are disciplined and don't tell you to not clean up old code because of deadline pressure.
  - abritinthebay 10y ago
    This is exactly what is supposed to happen in a Scrum system. If someone is not doing this they're not doing Scrum - just something they called Scrum to pacify people
- fsloth 10y ago
  In a non-broken organization technical leads (or equivalent) have same level of authority to guide development as feature driven personnel and the necessity of features vs. quality can be triaged in bright daylight. If the organization is not sane, that's another thing of course (and often the case, sadly).
  - maxxxxx 10y ago
    True, but don't forget that non-broken organizations are very rare :-)
    - clifanatic 10y ago
      Do you know of one? I've been looking. For 25 years.
      - engi_nerd 10y ago
        All organizations are broken, but some are more broken than others.
      - fsloth 10y ago
        There are several in the Helsinki area (that I'm aware of - that have been at some point at least - sane in this regard. Many broken of course as well).
- gedy 10y ago
  TBH this is what I've always assumed it should be from day one with Scrum, after training with Ken Schwaber etc. The backlog is not a Todo list. It's the only way that seemed sane without twisting user stories into weird refactoring or architecture stories and keeping team from writing garbage code waiting for these "tech stories" to clean it up.
- Zigurd 10y ago
  This is a typical way that agile and related approaches become a barrier between implementers and management. The fault is almost all on the management side because, ipso facto, they're the ones making the decisions that drive this outcome. In the real world, there are outcomes that are much worse than this. But this does have the effect of neutralizing the value of scrum: Management gets what implementers decide they get, and scrum is just the way middle management is told the story that they pass up the chain.
tomelders 10y ago
I leave my current role in just under two weeks for the very reason you describe. I'm tired of being the "difficult dev" that everyone has grown to hate. Let them launch. Let it fail. Let's see who the arsehole is then.
- taneq 10y ago
  You, because the moment you leave you become that-guy-who-we-can-safely-blame. Mostly joking but plenty of places do work like that.
  - mgkimsal 10y ago
    Yes, they'll blame him. It won't make sense for the most part, but it'll still happen. Probably largely things like "it only tomelders had worked with us more instead of being 'difficult', we wouldn't be in this mess", even if the 'mess' is precisely what he was advocating people work on before he left...
  - JustSomeNobody 10y ago
    It will indeed be his fault. He set them up for failure and left, they'll say.
  - Ntrails 10y ago
    I will always blame whoever most recently left for everything. The rain, the chair being broken, the mouldy cream cheese in the fridge. No jokes. It's Brads fault.
  - Bartweiss 10y ago
    Isn't there a name for this pattern? I swear I remember reading a piece on how having a team member leave can be good for cohesion and efficiency, because everyone is free to vent about old, bad decisions without targeting anyone still at the company. I think it was part of a story about a team clearing their technical debt by convincing a reluctant manager that they needed to undo the last guy's mistakes.
    - 10y ago
thackerhacker 10y ago
It's a long time since I've had to deal with "strict" Scrummers. But I do remember being utterly baffled as to the insistence on user features being ready by the end of a sprint, even for quite complex, technical components. Why can't we make some sub-component this sprint then the UI bit the next? I tried various ways of reframing it such that the developer of the UI be the "user" but it didn't wash.
- lmm 10y ago
  > Why can't we make some sub-component this sprint then the UI bit the next? Because that's how you get bad UI. The user-facing design needs to drive the API interface, not the other way around.
  - tener 10y ago
    Not true. If you have design ready before the implementation starts (as you should unless it is very simple) then you are free to implementation things separately. Just have the interface well specified.
    - matthewmacleod 10y ago
      Speaking from a UI developer perspective, this never works :) But YMMV, I suppose…
      - tener 10y ago
        We always do a brainstorming before implementing any serious feature / change. Once we have the initial design business gets involved (if we can get their attention...). The results usually works without needing any drastic changes, which is still better than no planning at all. It does require involvement from a number of parties though.
    - beerwulf 10y ago
      Sounds like you've never worked in a shop where they think "agile" == "no need to plan ahead"
deejbee 10y ago
This sounds like more of an issue with technical debt management. I'm much more of a proponent of Kanban, whenever possible, so in this case if a user story requires a refactor or architectural change then that's just fine. Also regular code reviews to eliminate hacking and sanity checking automated tests shouldn't be made optional, they should be actively encouraged by the team leads and dev manager.
- specialist 10y ago
  Kanban is the software development equivalent of the Concentration (Memory Match) card game. https://en.wikipedia.org/wiki/Concentration_(game) https://en.wikipedia.org/wiki/Concentration_(game) As though a useful, usable product design (architecture) could be divined thru piecemeal revelation.
- Noseshine 10y ago
  > if a user story requires a refactor or architectural change then that's just fine Those clear cases are not the problem since they are... clear. There is a lot of need for refactoring that arises only slowly and not through one feature. It's more like the boiling frog thing (which by the way is a bogus story but I still use it nevertheless because everybody understands the point). So a problem is you can never really justify the refactoring effort using one specific new feature. It's like partial vs. full cost accounting, sometimes you just have things you can't break down but it still needs to be done (paid). Which makes this a real problem for organizations who don't have the accounting set up for this. In my experience especially large firms may have teams that get their budget "per feature", so even the best manager can't solve that problem - the next (management) layer above doesn't care though because your little software team is too small to get them to laboriously change the accounting process and the SAP system.
crdoconnor 10y ago
One amendment I always wanted to make was to have a "technical debt dial". You could stick it on the wall next to the scrum board. The debt dial should be from 0% (spend all the time refactoring) to 100% (spend zero time refactoring, get it out at all costs). Management should have complete control over the dial. Developers should have control over what kind of refactoring they do (ideally the retro should have a question "what debt/tooling issues caused you the most pain this sprint? and a parallel track for debt/tooling stories").
- gearhart 10y ago
  When I read this I assumed you meant a dial that represents the current level of technical debt, rather than a dial that represents how much technical debt you're allowed to accrue. The other interpretation might be a neat solution. Allow developers to indicate publicly what proportion of the time their last tasks took, they should have taken had there been no technical debt. It would handle the communication from developer into business-speak pretty effectively.
  - ScottBurson 10y ago
    At the very least you need both dials! If you have only the one that management sets, saying how much time to spend refactoring, it will always be at 0. (Okay, they don't literally have to be dials, but there has to be some communication of the kind you describe, so management has some idea how much time is being lost.)
    - user5994461 10y ago
      That seems utterly wrong to me. I'd expect any manager worth is salt to put that at a 10% or 20% (basically anywhere NOT 0) and uses that number as a reminder to devs that part of the job description includes MAINTAINING the system in proper condition, not just piling up new stuff on top of the existing random stuff. Same as a car needs regular maintenance, a dev project needs regular maintenance.
    - crdoconnor 10y ago
afroisalreadyin 10y ago
Author of the post here, thanks for your comment. As you said, the fact that you have to stick out your neck and fight for code quality is very annoying, and after some time, one just stops doing it, and starts flowing with the "add the one more line of crap to get things working and collect the points".
- shepbook 10y ago
  I feel much the same way as much of what you've said in this post. I'm curious what you think of things like the #NoEstimates camp that removes the ambiguity and "gut check" nature of things like points or even time estimates?
  - afroisalreadyin 10y ago
    I read the first chapter of the no estimates book, and have to admit that I found it weak on arguments and poorly written. I think there is real value into doing software estimations if you take them seriously, even using story points, but there must be better ways of using those estimates. One really interesting way of doing this is the Monte Carlo estimate method I linked to in the blog post. Would love to try that sometime, and see how it works out.
    - galleman 10y ago
      The notion that the business can make decision in the presence of uncertainty is the basis of No Estimates. As a participant in another accelerator our funding sources would find that laughable at best and toss us out the cohort at worst.
- discreteevent 10y ago
  "Lean" is supposed to address quality and habitability more. 2 of the 7 tenets of lean focus on it: Build Quality In Optimize the Whole I've never practiced it and my only exposure has been reading some of the book "Lean Architecture". But it certainly seems like an improvement over Scrum which seems blind to some of the most critical things in developing complex innovative systems. In fact I'm convinced that Scrum was designed to work with basic information systems where a feature consists of essentially adding a new data entry form or report for a database (e.g. a lot of web stuff)
mgkimsal 10y ago
I've been in situations like this, and when I've brought up techdebt issues, I get "YAGNI". I've tried to frame technical debt issues in terms of how it will impact future feature requests, performance, etc, and YAGNI is what normally comes back. Until... they actually NI. Then it's a hair on fire crisis. Fortunately have not been in that type of situation for years, but I know they still happen.
- WhitneyLand 10y ago
  For those that didn't know - "You aren't gonna need it" (acronym: YAGNI) is a principle of extreme programming (XP) that states a programmer should not add functionality until deemed necessary.
  - mgkimsal 10y ago
    Sorry - thanks - was in a bit of a rush and forgot to add that. "YAGNI" is... in general... not that useful when used by people who've never worked on a particular type of project, because, almost by definition, they don't what what they will and won't need. And... defining "needs" is its own set of headache. Needed by who? I'll tell you what, we need to ensure we have a logging system in place that can alert folks, and we'll need the ability to view logs in production, and share access to those. Years ago I got "YAGNI" back on that, and months later... weird bugs that no one could reproduce, and the minimal logging in place was only accessible by one guy who was out of town for a week. But hey... we got those "rounded corners" to work in IE5 and IE6 with only an extra week of work - yay... YAGNI is a good principle, but as with most ideas, you get problems when you introduce various types of people in to the mix. Someone who's never done a project type X should not be the one making YAGNI decisions when other people on the team have done multiple "project type X" before, and are trying to introduce basic requirements.
draz 10y ago
I am Product in a large corporation, and it's a certainly a fine balance. In a previous life, I was a developer, and I dealt with similar issues. I have a lot of respect and empathy for the the Tech team. Therefore, I asked my Tech team to inform me when things are getting out of hand -- that's the responsible Product thing to do. I instituted that technical debt is part of our KPIs. If it's not captured, it's not actionable. A few examples: - Documentation: required when method's cyclomatic complexity is high or method is just plain long (we defined what we consider "long"). Every customer facing method has a working usage example (%), etc. - Test Coverage: % unit tests (defined by modules) -- higher is not necessarily better, but it gives us a sense of where we stand. % of test automation: manual vs. automated. - Refactoring delays: # of TODO comments, # of compiler and static code analysis warnings (e.g., “this method is deprecated”) --> that is a sure sign that things will break in the future and an investment is needed. There's some development work to be able to capture these things, but it gives visibility and power to Tech to inject some maintenance work into normal sprint development. I'd love to hear what other people are doing.
- uklenny 10y ago
  The problem with raising issues with a framework like Scrum is that despite many share references, most people are in environments which apply Scrum in a slightly different way. We talk about organisations applying Scrum but what we're really referring to are groups of people, and depending on those people, their mindset, their history and their current role requirements - they will use the Scrum model in their own idiosyncractic way. Take the point about technical debt and running so fast with an Agile development workflow that you never get to refactor code or properly document etc. Even if you just took that in reference to one specific company, the value of, cost of, and importance of these things could be different at different times. When a startup is running to get MVP out to get customer feedback, it's usually much more efficient to set the basic expectation that the first version of your product will get thrown away once you've learned all the important aspects of what you need to deliver. With that in mind - startup development is a completely different beast than when your product is established and you have paying customers with expectations of up-to-date documentation etc. Some managers are do not have good people skills and manage by just holding developers to account based on the expectations set in their Scrum planning day. Discussions about whether refactoring should be done yet or not may not even be something they want to know/talk about and they rely on engineers to factor that into their sprint estimates. Depending on whether the company is being driven by sales/demo opportunities, or feature roll-out timescales etc. then the call about whether to do something 'quick' or 'right' may drop either way. For developers it's important to understand the dynamics that are driving the business and what their role is (sometimes you have to 'JFDI'). If you have a good engineering team with a strong leader then these things shouldn't be that visible to anyone else. It's also important to be able to meet the expectations you set. If you keep delivering late then you'll also struggle to get people to let you do more than the minimum required at the time. For the business as a whole it's all about the big picture and understanding the decisions you make and what their impact (short and medium term) is. If you want quick releases and don't have the resources for that to allow for good documentation and/or scalable code, then you need to understand that technical debt is building up and at some point it will need to be paid.
jroseattle 10y ago
> Because the stories are always articulated in terms of user facing features they encourage developers to hack things together in the most expedient way possible and completely fail to capture the need to address cross cutting concerns, serious consideration of architecture, and refactoring. This is one of our biggest challenges. Much of the work we're doing isn't confined just to a user feature, so square-peg-round-hole syndrome affects us a lot.
patothon 10y ago
I don't like scrum myself, but let me try to defend it because I think the problem of most of the people who don't have a good experience with scrum is a misunderstanding of the underlying principles behind it. Let's just remember that Scrum is just a tool that is trying to replicate some practices of the toyota production system (TPS), and an important principle of the TPS is continuous improvement, and Scrum has this through restrospectives. Another principle is quality built in. Now, how to avoid having a rat's nest after 2 years? First of all, you have to remove velocity as a goal for your team. It's easy to game, and it de-incentivizes finding a _predictable_ velocity, which is the goal of that tool. The goal of velocity, is to know what's the "cost" of building an healthy system. i.e.: Your team ships an arbitrarily 10 points per sprint by doing what they think is right. The same team could ship an arbitrarily 20 points per week. 10 is the _true_ cost of building a healthy system. There is not a lot more to do for good team, honestly, at that point. Management can't say anything, because you bring them predictability, and that's mainly what they want. They might think you're slow, but hey, all of this is relative. Suddenly, what does that mean when your team only ships 5 arbitrary points? That they had to push extra. That there was an unexpected problem. That things were on fire and you had to stop working on features entirely. Basically, that there is something to retro on, find the root cause and anticipate for next sprints, and use to go back to the healthy level of points. I used this at several places. It demands trust, but it always pays off. With some teams I worked with, we set up a technical prioritization planning meeting every week, where basically decided what was going to be prioritized for that week. Each team was picking 2/3 things to do on top of the features. You don't tell anyone. You do it.
- ktRolster 10y ago
  Your team ships an arbitrarily 10 points per sprint by doing what they think is right. The same team could ship an arbitrarily 20 points per week. 10 is the _true_ cost of building a healthy system. There is not a lot more to do for good team, honestly, at that point. Management can't say anything This sort of thing is only useful for managers who don't understand what is going on. Since they are unable to look at code and understand it, 'points' give them a semi-opaque substitute. If management trusts the team, it is not necessary.
abritinthebay 10y ago
> the stories are always articulated in terms of user facing features Reminder: the "User" doesn't have to be an external person to the team. It can be internal for tools, or a tech debt task for internal issues like the ones you mention. Thinking Scrum is just about end/paying User stories is mistake #2 I see when people do Scrum (#1 is the classic "We're going to not actually do Scrum but call it Scrum" that leads to a lot of "Scrum doesn't work" articles itself).

maxxxxx 10y ago

In my view the only way to handle this is to make constant refactoring part of the work without telling management. If you ask for permission for refactoring you will almost always get a "No".

humanrebar 10y ago
This is mentioned in the article as possible only for small changes. When the refactoring involves more systemic problems, it becomes impossible to make the change in the course of making other client-facing changes. I'll add that some subsystems are a mess but stable. It's difficult to make refactorings with other changes when there aren't other changes to make. I see this in particular with old but stable subsystems that will need to be ported to new environments someday, but it's never a good day to lay the groundwork for that inevitable change.
- lmm 10y ago
  > I see this in particular with old but stable subsystems that will need to be ported to new environments someday, but it's never a good day to lay the groundwork for that inevitable change. I think it's correct to defer that work until you need it. You may end up never porting that subsystem, in which case refactoring the currently-stable implementation is wasted effort. As and when porting becomes an actual business requirement it can be prioritized appropriately.
  - humanrebar 10y ago
    > You may end up never porting that subsystem, in which case refactoring the currently-stable implementation is wasted effort. I understand this approach, and it works many times. But when the porting is delayed until the last possible minute, it's more likely that hacks are put in because the requirement turned into a hard deadline. Instead of defining a sensible OS abstraction layer, the developers might find and replace "Windows XP" with "Windows 7".
    - lmm 10y ago
      > I understand this approach, and it works many times. But when the porting is delayed until the last possible minute, it's more likely that hacks are put in because the requirement turned into a hard deadline. Instead of defining a sensible OS abstraction layer, the developers might find and replace "Windows XP" with "Windows 7". Which may well be the right choice for the business at that point. More generally, it's not like doing it now makes it faster than doing it later: you have to put the same amount of total work in either way. In fact you have to put more work in if you do it now, because you will have to ensure that any subsequent changes don't break the porting. If you have other tasks that are a higher priority than porting, you should do them first, almost by definition. Of course if the port is your highest priority then that is what you should be working on (again almost by definition). Porting should be left as late as possible, but no later.
      - humanrebar 10y ago
        You misunderstand. If more than a small minority of your code cares what OS it's running on (beyond obvious excepted), you have a large amount of technical debt. Cleaning that up, either by refactoring or replacing the offending code, is in the business's best interest because it keeps costs down and keeps more options available to the business. The 'business people' are not qualified by themselves to say that the cost-benefit analysis of the refactoring is worth the effort. Because while they may have a good idea of the customer benefits, they generally have little insight into what the future costs of technical debt are. You see this obviously demonstrated by 'business people' who are shocked that there is work to do on five-year-old code that has been working fine for years. Any reasonably experienced developer, any many junior developers besides, can tell you that code rots if it's not maintained properly.
blub 10y ago
Why is management getting involved? The PO should try to balance this with feature impementation and the team + SM should make it clear when it's needed. This seems to work fine in an environment based on collaboration between PO,SM and dev team, but will probably fail in an adversarial one.
- JustSomeNobody 10y ago
  Because managers have to report to other managers and they don't want to report that the devs are working on something that already has been implemented and "works".
- maxxxxx 10y ago
  I have rarely seen a PO as the single contact to the "business". Most of the time there are multiple project managers, product mangers and line managers who have a say and put pressure on the dev team.
cel1ne 10y ago
Indeed. That's sometimes the case with other stuff like playability too. I've added simple animations which delighted users from time to time. If I had asked beforehand to invest time for this, the response would be "let's postpone this (forever)".
JustSomeNobody 10y ago
Can't do that for deep refactoring. You'll need QA involved and the boss will never sign off on that because, well, he's a boss not a proper manager.
s_kilk 10y ago
This is how we ended up with the "Shadow Sprint" at a previous workplace. A whole bunch of utterly essential engineering work was left out of the sprint process, so many of us would just go ahead and do what needed to be done anyway while working on whatever tickets we'd picked up frome the "real" sprint. Utterly dysfunctional of course, but if it's do-or-die, I'd prefer to 'do'.
- bmj 10y ago
  How did you handle regression testing of the refactored functionality? That's our biggest hurdle. QC is swamped with "regular" sprint work, so there's no way they manage any additional load during the sprint.
  - s_kilk 10y ago
    We had pretty good (but not amazing) unit and integration tests in many of those systems, and a willingness to manually test the systems where we didn't have tests. If we had to lean on a QA department to get this stuff done, I'm not convinced we would have been able to actually deliver that project.
  - ktRolster 10y ago
    Only refactor things in sections of code that you're changing anyway. That way QA already needs to test those sections of code, and gets no extra work.
matthewmacleod 10y ago
We tend to bundle that work up and include it in tickets with the full acknowledgement of the rest of the team. "Oh, since we're adding new email features, we'll need to clean up some of the old email code. That will add additional complexity to this task, so we will estimate it higher." It works, allows us to still track the impact on velocity, keeps everyone informed, and makes technical debt clear and trackable.
- xwkd 10y ago
  I believe this is the correct approach. You're not hiding the complexity of the work, and you're also not compromising on quality. This leaves only the task of saying "no" when asked if you, "for now, could just..."
- maxxxxx 10y ago
  This is how it should work in my view. You just need managers that are disciplined and don't tell you to not clean up old code because of deadline pressure.
- abritinthebay 10y ago
  This is exactly what is supposed to happen in a Scrum system. If someone is not doing this they're not doing Scrum - just something they called Scrum to pacify people
fsloth 10y ago
In a non-broken organization technical leads (or equivalent) have same level of authority to guide development as feature driven personnel and the necessity of features vs. quality can be triaged in bright daylight. If the organization is not sane, that's another thing of course (and often the case, sadly).
- maxxxxx 10y ago
  True, but don't forget that non-broken organizations are very rare :-)
  - clifanatic 10y ago
    Do you know of one? I've been looking. For 25 years.
    - engi_nerd 10y ago
      All organizations are broken, but some are more broken than others.
    - fsloth 10y ago
      There are several in the Helsinki area (that I'm aware of - that have been at some point at least - sane in this regard. Many broken of course as well).
gedy 10y ago
TBH this is what I've always assumed it should be from day one with Scrum, after training with Ken Schwaber etc. The backlog is not a Todo list. It's the only way that seemed sane without twisting user stories into weird refactoring or architecture stories and keeping team from writing garbage code waiting for these "tech stories" to clean it up.
Zigurd 10y ago
This is a typical way that agile and related approaches become a barrier between implementers and management. The fault is almost all on the management side because, ipso facto, they're the ones making the decisions that drive this outcome. In the real world, there are outcomes that are much worse than this. But this does have the effect of neutralizing the value of scrum: Management gets what implementers decide they get, and scrum is just the way middle management is told the story that they pass up the chain.

tomelders 10y ago

I leave my current role in just under two weeks for the very reason you describe. I'm tired of being the "difficult dev" that everyone has grown to hate. Let them launch. Let it fail. Let's see who the arsehole is then.

taneq 10y ago
You, because the moment you leave you become that-guy-who-we-can-safely-blame. Mostly joking but plenty of places do work like that.
- mgkimsal 10y ago
  Yes, they'll blame him. It won't make sense for the most part, but it'll still happen. Probably largely things like "it only tomelders had worked with us more instead of being 'difficult', we wouldn't be in this mess", even if the 'mess' is precisely what he was advocating people work on before he left...
- JustSomeNobody 10y ago
  It will indeed be his fault. He set them up for failure and left, they'll say.
- Ntrails 10y ago
  I will always blame whoever most recently left for everything. The rain, the chair being broken, the mouldy cream cheese in the fridge. No jokes. It's Brads fault.
- Bartweiss 10y ago
  Isn't there a name for this pattern? I swear I remember reading a piece on how having a team member leave can be good for cohesion and efficiency, because everyone is free to vent about old, bad decisions without targeting anyone still at the company. I think it was part of a story about a team clearing their technical debt by convincing a reluctant manager that they needed to undo the last guy's mistakes.
  - 10y ago

thackerhacker 10y ago

It's a long time since I've had to deal with "strict" Scrummers. But I do remember being utterly baffled as to the insistence on user features being ready by the end of a sprint, even for quite complex, technical components. Why can't we make some sub-component this sprint then the UI bit the next? I tried various ways of reframing it such that the developer of the UI be the "user" but it didn't wash.

lmm 10y ago
> Why can't we make some sub-component this sprint then the UI bit the next? Because that's how you get bad UI. The user-facing design needs to drive the API interface, not the other way around.
- tener 10y ago
  Not true. If you have design ready before the implementation starts (as you should unless it is very simple) then you are free to implementation things separately. Just have the interface well specified.
  - matthewmacleod 10y ago
    Speaking from a UI developer perspective, this never works :) But YMMV, I suppose…
    - tener 10y ago
      We always do a brainstorming before implementing any serious feature / change. Once we have the initial design business gets involved (if we can get their attention...). The results usually works without needing any drastic changes, which is still better than no planning at all. It does require involvement from a number of parties though.
  - beerwulf 10y ago
    Sounds like you've never worked in a shop where they think "agile" == "no need to plan ahead"

deejbee 10y ago

This sounds like more of an issue with technical debt management. I'm much more of a proponent of Kanban, whenever possible, so in this case if a user story requires a refactor or architectural change then that's just fine. Also regular code reviews to eliminate hacking and sanity checking automated tests shouldn't be made optional, they should be actively encouraged by the team leads and dev manager.

specialist 10y ago
Kanban is the software development equivalent of the Concentration (Memory Match) card game. https://en.wikipedia.org/wiki/Concentration_(game) https://en.wikipedia.org/wiki/Concentration_(game) As though a useful, usable product design (architecture) could be divined thru piecemeal revelation.
Noseshine 10y ago
> if a user story requires a refactor or architectural change then that's just fine Those clear cases are not the problem since they are... clear. There is a lot of need for refactoring that arises only slowly and not through one feature. It's more like the boiling frog thing (which by the way is a bogus story but I still use it nevertheless because everybody understands the point). So a problem is you can never really justify the refactoring effort using one specific new feature. It's like partial vs. full cost accounting, sometimes you just have things you can't break down but it still needs to be done (paid). Which makes this a real problem for organizations who don't have the accounting set up for this. In my experience especially large firms may have teams that get their budget "per feature", so even the best manager can't solve that problem - the next (management) layer above doesn't care though because your little software team is too small to get them to laboriously change the accounting process and the SAP system.

crdoconnor 10y ago

One amendment I always wanted to make was to have a "technical debt dial". You could stick it on the wall next to the scrum board. The debt dial should be from 0% (spend all the time refactoring) to 100% (spend zero time refactoring, get it out at all costs). Management should have complete control over the dial. Developers should have control over what kind of refactoring they do (ideally the retro should have a question "what debt/tooling issues caused you the most pain this sprint? and a parallel track for debt/tooling stories").

gearhart 10y ago
When I read this I assumed you meant a dial that represents the current level of technical debt, rather than a dial that represents how much technical debt you're allowed to accrue. The other interpretation might be a neat solution. Allow developers to indicate publicly what proportion of the time their last tasks took, they should have taken had there been no technical debt. It would handle the communication from developer into business-speak pretty effectively.
- ScottBurson 10y ago
  At the very least you need both dials! If you have only the one that management sets, saying how much time to spend refactoring, it will always be at 0. (Okay, they don't literally have to be dials, but there has to be some communication of the kind you describe, so management has some idea how much time is being lost.)
  - user5994461 10y ago
    That seems utterly wrong to me. I'd expect any manager worth is salt to put that at a 10% or 20% (basically anywhere NOT 0) and uses that number as a reminder to devs that part of the job description includes MAINTAINING the system in proper condition, not just piling up new stuff on top of the existing random stuff. Same as a car needs regular maintenance, a dev project needs regular maintenance.
  - crdoconnor 10y ago

afroisalreadyin 10y ago

Author of the post here, thanks for your comment. As you said, the fact that you have to stick out your neck and fight for code quality is very annoying, and after some time, one just stops doing it, and starts flowing with the "add the one more line of crap to get things working and collect the points".

shepbook 10y ago
I feel much the same way as much of what you've said in this post. I'm curious what you think of things like the #NoEstimates camp that removes the ambiguity and "gut check" nature of things like points or even time estimates?
- afroisalreadyin 10y ago
  I read the first chapter of the no estimates book, and have to admit that I found it weak on arguments and poorly written. I think there is real value into doing software estimations if you take them seriously, even using story points, but there must be better ways of using those estimates. One really interesting way of doing this is the Monte Carlo estimate method I linked to in the blog post. Would love to try that sometime, and see how it works out.
  - galleman 10y ago
    The notion that the business can make decision in the presence of uncertainty is the basis of No Estimates. As a participant in another accelerator our funding sources would find that laughable at best and toss us out the cohort at worst.
discreteevent 10y ago
"Lean" is supposed to address quality and habitability more. 2 of the 7 tenets of lean focus on it: Build Quality In Optimize the Whole I've never practiced it and my only exposure has been reading some of the book "Lean Architecture". But it certainly seems like an improvement over Scrum which seems blind to some of the most critical things in developing complex innovative systems. In fact I'm convinced that Scrum was designed to work with basic information systems where a feature consists of essentially adding a new data entry form or report for a database (e.g. a lot of web stuff)

mgkimsal 10y ago

I've been in situations like this, and when I've brought up techdebt issues, I get "YAGNI". I've tried to frame technical debt issues in terms of how it will impact future feature requests, performance, etc, and YAGNI is what normally comes back. Until... they actually NI. Then it's a hair on fire crisis. Fortunately have not been in that type of situation for years, but I know they still happen.

WhitneyLand 10y ago
For those that didn't know - "You aren't gonna need it" (acronym: YAGNI) is a principle of extreme programming (XP) that states a programmer should not add functionality until deemed necessary.
- mgkimsal 10y ago
  Sorry - thanks - was in a bit of a rush and forgot to add that. "YAGNI" is... in general... not that useful when used by people who've never worked on a particular type of project, because, almost by definition, they don't what what they will and won't need. And... defining "needs" is its own set of headache. Needed by who? I'll tell you what, we need to ensure we have a logging system in place that can alert folks, and we'll need the ability to view logs in production, and share access to those. Years ago I got "YAGNI" back on that, and months later... weird bugs that no one could reproduce, and the minimal logging in place was only accessible by one guy who was out of town for a week. But hey... we got those "rounded corners" to work in IE5 and IE6 with only an extra week of work - yay... YAGNI is a good principle, but as with most ideas, you get problems when you introduce various types of people in to the mix. Someone who's never done a project type X should not be the one making YAGNI decisions when other people on the team have done multiple "project type X" before, and are trying to introduce basic requirements.

draz 10y ago

I am Product in a large corporation, and it's a certainly a fine balance. In a previous life, I was a developer, and I dealt with similar issues. I have a lot of respect and empathy for the the Tech team. Therefore, I asked my Tech team to inform me when things are getting out of hand -- that's the responsible Product thing to do. I instituted that technical debt is part of our KPIs. If it's not captured, it's not actionable. A few examples: - Documentation: required when method's cyclomatic complexity is high or method is just plain long (we defined what we consider "long"). Every customer facing method has a working usage example (%), etc. - Test Coverage: % unit tests (defined by modules) -- higher is not necessarily better, but it gives us a sense of where we stand. % of test automation: manual vs. automated. - Refactoring delays: # of TODO comments, # of compiler and static code analysis warnings (e.g., “this method is deprecated”) --> that is a sure sign that things will break in the future and an investment is needed. There's some development work to be able to capture these things, but it gives visibility and power to Tech to inject some maintenance work into normal sprint development. I'd love to hear what other people are doing.

uklenny 10y ago
The problem with raising issues with a framework like Scrum is that despite many share references, most people are in environments which apply Scrum in a slightly different way. We talk about organisations applying Scrum but what we're really referring to are groups of people, and depending on those people, their mindset, their history and their current role requirements - they will use the Scrum model in their own idiosyncractic way. Take the point about technical debt and running so fast with an Agile development workflow that you never get to refactor code or properly document etc. Even if you just took that in reference to one specific company, the value of, cost of, and importance of these things could be different at different times. When a startup is running to get MVP out to get customer feedback, it's usually much more efficient to set the basic expectation that the first version of your product will get thrown away once you've learned all the important aspects of what you need to deliver. With that in mind - startup development is a completely different beast than when your product is established and you have paying customers with expectations of up-to-date documentation etc. Some managers are do not have good people skills and manage by just holding developers to account based on the expectations set in their Scrum planning day. Discussions about whether refactoring should be done yet or not may not even be something they want to know/talk about and they rely on engineers to factor that into their sprint estimates. Depending on whether the company is being driven by sales/demo opportunities, or feature roll-out timescales etc. then the call about whether to do something 'quick' or 'right' may drop either way. For developers it's important to understand the dynamics that are driving the business and what their role is (sometimes you have to 'JFDI'). If you have a good engineering team with a strong leader then these things shouldn't be that visible to anyone else. It's also important to be able to meet the expectations you set. If you keep delivering late then you'll also struggle to get people to let you do more than the minimum required at the time. For the business as a whole it's all about the big picture and understanding the decisions you make and what their impact (short and medium term) is. If you want quick releases and don't have the resources for that to allow for good documentation and/or scalable code, then you need to understand that technical debt is building up and at some point it will need to be paid.

jroseattle 10y ago

> Because the stories are always articulated in terms of user facing features they encourage developers to hack things together in the most expedient way possible and completely fail to capture the need to address cross cutting concerns, serious consideration of architecture, and refactoring. This is one of our biggest challenges. Much of the work we're doing isn't confined just to a user feature, so square-peg-round-hole syndrome affects us a lot.

patothon 10y ago

I don't like scrum myself, but let me try to defend it because I think the problem of most of the people who don't have a good experience with scrum is a misunderstanding of the underlying principles behind it. Let's just remember that Scrum is just a tool that is trying to replicate some practices of the toyota production system (TPS), and an important principle of the TPS is continuous improvement, and Scrum has this through restrospectives. Another principle is quality built in. Now, how to avoid having a rat's nest after 2 years? First of all, you have to remove velocity as a goal for your team. It's easy to game, and it de-incentivizes finding a _predictable_ velocity, which is the goal of that tool. The goal of velocity, is to know what's the "cost" of building an healthy system. i.e.: Your team ships an arbitrarily 10 points per sprint by doing what they think is right. The same team could ship an arbitrarily 20 points per week. 10 is the _true_ cost of building a healthy system. There is not a lot more to do for good team, honestly, at that point. Management can't say anything, because you bring them predictability, and that's mainly what they want. They might think you're slow, but hey, all of this is relative. Suddenly, what does that mean when your team only ships 5 arbitrary points? That they had to push extra. That there was an unexpected problem. That things were on fire and you had to stop working on features entirely. Basically, that there is something to retro on, find the root cause and anticipate for next sprints, and use to go back to the healthy level of points. I used this at several places. It demands trust, but it always pays off. With some teams I worked with, we set up a technical prioritization planning meeting every week, where basically decided what was going to be prioritized for that week. Each team was picking 2/3 things to do on top of the features. You don't tell anyone. You do it.

ktRolster 10y ago
Your team ships an arbitrarily 10 points per sprint by doing what they think is right. The same team could ship an arbitrarily 20 points per week. 10 is the _true_ cost of building a healthy system. There is not a lot more to do for good team, honestly, at that point. Management can't say anything This sort of thing is only useful for managers who don't understand what is going on. Since they are unable to look at code and understand it, 'points' give them a semi-opaque substitute. If management trusts the team, it is not necessary.

abritinthebay 10y ago

> the stories are always articulated in terms of user facing features Reminder: the "User" doesn't have to be an external person to the team. It can be internal for tools, or a tech debt task for internal issues like the ones you mention. Thinking Scrum is just about end/paying User stories is mistake #2 I see when people do Scrum (#1 is the classic "We're going to not actually do Scrum but call it Scrum" that leads to a lot of "Scrum doesn't work" articles itself).