8 ms·

I work for a corporation that has worshipped Waterfall for many decades. We are now in the middle of a years-long large scale “switch” to Agile, which is a vast

by jl6 8y ago

I work for a corporation that has worshipped Waterfall for many decades. We are now in the middle of a years-long large scale “switch” to Agile, which is a vast cultural change. My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software. Obvious, right? What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development. Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control... These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them. If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.

pasta 8y ago
Specifications are different than constraints. Agile and constraints can go hand in hand imho.
groestl 8y ago
The thing is, for software projects, nothing else really works. So if your core business is software, and your organization doesn't embrace this inherent fact and instead does waterfall, there is a risk that all your examples are fiction, including maintaining financial control.
- gronne 8y ago
  Or you get away with selling bad software at exorbitant prices. Alot of big companies do fine (financially) with the waterfall methodology.
  - close04 8y ago
    > Alot of big companies do fine (financially) with the waterfall methodology. Because it works for them. Right tool for the right job maybe. I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.
    - gronne 8y ago
      I guess. Ime. all project are are combination of waterfall and agile. Wrt. infrastructure the initial architecture plan needs to be almost all-encompassing. On the other hand you need to be able to react to new information and changing circumstances. Being religious with methodology is going to backfire 99% of the time.
    - corpMaverick 8y ago
      > Because it works for them. Right tool for the right job maybe. They are able to deliver software in spite of waterfall rather than because of waterfall. They just keep injecting time and money until they 'declare' their projects to be complete.
- blub 8y ago
  Agile is such a vague thing that it's impossible to say whether it works. So let's take Scrum, which is probably the face of Agile, whether its parents want it or not: Scrum doesn't work either for a lot of projects. Typically the implementors are blamed, but if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself. Maybe the real problem with software is that it's become too unstable because of market forces and all these methods designed to tame that are just a band-aid. Releasing a new version of an OS yearly seems to be a pretty silly idea... At the current level of complexity and churn, everything seems to deliver less than impressive results in the end, but people are kind of used to perpetual crappy software.
  - Holybeds 8y ago
    > if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself. I have heard that sentiment many times but it doesn't really make sense to me. If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then? Or do you mean that the method should be designed in a way that only companies who will succeed will adopt? By your standard, is there any successful method to this area?
    - blub 8y ago
      Yes, I would blame the choice of the method if most patients are too lazy or busy. A good choice would instead work for busy and lazy patients or would convince them to be less lazy and/or busy :) Multiple options should be offered depending on the patient. I've given actually a similar example regarding the "eat well and do sports" advice. It's not bad advice, it is correct, but it's not working for a large segment of the population, causing massive problems for health care and later (or now?) society. Instead of cheap advice maybe the doctor should prescribe physical therapy with a specialist where the patient can be educated on the what and the how. Or the appropriate gov organisation should ban or heavily tax soft drinks and crappy foods and make good, healthy food affordable and available.
    - 8y ago
CommodoreBeard 8y ago
Very insightful observation. As someone who has worked in companies just like the one you described, fully agile startups and horrible hybrids invetween, I think that the biggest failure of the application of agile is that persons outside of the development bubble see agile as a solution to old late, overspent, under delivered solutions. Really the agile manifesto needs to be applied top down. Frameworks like SAFe attempt to solve this. In truth the most powerful application of ‘agile’ every entity in an organisation can apply is to fail as fast as possible. third party is slow to interact and hard to deal with? Switch provider or offer assistance. Hiring staff is tough? Look internally at recruitment team and remuneration offered. Project is late and costing to much? Cancel it, don’t pump more more resource in
- TheOtherHobbes 8y ago
  Not always possible. The UK has a big infrastructure project called Crossrail, which is running about £2bn over budget is two years late. You can't just walk away from a project on that scale. Nor can you get 10% of the way in, realise you've made some fundamental errors, and start again. In reality project management comes down to three problems - finding and optimising competence, dealing with politics and corruption, and minimising danger from external threats. When projects fail it's almost always because something went wrong in at least one of those areas. Agile can sometimes help with optimising competence, but only if it's run competently. It won't magically transform any org or situation that lacks fundamental competence and/or has problems with the other two areas. Agile itself doesn't seem to recognise this.
  - Wildgoose 8y ago
    I thought the problem with Crossrail wasn't that the track wouldn't be ready, it was that the stations wouldn't be ready? In that case, you could concentrate effort on key stations and only add the remaining stations to the timetable when they are available. (I don't know the actual details so I may be missing something. All I know is that there are Barnett consequentials for Crossrail that hands money to Scotland, Wales and Northern Ireland but nothing for we English who don't live in the South East).
jrumbut 8y ago
I think the reason for that hostility is that, since it was decided software needs to be made and software developers are the only ones who can make it, tasks that don't involve making software may be better handled by others. No one has solved contract writing, and of course developers need to contribute sometimes, but the less time they spend on contracts, the more energy that is put into delivery of working products. Working products soothe stakeholders better than developers detailing why there's no product yet. A lot of success with Agile comes down to having great administrators/managers/etc who are capable of dealing with as much of the surrounding efforts needed to make software in an organization as possible.
jdlshore 8y ago
> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control... These aren't part of Agile 101, but the Agile community has been discussing these things for years. I first heard Mary Poppendieck discuss Agile contracts in 2006 or so [1]. Pat Reed has been talking about Agile accounting since at least 2014, and I doubt that was the beginning of it [2]. My book discusses techniques for making commitments to stakeholders, and it was written in 2007 [3]. Literally Google the phrase "Agile" + anything you wrote and you'll find a wealth of resources. If you're not finding that information, it's because you're not looking. [1] http://agileconsortium.pbworks.com/f/AgileContracts.pdf http://agileconsortium.pbworks.com/f/AgileContracts.pdf [2] https://www.slideshare.net/ITRevolution/wednesday-530-pat-reed-gb-final https://www.slideshare.net/ITRevolution/wednesday-530-pat-re... [3] https://www.jamesshore.com/Agile-Book/risk_management.html https://www.jamesshore.com/Agile-Book/risk_management.html
- blub 8y ago
  The agile community is currently an amorphous blob of people connected by words like "Agile!", "Scrum", "TDD" and so on. Agile was stolen from its creators and mutated into a corporate version - which I have to say I find pretty darn funny. Point is, that info is certainly not included in any of the Agile resources I've seen. The only one that handles some of this is Ambler's D.A.D, but I didn't bother reading that for a simple reason: after seeing various Agile methods (mostly Scrum) fail so much, I've come to the conclusion that Agile development processes do not represent a real-world solution. There's always an excuse why Scrum fails in practice, but in the end it requires such a paradigm shift from our typical development approaches in order to succeed, that it's about as useful as a diet plan that only works on super-athletes. XP is in zombie mode for a decade, Scrum is only good for mobile apps and small websites, Kanban is not even a way of making software. What are we left with? Scaled Agile, LeSS, which even the original Agile people bitterly complain about? These won't work either, but I know some consultants make a crapton of money selling them to corps.
  - 8y ago
specialist 8y ago
The Agile Methodology is to argue about The Agile Methodology. -- The only durable thing I've learned about methodologies: Alistair Cockburn "Characterizing People as Non-linear, First Order Components in Software Development." Oops. https://alistair.cockburn.us https://alistair.cockburn.us is "under construction". Sorry. Meanwhile, here's a pretty good TL;DR: http://www.johngoodpasture.com/2009/11/people-puzzle-non-linear-devices.html http://www.johngoodpasture.com/2009/11/people-puzzle-non-lin... Anywho. My current position is "just tell me what you want, I'll adapt." As a recovering methodology junkie, I can no longer stand talking about it. If everyone is on the same page (roles, rules, expectations, etc), I'm happy. If someone dictates this ceremony by fiat, minimizing discussion, all the better.
pif 8y ago
> My observation is that Agile is all about ... and for the benefit of software developers ... No, it's the opposite! Agile is bad news for software developers: 1) hardware developers will insist on having predetermined requirements and will stick to them; at most, they'll throw the mantra: "Software will fix it!"; 2) software users will use Agile as an excuse to avoid at all cost deciding what they want and limit themselves to bikeshedding after the release; 3) those idiot project leaders who manage by ticking cases on an Excel printout will explain you how Agile is a wonderful panacea which will solve all your problems: you will "just" have to write the code and you'll have no excuse to miss the arbitrarily imposed deadline. No no no, not at all! Agile is for those who ponder about developing software, but not for those who do the job!
- corpMaverick 8y ago
  Paraphrasing here. "Agile is the worst software development process, except all the rest" More seriously, doing big upfront design was a really bad idea. Agile and more specifically Scrum brings a number of ceremonies to keep the "managers" happy and allow the transition. When and if ever companies understand that they can trust their software engineers they can let them to their job. These ceremonies are more like a security blanked for the non software engineers who are frequently clueless about how software is developed. I prefer to focus on the Agile principles. My favorite is "Continuous attention to technical excellence and good design enhances agility."
  - pif 8y ago
    > Agile is the worst software development process, except all the rest. That's not my opinion. I'll quote Alex' worlds: In the past decade or so, we’ve seen a lot of “fad development methodologies” crop up. [...] They all promise the same thing – cheaper, better software built faster – and they all generally fail to deliver. Like fad diets, these methodologies try to avoid a simple, painful truth: you have to know what to build before you can start building it. (http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile)
    - 8y ago
lmm 8y ago
> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control... > These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them. A number of attacks on Agile I've read have made precisely the opposite claim: that Agile is "too generic" and not focused enough on the specifics of software development. My experience is that an Agile approach can be effective at all the things you mention, and you don't particularly need to do anything different on the Agile side of things. What is true is that Agile relies on being able to iterate quickly with the end customer (in the sense of "whoever is giving you this requirement"), and only adds value when the requirements change quickly and/or the customer isn't actually quite sure what they want. In my experience this is a lot more common than you might think even in non-software, and creating the conditions for success can be easy once you try. E.g. if you ask your regulator whether you can take a more collaborative approach this time, have a frequent back-and-forth where you show them the current state of things and see whether they're getting what they want, or have any changes to what they want from you - you might be surprised how willing they are to try, and how happy both of you are with the resulting process. But yeah, it is a substantial change, and I can believe there are domains where the costs outweigh the benefits. If you're working in a domain that's very well-understood, where the customer really can figure out the requirements up-front without even a prototype to go on and those requirements really are going to stay the same for however long it takes to implement them, then fair enough, maybe waterfall will work better. I do think very little of business actually works that way though; in my experience even "successful" waterfall projects that deliver all the requirements that were originally identified often turn out as something most stakeholders aren't very happy with. "It's just what we asked for, but not what we want" is very real.
watwut 8y ago
> My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software. No, it is not. It is definitely not for developers benefit. It is sorta kinda good for junior developers I think, but mostly it is for management. It effectively ignores human psychology, treats developers as machines and individual developer is mostly powerless with no responsibility, no accountability and zero autonomy. It effectively amounts to large dose of micromanagement. It may sound weird given agile rhetoric and some its processes are really good and improvement, but in it is does creates pleasant work environment in practice.
marcosdumay 8y ago
Agile is all about how contractor developers should interact with their clients. By coincidence, some of it applies to software development.

by jl6 8y ago

pasta 8y ago

Specifications are different than constraints. Agile and constraints can go hand in hand imho.

groestl 8y ago

The thing is, for software projects, nothing else really works. So if your core business is software, and your organization doesn't embrace this inherent fact and instead does waterfall, there is a risk that all your examples are fiction, including maintaining financial control.

gronne 8y ago
Or you get away with selling bad software at exorbitant prices. Alot of big companies do fine (financially) with the waterfall methodology.
- close04 8y ago
  > Alot of big companies do fine (financially) with the waterfall methodology. Because it works for them. Right tool for the right job maybe. I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.
  - gronne 8y ago
    I guess. Ime. all project are are combination of waterfall and agile. Wrt. infrastructure the initial architecture plan needs to be almost all-encompassing. On the other hand you need to be able to react to new information and changing circumstances. Being religious with methodology is going to backfire 99% of the time.
  - corpMaverick 8y ago
    > Because it works for them. Right tool for the right job maybe. They are able to deliver software in spite of waterfall rather than because of waterfall. They just keep injecting time and money until they 'declare' their projects to be complete.
blub 8y ago
Agile is such a vague thing that it's impossible to say whether it works. So let's take Scrum, which is probably the face of Agile, whether its parents want it or not: Scrum doesn't work either for a lot of projects. Typically the implementors are blamed, but if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself. Maybe the real problem with software is that it's become too unstable because of market forces and all these methods designed to tame that are just a band-aid. Releasing a new version of an OS yearly seems to be a pretty silly idea... At the current level of complexity and churn, everything seems to deliver less than impressive results in the end, but people are kind of used to perpetual crappy software.
- Holybeds 8y ago
  > if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself. I have heard that sentiment many times but it doesn't really make sense to me. If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then? Or do you mean that the method should be designed in a way that only companies who will succeed will adopt? By your standard, is there any successful method to this area?
  - blub 8y ago
    Yes, I would blame the choice of the method if most patients are too lazy or busy. A good choice would instead work for busy and lazy patients or would convince them to be less lazy and/or busy :) Multiple options should be offered depending on the patient. I've given actually a similar example regarding the "eat well and do sports" advice. It's not bad advice, it is correct, but it's not working for a large segment of the population, causing massive problems for health care and later (or now?) society. Instead of cheap advice maybe the doctor should prescribe physical therapy with a specialist where the patient can be educated on the what and the how. Or the appropriate gov organisation should ban or heavily tax soft drinks and crappy foods and make good, healthy food affordable and available.
  - 8y ago

CommodoreBeard 8y ago

Very insightful observation. As someone who has worked in companies just like the one you described, fully agile startups and horrible hybrids invetween, I think that the biggest failure of the application of agile is that persons outside of the development bubble see agile as a solution to old late, overspent, under delivered solutions. Really the agile manifesto needs to be applied top down. Frameworks like SAFe attempt to solve this. In truth the most powerful application of ‘agile’ every entity in an organisation can apply is to fail as fast as possible. third party is slow to interact and hard to deal with? Switch provider or offer assistance. Hiring staff is tough? Look internally at recruitment team and remuneration offered. Project is late and costing to much? Cancel it, don’t pump more more resource in

TheOtherHobbes 8y ago
Not always possible. The UK has a big infrastructure project called Crossrail, which is running about £2bn over budget is two years late. You can't just walk away from a project on that scale. Nor can you get 10% of the way in, realise you've made some fundamental errors, and start again. In reality project management comes down to three problems - finding and optimising competence, dealing with politics and corruption, and minimising danger from external threats. When projects fail it's almost always because something went wrong in at least one of those areas. Agile can sometimes help with optimising competence, but only if it's run competently. It won't magically transform any org or situation that lacks fundamental competence and/or has problems with the other two areas. Agile itself doesn't seem to recognise this.
- Wildgoose 8y ago
  I thought the problem with Crossrail wasn't that the track wouldn't be ready, it was that the stations wouldn't be ready? In that case, you could concentrate effort on key stations and only add the remaining stations to the timetable when they are available. (I don't know the actual details so I may be missing something. All I know is that there are Barnett consequentials for Crossrail that hands money to Scotland, Wales and Northern Ireland but nothing for we English who don't live in the South East).

jrumbut 8y ago

I think the reason for that hostility is that, since it was decided software needs to be made and software developers are the only ones who can make it, tasks that don't involve making software may be better handled by others. No one has solved contract writing, and of course developers need to contribute sometimes, but the less time they spend on contracts, the more energy that is put into delivery of working products. Working products soothe stakeholders better than developers detailing why there's no product yet. A lot of success with Agile comes down to having great administrators/managers/etc who are capable of dealing with as much of the surrounding efforts needed to make software in an organization as possible.

jdlshore 8y ago

blub 8y ago
The agile community is currently an amorphous blob of people connected by words like "Agile!", "Scrum", "TDD" and so on. Agile was stolen from its creators and mutated into a corporate version - which I have to say I find pretty darn funny. Point is, that info is certainly not included in any of the Agile resources I've seen. The only one that handles some of this is Ambler's D.A.D, but I didn't bother reading that for a simple reason: after seeing various Agile methods (mostly Scrum) fail so much, I've come to the conclusion that Agile development processes do not represent a real-world solution. There's always an excuse why Scrum fails in practice, but in the end it requires such a paradigm shift from our typical development approaches in order to succeed, that it's about as useful as a diet plan that only works on super-athletes. XP is in zombie mode for a decade, Scrum is only good for mobile apps and small websites, Kanban is not even a way of making software. What are we left with? Scaled Agile, LeSS, which even the original Agile people bitterly complain about? These won't work either, but I know some consultants make a crapton of money selling them to corps.
- 8y ago

specialist 8y ago

The Agile Methodology is to argue about The Agile Methodology. -- The only durable thing I've learned about methodologies: Alistair Cockburn "Characterizing People as Non-linear, First Order Components in Software Development." Oops. https://alistair.cockburn.us https://alistair.cockburn.us is "under construction". Sorry. Meanwhile, here's a pretty good TL;DR: http://www.johngoodpasture.com/2009/11/people-puzzle-non-linear-devices.html http://www.johngoodpasture.com/2009/11/people-puzzle-non-lin... Anywho. My current position is "just tell me what you want, I'll adapt." As a recovering methodology junkie, I can no longer stand talking about it. If everyone is on the same page (roles, rules, expectations, etc), I'm happy. If someone dictates this ceremony by fiat, minimizing discussion, all the better.

pif 8y ago

> My observation is that Agile is all about ... and for the benefit of software developers ... No, it's the opposite! Agile is bad news for software developers: 1) hardware developers will insist on having predetermined requirements and will stick to them; at most, they'll throw the mantra: "Software will fix it!"; 2) software users will use Agile as an excuse to avoid at all cost deciding what they want and limit themselves to bikeshedding after the release; 3) those idiot project leaders who manage by ticking cases on an Excel printout will explain you how Agile is a wonderful panacea which will solve all your problems: you will "just" have to write the code and you'll have no excuse to miss the arbitrarily imposed deadline. No no no, not at all! Agile is for those who ponder about developing software, but not for those who do the job!

corpMaverick 8y ago
Paraphrasing here. "Agile is the worst software development process, except all the rest" More seriously, doing big upfront design was a really bad idea. Agile and more specifically Scrum brings a number of ceremonies to keep the "managers" happy and allow the transition. When and if ever companies understand that they can trust their software engineers they can let them to their job. These ceremonies are more like a security blanked for the non software engineers who are frequently clueless about how software is developed. I prefer to focus on the Agile principles. My favorite is "Continuous attention to technical excellence and good design enhances agility."
- pif 8y ago
  > Agile is the worst software development process, except all the rest. That's not my opinion. I'll quote Alex' worlds: In the past decade or so, we’ve seen a lot of “fad development methodologies” crop up. [...] They all promise the same thing – cheaper, better software built faster – and they all generally fail to deliver. Like fad diets, these methodologies try to avoid a simple, painful truth: you have to know what to build before you can start building it. (http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile)
  - 8y ago

lmm 8y ago

> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control... > These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them. A number of attacks on Agile I've read have made precisely the opposite claim: that Agile is "too generic" and not focused enough on the specifics of software development. My experience is that an Agile approach can be effective at all the things you mention, and you don't particularly need to do anything different on the Agile side of things. What is true is that Agile relies on being able to iterate quickly with the end customer (in the sense of "whoever is giving you this requirement"), and only adds value when the requirements change quickly and/or the customer isn't actually quite sure what they want. In my experience this is a lot more common than you might think even in non-software, and creating the conditions for success can be easy once you try. E.g. if you ask your regulator whether you can take a more collaborative approach this time, have a frequent back-and-forth where you show them the current state of things and see whether they're getting what they want, or have any changes to what they want from you - you might be surprised how willing they are to try, and how happy both of you are with the resulting process. But yeah, it is a substantial change, and I can believe there are domains where the costs outweigh the benefits. If you're working in a domain that's very well-understood, where the customer really can figure out the requirements up-front without even a prototype to go on and those requirements really are going to stay the same for however long it takes to implement them, then fair enough, maybe waterfall will work better. I do think very little of business actually works that way though; in my experience even "successful" waterfall projects that deliver all the requirements that were originally identified often turn out as something most stakeholders aren't very happy with. "It's just what we asked for, but not what we want" is very real.

watwut 8y ago

> My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software. No, it is not. It is definitely not for developers benefit. It is sorta kinda good for junior developers I think, but mostly it is for management. It effectively ignores human psychology, treats developers as machines and individual developer is mostly powerless with no responsibility, no accountability and zero autonomy. It effectively amounts to large dose of micromanagement. It may sound weird given agile rhetoric and some its processes are really good and improvement, but in it is does creates pleasant work environment in practice.

marcosdumay 8y ago

Agile is all about how contractor developers should interact with their clients. By coincidence, some of it applies to software development.