5 ms·

Agile Development: Where’s the Proof?

21 points by fogus 16y ago

stcredzero 16y ago
I do agree (in part) with his answer to this question that I asked: “How should the industry be approaching software development, in your opinion?” He answered, “With humility and measurement. "The Principle of Dispassionate Methodology: Don’t get your method advice from a method enthusiast. The best advice comes from people who care more about your problem than about their solution." This makes me want to read that other guy's blog! Sounds like an approach to software methodology that would come out of LessWrong.
- plinkplonk 16y ago
  "This makes me want to read that other guy's blog! Sounds like an approach to software methodology that would come out of LessWrong." IIRC Isaac (the guy who is replying to the Agile enthusiast on the thread) did a superb job of demolishing the agile "evidence" presented by Craig Larman and the article was first put up on Ron Jeffries's site for "fairness" but was later pulled (I guess because it was causing serious credibility issues for the methodology vendor folks). As he says, "Ron Jeffries challenged me to write specifically about Craig Larman's "evidence", and for a while what I wrote and Craig Larman's response was shown on www.xprogramming.com - but now it seems to be the only article going back through 1998 that isn't available from the website archives ;-) Doing that research taught me that I don't enjoy debunking - proximity to snake oil leaves a bad taste." "The Principle of Dispassionate Methodology: Don’t get your method advice from a method enthusiast. The best advice comes from people who care more of Wisdom.
  - alextp 16y ago
    Does he have a blog or any such thing? Is this text on archive.org?
    - plinkplonk 16y ago
      Not that I know of.
    - stcredzero 16y ago
      The world really needs a "Dispassionate Meta-methodology." We need some group that goes around doing analyses and meta-analyses of various practices. I'm sure there are lots of not-so dispassionate methodology people who would love to provide data. (One would collect the data and ignore the excuses.)
    - nickelplate 16y ago
      http://web.archive.org/web/20060720193601/www.xprogramming.com/xpmag/misstatingtheevidence.htm http://web.archive.org/web/20060720193601/www.xprogramming.c... Edit: Damn, he indeed superbly demolishes Larman's "evidence".
j_baker 16y ago
I think that the agilist response to the issue is that there isn't any way to measure agile's effectiveness. See Martin Fowler's article on the subject: http://www.martinfowler.com/bliki/CannotMeasureProductivity.html http://www.martinfowler.com/bliki/CannotMeasureProductivity.... Like it or not, programmer productivity is difficult if not impossible to measure. I'm all for reasoned debate on the subject, but methodologies are very subjective. Expecting proof out of an agilist is a bit like asking a liberal to prove that liberalism is correct: you're never going to get it because it can't be provided.
- Tamerlin 16y ago
  All of the (few) successful projects I've been on have been ones where the team whole-heartedly rejected agile practices. On the other hand, every single "agile" project I've been on devolved rapidly into a sweatshop environment with a hack-ridden nightmare of tightly coupled bad code cobbled together with more tightly coupled bad code. I've come to the conclusion that "agile" is what the talentless hackers hide behind in order to avoid the scrutiny of being found out as incompetent. As long as "heroes" garner rewards, those folks will be the norm rather than the rule, and any process you use with a team like that will lead to failure. Making things worse (for agile) is that the most vocal proponents for it are typically the sort of people who would rather avoid the had part of actually doing any sort of engineering, and would rather just jump in and start writing code, even if that means writing code with an actual goal in mind. Managers appear to love this, because they still think that lines of code = productivity. The result? Agile projects fail. No process can save a project from its own team, though.
  - calibraxis 16y ago
    Which aspects of the Agile Manifesto led to problems? And were the teams using sufficiently agile languages? (Say, Python or Lisp?) Or was it more a matter of the teams saying "agile" and meaning "cowboy coding"?
    - Tamerlin 16y ago
      Ruby. The strongest agile proponents that I have met prefer Java. I think that's probably enough to estimate their "caliber" right off the bat. The strongest proponents of agile that I've met use agile as an excuse for cowboy coding, though my friends and I generally refer to them as hackers, because all they do is hack things together in a big hurry. You work in "agile" environments like that for a few years, and you start to equate "agile" with "sweatshop." Since without exception every "agile" project I've worked on has been a sweatshop with an agile label, you'd be hard pressed to convince me to use agile methods for anything even vaguely related to developing software.
bguthrie 16y ago
I am a developer who works on teams that use agile processes. I happen to believe they work better, based on my subjective experiences with other approaches, but I don't really have the kind of numbers one really needs to back up each individual practice, which is something I struggle with. I don't struggle much, though. The most important aspect of my definition of agile is that it speeds up the feedback loop, which allows the person signing the checks to gather whatever kinds of metrics they need to on a week to week basis. Essentially, weekly iterations make quantitative analysis of the software development process possible. Every other practice teams do or don't adopt falls out from this newfound ability to gather and disseminate metrics which are, in principle, of much higher quality. Do write tests, or don't [ed. note: do]; but either way, track weekly, and let that guide your process.
agentultra 16y ago
Evidence would be hard to obtain. I find it highly improbable that any software development methodology can provide deterministic, repeatable results. The metric to measure isn't productivity, rather one should be looking at happiness. If it works for your team and they feel like they get a lot done by using an agile approach then you're using the right methodology. Agile isn't for everyone though and that's not a failing of the methodology.
spivey 16y ago
Scott Ambler has conducted surveys that may interest you. http://www.ambysoft.com/surveys/ http://www.ambysoft.com/surveys/
mhd 16y ago
I think the first comment says it all: "Agile doesn’t exist, therefore, I am “not a fan” and neither are you." Agile/XP is a hodgepodge of methodologies and approaches to software development, plus the usual empty phrases any software engineering practice seems to carry with them (at least it’s not on a level with the ol’ Unified Process in this regard). I could see some good scientific studies done on individual parts, e.g. TDD or pair programming, but anything else would be a rather blurry definition. (I remember some TDD studies been done, but if I remember correctly both the approach and the conclusions were a bit weird).
- jacabado 16y ago
  I guess the problem comes from the fact that you can't convince any programmer(engineer?) with rational arguments so the evangelists and team leaders need to resort to fairy tales. It's an art in itself which can be mastered without any technical knowledge. When this art comes to discussion it's easily taken down. I'm still reading all the related posts here but the humility and measurement could be my new passion. :D
  - mhd 16y ago
    You can't convince engineers with facts? That's a very, very sad assumption and/or experience. Care to elaborate?
caustic 16y ago
The problem with software development is it is just a very broad field. When someone talks about programming, he/she usually talk about some specific kind of programming. You see, there are many programming worlds, like system programming, middleware/framework programming, multimedia/game programming, embedded devices programming, business applications development and so on. These are very different and require different skill sets. One will have hard time transitioning from writing dating sites in PHP to writing high-performant web servers, from writing SQL queries to designing database engines. I am convinced that Agile is implicitly related to business application development, and when someone is saying he/she is using Agile methodology, they most likely develop web applications for enterprises. And probably this is the most widespread niche today. But frankly speaking, I don't think this is not the most challenging work for programmers. Been there, done that. But I am yet to see Agile evangelist who clearly states what kind of projects Agile is appropriate for. They all speak about software development in general without making distinction, as if Linux kernel or Java virtual machine or Ruby on Rails framework should be developed with their methodology. Certainly you can come up with many more examples of successfull hi-tech projects that didn't use Agile. Ironically, these projects is what makes Agile possible at all. Millions of man-haours were invested in development of operating systems, databases, virtual machines, libraries, middleware frameworks, programming languages, protocols and so on and on. Agile developers take all this as granted and try to teach everyone else how to do software development. Or thise evangelists simply have too narrow scope, then this ignorance annoying. Does Agile work for software projects that require heavy engineering? I don't think so.
- philwelch 16y ago
  Your comment reminds me of this: http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s... http://pindancing.blogspot.com/2009/09/sudoku-in-coders-at-work.html http://pindancing.blogspot.com/2009/09/sudoku-in-coders-at-w... It's a discussion of two people, Peter Norvig and a TDD enthusiast, both building a Sudoku solver. Norvig takes a no-nonsense AI approach and the TDD enthusiast flails for five blog posts without getting anywhere. To me, the moral of the story isn't to mock TDD, but to point out that different problems require different approaches. TDD can't be naively applied to an algorithmic problem the way it can be applied to a nuts-on-bolts web app, for instance.
spenrose 16y ago
See also: "Evidence-based Software Engineering": http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113.571&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113... and the work of Greg Wilson: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 http://www.slideshare.net/gvwilson/bits-of-evidence-2338367

21 points by fogus 16y ago

I do agree (in part) with his answer to this question that I asked: “How should the industry be approaching software development, in your opinion?” He answered, “With humility and measurement. "The Principle of Dispassionate Methodology: Don’t get your method advice from a method enthusiast. The best advice comes from people who care more about your problem than about their solution." This makes me want to read that other guy's blog! Sounds like an approach to software methodology that would come out of LessWrong.

plinkplonk 16y ago
"This makes me want to read that other guy's blog! Sounds like an approach to software methodology that would come out of LessWrong." IIRC Isaac (the guy who is replying to the Agile enthusiast on the thread) did a superb job of demolishing the agile "evidence" presented by Craig Larman and the article was first put up on Ron Jeffries's site for "fairness" but was later pulled (I guess because it was causing serious credibility issues for the methodology vendor folks). As he says, "Ron Jeffries challenged me to write specifically about Craig Larman's "evidence", and for a while what I wrote and Craig Larman's response was shown on www.xprogramming.com - but now it seems to be the only article going back through 1998 that isn't available from the website archives ;-) Doing that research taught me that I don't enjoy debunking - proximity to snake oil leaves a bad taste." "The Principle of Dispassionate Methodology: Don’t get your method advice from a method enthusiast. The best advice comes from people who care more of Wisdom.
- alextp 16y ago
  Does he have a blog or any such thing? Is this text on archive.org?
  - plinkplonk 16y ago
    Not that I know of.
  - stcredzero 16y ago
    The world really needs a "Dispassionate Meta-methodology." We need some group that goes around doing analyses and meta-analyses of various practices. I'm sure there are lots of not-so dispassionate methodology people who would love to provide data. (One would collect the data and ignore the excuses.)
  - nickelplate 16y ago
    http://web.archive.org/web/20060720193601/www.xprogramming.com/xpmag/misstatingtheevidence.htm http://web.archive.org/web/20060720193601/www.xprogramming.c... Edit: Damn, he indeed superbly demolishes Larman's "evidence".

j_baker 16y ago

I think that the agilist response to the issue is that there isn't any way to measure agile's effectiveness. See Martin Fowler's article on the subject: http://www.martinfowler.com/bliki/CannotMeasureProductivity.html http://www.martinfowler.com/bliki/CannotMeasureProductivity.... Like it or not, programmer productivity is difficult if not impossible to measure. I'm all for reasoned debate on the subject, but methodologies are very subjective. Expecting proof out of an agilist is a bit like asking a liberal to prove that liberalism is correct: you're never going to get it because it can't be provided.

Tamerlin 16y ago
All of the (few) successful projects I've been on have been ones where the team whole-heartedly rejected agile practices. On the other hand, every single "agile" project I've been on devolved rapidly into a sweatshop environment with a hack-ridden nightmare of tightly coupled bad code cobbled together with more tightly coupled bad code. I've come to the conclusion that "agile" is what the talentless hackers hide behind in order to avoid the scrutiny of being found out as incompetent. As long as "heroes" garner rewards, those folks will be the norm rather than the rule, and any process you use with a team like that will lead to failure. Making things worse (for agile) is that the most vocal proponents for it are typically the sort of people who would rather avoid the had part of actually doing any sort of engineering, and would rather just jump in and start writing code, even if that means writing code with an actual goal in mind. Managers appear to love this, because they still think that lines of code = productivity. The result? Agile projects fail. No process can save a project from its own team, though.
- calibraxis 16y ago
  Which aspects of the Agile Manifesto led to problems? And were the teams using sufficiently agile languages? (Say, Python or Lisp?) Or was it more a matter of the teams saying "agile" and meaning "cowboy coding"?
  - Tamerlin 16y ago
    Ruby. The strongest agile proponents that I have met prefer Java. I think that's probably enough to estimate their "caliber" right off the bat. The strongest proponents of agile that I've met use agile as an excuse for cowboy coding, though my friends and I generally refer to them as hackers, because all they do is hack things together in a big hurry. You work in "agile" environments like that for a few years, and you start to equate "agile" with "sweatshop." Since without exception every "agile" project I've worked on has been a sweatshop with an agile label, you'd be hard pressed to convince me to use agile methods for anything even vaguely related to developing software.

bguthrie 16y ago

I am a developer who works on teams that use agile processes. I happen to believe they work better, based on my subjective experiences with other approaches, but I don't really have the kind of numbers one really needs to back up each individual practice, which is something I struggle with. I don't struggle much, though. The most important aspect of my definition of agile is that it speeds up the feedback loop, which allows the person signing the checks to gather whatever kinds of metrics they need to on a week to week basis. Essentially, weekly iterations make quantitative analysis of the software development process possible. Every other practice teams do or don't adopt falls out from this newfound ability to gather and disseminate metrics which are, in principle, of much higher quality. Do write tests, or don't [ed. note: do]; but either way, track weekly, and let that guide your process.

agentultra 16y ago

Evidence would be hard to obtain. I find it highly improbable that any software development methodology can provide deterministic, repeatable results. The metric to measure isn't productivity, rather one should be looking at happiness. If it works for your team and they feel like they get a lot done by using an agile approach then you're using the right methodology. Agile isn't for everyone though and that's not a failing of the methodology.

spivey 16y ago

Scott Ambler has conducted surveys that may interest you. http://www.ambysoft.com/surveys/ http://www.ambysoft.com/surveys/

mhd 16y ago

I think the first comment says it all: "Agile doesn’t exist, therefore, I am “not a fan” and neither are you." Agile/XP is a hodgepodge of methodologies and approaches to software development, plus the usual empty phrases any software engineering practice seems to carry with them (at least it’s not on a level with the ol’ Unified Process in this regard). I could see some good scientific studies done on individual parts, e.g. TDD or pair programming, but anything else would be a rather blurry definition. (I remember some TDD studies been done, but if I remember correctly both the approach and the conclusions were a bit weird).

jacabado 16y ago
I guess the problem comes from the fact that you can't convince any programmer(engineer?) with rational arguments so the evangelists and team leaders need to resort to fairy tales. It's an art in itself which can be mastered without any technical knowledge. When this art comes to discussion it's easily taken down. I'm still reading all the related posts here but the humility and measurement could be my new passion. :D
- mhd 16y ago
  You can't convince engineers with facts? That's a very, very sad assumption and/or experience. Care to elaborate?

caustic 16y ago

The problem with software development is it is just a very broad field. When someone talks about programming, he/she usually talk about some specific kind of programming. You see, there are many programming worlds, like system programming, middleware/framework programming, multimedia/game programming, embedded devices programming, business applications development and so on. These are very different and require different skill sets. One will have hard time transitioning from writing dating sites in PHP to writing high-performant web servers, from writing SQL queries to designing database engines. I am convinced that Agile is implicitly related to business application development, and when someone is saying he/she is using Agile methodology, they most likely develop web applications for enterprises. And probably this is the most widespread niche today. But frankly speaking, I don't think this is not the most challenging work for programmers. Been there, done that. But I am yet to see Agile evangelist who clearly states what kind of projects Agile is appropriate for. They all speak about software development in general without making distinction, as if Linux kernel or Java virtual machine or Ruby on Rails framework should be developed with their methodology. Certainly you can come up with many more examples of successfull hi-tech projects that didn't use Agile. Ironically, these projects is what makes Agile possible at all. Millions of man-haours were invested in development of operating systems, databases, virtual machines, libraries, middleware frameworks, programming languages, protocols and so on and on. Agile developers take all this as granted and try to teach everyone else how to do software development. Or thise evangelists simply have too narrow scope, then this ignorance annoying. Does Agile work for software projects that require heavy engineering? I don't think so.

philwelch 16y ago
Your comment reminds me of this: http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s... http://pindancing.blogspot.com/2009/09/sudoku-in-coders-at-work.html http://pindancing.blogspot.com/2009/09/sudoku-in-coders-at-w... It's a discussion of two people, Peter Norvig and a TDD enthusiast, both building a Sudoku solver. Norvig takes a no-nonsense AI approach and the TDD enthusiast flails for five blog posts without getting anywhere. To me, the moral of the story isn't to mock TDD, but to point out that different problems require different approaches. TDD can't be naively applied to an algorithmic problem the way it can be applied to a nuts-on-bolts web app, for instance.

spenrose 16y ago

See also: "Evidence-based Software Engineering": http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113.571&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.113... and the work of Greg Wilson: http://www.slideshare.net/gvwilson/bits-of-evidence-2338367 http://www.slideshare.net/gvwilson/bits-of-evidence-2338367