10 ms·
My 20-Year Experience of Software Development Methodologies
- zwischenzug 9y agoAuthor here: interested in any strong views on this - @ianmiell on twitter.
- k__ 9y agoGood writeup :) I did the whole rollercoaster only for 11 years now, but I also was in different companies from startup to big corp. Some people are totally into their methodology and won't do a step aside the predefined path. Some people are totally elusive to process. They will talk to you via email, Skype, WhatsApp and JIRA and you won't find any infos later. I have the feeling there is no methodology for devs to work better, only for devs to get peace from non-devs who work with them.
- Jare 9y agoMe and my brother wrote our first commercial videogame in 1987 as a 16-17yo pair of self taught programmer/artist/designers. We just did what seemed like common sense. - We had periodic meetings with the client (publisher). we showed them progress and they suggested directions to move towards - We didn't stick religiously to the initial design, and adapted as we saw things work or not work to make the game fun - We stuck to the deadline and just cut what didn't make it It felt quite agile to me.
- georgeecollins 9y agoThat sounds like heaven. What was the game?
- erikbye 9y agoStar Dust?
- Jare 9y agoErikbye is correct, Stardust for Topo Soft on the Sinclar Spectrum and Amstrad CPC 8-bit computers. I attribute a lot of the merit to us being mentally organized, but mostly to our producer Javier Cano (RIP), who had that magic touch for ensuring that things around him just ran smooth, always with a smile and making it feel natural. My god he was amazing.
- iridium 9y agoSlightly OT: The biggest collective fiction today is probably this idea that as tech person, we can change the world, and as a result we should try to change the world. Anything less impactful feels like a waste of potential. The general society has bought into it - because we've been hearing this for years: Johnny - you are such a smart programmer, where is your world changing app ? Among other things, this translates into working ridiculous hours in expensive cities and convincing a set of investors that they too can be part of this world changing process in exchange for a few shekels. Now these shekels will drive innovation, and they will push civilisation forward. But the magic of the fiction is that poor Johnny will never feel like he did much to change the world, and so he will keep trying ad infinitum.
- zwischenzug 9y agoYes, capital is still king. However, I would say that Zuckerberg (for example) did achieve a lot without selling his product for capital. That is pretty magical.
- humanrebar 9y ago> ...without selling his product for capital... Maybe I'm missing what you mean somewhere in there, but facebook raised a lot of capital through investment rounds and an IPO.
- zwischenzug 9y agoHe retained control of the company (is what I meant).
- georgeecollins 9y agoHaving lived through a variety of methodologies, all of which were taken very seriously and equally successful, this rings true. The one thing I would maybe add is that the tools of the trade have changed for the better in thirty years, particularly source control management, document tracking, and the ability to collaborate remotely. I think ways of development have changed in response to that. Imagine a world where compiling a build of a piece of software took as a long as an hour, source code was stored on a single server so that you didn't even want to have two people working on the same set of code, automated testing didn't exist.. I can remember cases of that in my youth. It seemed to call for more planning and a more top down approach.
- zwischenzug 9y agoThanks! Yes, I remember when I deployed to live money-taking systems using tar files as my source control. Collaboration tools have completely changed the game, totally agree. As has the death of the physical artifact (that couldn't be patched, recalled, or 'bounced').
- dahart 9y agoI'm not sure I have any particularly strong views to share, but a couple of notes. I felt like "collective fiction" was a harsh term that implies we're always lying to ourselves. While that might be true, I like to use the more ambiguous term "narrative" for the same concept. The need for a methodology is a narrative or story that may or may not be true. I guess I do believe ;) that some of our collective narratives are definitely true, and calling them fiction undermines the good ones, or at least makes it harder to evaluate which collective beliefs are more valuable & closer to truth and reality than others. Side note - collective narratives are a human condition, not in the least limited to programming methodologies. We build collective narratives for everything, and our physiology is prone to it. Having kids, it was fun watching them repeat stories they hear about the world over and over, as a way to cement their narratives with their friends and parents. When they were younger it was almost in the form of a question, they were waiting to be challenged or get more specific information. As they grow older, they get more firm in their statements and beliefs. After watching them do it, I became more aware of how often adults do the same thing, and how pretty much all communication, even technical communication, is just narrative. My own 20 years of experience with methodologies and with other programmers has led me to believe that pushing coders to communicate frequently about what they're doing and what needs to be done is a good thing; most programmers I've known (including me) tend to wander in directions that they want to go and don't naturally want to do all the things that really need to be done. I sometimes don't like being transparent about what I'm doing, sometimes because I'm working on a pet project that I shouldn't be working on, sometimes because I hate budgeting and I don't know when I'll be done, sometimes because I want to deliver a higher quality than was requested and I know I'll be asked to stop being a perfectionist and just check in my changes. I see others not wanting to be transparent about what they're doing for the same reasons. But making sure that transparency exists is probably the only thing I've seen that works. Checking in often, making sure incremental progress is going the right direction, and re-iterating the requirements and users' point of view. Too much formal process is usually bad unless you're NASA, and by the time it's called a "methodology", it's probably too much. It gets in the way of a lot of the good things that need to happen. Two week sprints get exhausting after a while, it teaches devs to aim smaller, and it causes undue burdens from management on tasks that legitimately take a couple of months. Being forced to estimate everything in story points has some advantages, but ultimately makes measurement harder, it's subjective and easily manipulated, and it's just contrived anyway. So, I agree about formal methologies not working well, by and large, but I think the basic intent to keep focused and communicate frequently are good goals. It is ironic that most software methodologies are reasonably good at helping people evaluate how to make certain kinds of focused technical decisions, but they're very bad at helping people evaluate when a formal process is wasteful or bad, or to suggest a different formal process.
- whipoodle 9y agoI work for one of those fancy startups. Everything is chaos all the time, you have all these extremely smart people in the small doing actually kinda dumb things in the large. This worked better when the company was small and is causing more and more problems as we grow. Anyway, the point being that there’s lots of smart individuals and little process. I mentioned this during a meeting with my team, and so a process was devised for the next project we embarked upon. The process was waterfall, except nobody referred to it that way (most of the people are like 25, and weren’t working back when it was the norm). Everything old is new again. It wasn’t important anyway; we didn’t follow the process.
- dmytroi 9y agoNice post! I worked in a few big and small companies, and from I can see software companies with more then ~40-50 devs don't scale well (I only talk about about single-ish product software companies here). The reason for this is in my opinion is absence of financial liability of devs: if you're a dev in a company with 500 other devs - most likely you have zero financial liability to produce great code, on other hand you have all time in the world to create job security code around you. Also if there is 500 devs on one project, most likely one will see a lot of infrastructure just to get this devs busy with something. This could be seen in companies were one department creates "the framework", and others departments create "features" based on this framework. In this case there is no "market" of frameworks - you basically stuck with it, and "the framework" has zero market competition. Compare it to normal market of middleware - if your framework is bad, I'll just go to your competitor. Free market is much healthier situation in my opinion (could be a strawman argument though). I am not wise enough to say that creating internal competition inside one company is always good thing, though financial contracts make a healthier political discussions.
- emodendroket 9y agoI don't believe job security code really works. If they're in a place where they're not paying any attention to you committing awful code why would they care when developers tell them that only you can work on your code?
- humanrebar 9y agoThe job security comes when you write great code but awful designs and architectures. The resulting objections are more abstract and harder for project managers to really understand. Or they'll understand the objections but not know how to weight their severity.
- emodendroket 9y agoThey'll just ignore them and do what they please, is where I'm going with this.
- mmjaa 9y agoMy 30 years of Software Development Methodology: * The user is more important than any of this. * The programmer is nothing. * The computer is doing all the work. Thus: use what makes all the above, True.
- icebraining 9y agoWhat does "the programmer is nothing" mean?
- tluyben2 9y agoAnd "the computer does all the work"?
- georgeecollins 9y agoI'm guessing: The machine does what the user (client and org) expects effectively in a way that they easily understand.
- hoorayimhelping 9y agoAutomate manual processes
- georgeecollins 9y agoI'm guessing this means, if the software works and can be maintained as the users (by that I mean client and the owner) needs it to be maintained, then all considerations are satisfied. Other considerations of aesthetics, architecture, language should be set aside.
- mmjaa 9y agoYup, you get it.
- georgeecollins 9y agoBy the way: I think this is brilliant.
- sytelus 9y agoThere is an insider story about how these methodologies comes about. So there are few groups of people whose sole job is to do consulting on failed/late/over budget projects. Mind you, they don't write code but rather they observe how things are going and then prescribe process/management improvements (McKinsey style). Once in a while, these folks bump in to terrible projects and whatever they prescribed sometime works like a charm. In that case, they take that prescription on road and advertise the hell out in conferences, magazines, blog posts. Unlike regular developers, they have all the time in the world to do these activities. They write books and give interviews and by the media power of distributing information suddenly they pop out as process gods who knows how to fix any project. Eventually the new things starts to fad, people realize what works in project X didn't worked in Y, speaker engagements starts drying out and then these folks need new thing to repeat the cycle. The obvious problem is that these folks prescribing the development process are not active developers. They are not even part of any real project over any long duration. They are in job of inventing and selling processes and handing out management advice as consultants. Whatever they prescribe, might have worked only in specific context and for specific symptoms, usually with huge dose of luck. Next time when you see new process fad, look up the history of originator of this process, what company he is part of, how much code has he written, how he makes money. You will know what I'm talking about.
- mmcnl 9y agoThis is not true for agile. The agile manifesto was in fact drafted by software developers. Also, agile doesn't prescribe any process at all, in fact, it does the opposite. Agile in essence is quite beautiful, unfortunately it gets twisted and turned upside down until it's just another methodology (which is exactly the opposite of its original meaning).
- baybal2 9y agoThere are no need for formal methodologies. A person with no technical expertise should not be but on a position managing technical teams. That requirement divides the industry on "tech companies" and everybody else. The example of a tech company with gigantic dev teams and very little number of MBA doctrinaires involved in development process is Intel. Companies with external development clients that practice "we will agree on any methodology you want for right amount of money, but we would not let you anywhere close to our developers (who will be doing their own thing anyways)" are Tatas, Wipros, Luxofts and others. Their biggest asset is their skill at managing the client.
- walshemj 9y agoAh and how would say produce a management system for a core part of a country's core data network (I was a team member developing a management system for the UK's SMDS network) with out a process? - In this case we used a hybrid waterfall /RAD system 12 weeks to develop the fist release
- baybal2 9y agoThe very very few cases where anything called a methodology works is when you deal with a repeating tasks: making a customized versions of standard tools, or what people call copy&paste development. SAP does it very well. Charges people 7 digits for "customized solution" that in reality is not customized beyond 20 lines of code from what the company sells to every sixth or seventh client. The goal there would be to not to dive too deep into the task that is already solved in a way that "works well enough," and for what the client is already ready to pay money for.
- walshemj 9y agoNot everyone always works on me to crud applications leaving aside SAP and other "miracle" tools :-)
- Clubber 9y agoI think the key word is "formal," to me, meaning a strict process to be used at all times in all situations. Like anything in development, it makes better sense to use whatever process works best for whatever team is implementing whatever solution is needed.
- GrumpyNl 9y agoAll these methodologies are there just to get things done. After many years involved in software development i have find out that it all comes down to leadership. Of course you need people with skills, but no good leader, no good product.
- siisewed9eddd 9y agowa1assdsudsywss
- siisewed9eddd 9y agou h
- baybal2 9y agoThe biggest skill when interacting with a "big serious man" type of client is to making him pay you for what you want him to pay you for. In my circle, we have a lot of discussions on clients like that: "and I told him, for what freaking thing do you need that freaking neural network when a banal statistics does the job better?". Having such guy, "a Wharton grad with stellar track record at the big 4" agreeing on the former requires compromising his ego, self image, exploiting him wanting to not to look incompetent in eyes of his superiors. In the end, all such guys are made to pay 6 or even 7 digit sums for a fancy UI over database querying tools and a stats package, and not a "big data, machine learning blah blah blah," or otherwise being given "the exact thing they wished for" that had no chance of even minimal functioning
- jacquesm 9y agoBut how will his company make it to 100M+ valuation if he can't claim it is being done with AI?
- tluyben2 9y agoWe use 'waterfall' (detailed, fleshed out specs in formal language) on the hardware, firmware & backend and 'agile' on the frontend (web/app). So far that served us well and it addresses the difference in developer skills and needs as well. With the danger of generalizing; frontend people are different from backend people who are different from hardware people (in my experience), and this gives all what they enjoy working with.
- _puk 9y agoHardware (and firmware to a lesser degree) almost require waterfall. Moving fast and breaking things doesn't really work well when you have to throw away the physical product you broke as opposed to iterating it to a fixed state. I can actually see the merit in waterfall for elements of the backend.. though how do you avoid pre-optimisation or even complete wastage building out APIs that end up never being needed?
- kasey_junk 9y agoThe two single biggest factors in determining what sort of process you should have are a) cost to ship/deploy and b) cost of mistake. On one extreme are things that are expensive to change and mistakes cause critical failures and on the other are things that are trivial to change and mistakes are easily mitigated.
- ozmbie 9y agoI've worked in a variety of environments, and the most important thing in my experience is simply to ensure that your team is creating a working, tested, and "shippable" update every ~2 weeks. It amazes me how many teams fail to do this, despite calling themselves "agile" and "customer driven". Use whatever methodology, technology, and processes you need to accomplish that singular task. Orient your entire organisation around doing that. Your customers don't care how you do it. They just want frequent updates and progress. And if you're not giving them useful updates (you'll know, they'll tell you) then the problem isn't methodology, it's probably your ideas. No methodology will fix that.
- amelius 9y agoCustomers want updates? I want my software to stop changing!
- tfigment 9y agoDepends on the product but indeed I want stability over all else with updates for security. We have a product sold to us 2 years ago and still hasn't passed UAT. It is enterprise and they use agile so after every 3 to 6 weeks we get a new build to verify. There is always something new that fails so we go back to waiting. At this point I wish I could write the thing ourselves as I would have been done 18 months ago.
- savrajsingh 9y agoHow about every week? :)
- maxxxxx 9y agoThis only works in certain kinds of projects. I often work on projects where you need to spend months figuring out some technologies and see what works best. For a while we tried to deliver "shippable" updates every 3 weeks but this just created a lot of work creating artificial prototypes without any real benefit so we gave up. If you work on some web site where the tech is well understood then shipping something every 2 weeks probably works.
- kenshi 9y agoMethodologies are a trap. The problem with any methodology is they tend to lead people to stop thinking about the context they are operating in. "The methodology says X, we should do X, why oh why aren't we doing X?" is a familiar lament in the development world. I have been there and done that, to be sure. It really isn't going to matter what the Methodology says, if something or someone in the context means it isn't being adhered to. If the person ultimately paying for the project isn't convinced by the Methodology, or wants to deviate from it badly enough, guess what? The Methodology will be ignored. You should always advocate for what the Right Thing To Do(tm) is, but you should ask yourself first: what is the most important criteria that determines what the Right Thing is? Sometimes writing the quickest, hackiest, throwaway code can be the Right Thing. It can be a horrifying truth for a software developer. Sometimes doing the ballsy re-write is the Right Thing. Sometimes letting a project fail is the Right Thing. Sometimes last weeks context and decisions no longer apply. More critical to project (and personal) success than any Methodology is gaining a thorough understanding of the context in which you are working in, as quickly as you can. And then doing all the Right Things that will make you effective within those constraints. This means there is no Single Truth for software development. No easy answers. But there are a wide variety of principles you can draw from and apply and discard as needed. Or as Bruce Lee put it: Be like water.
- oldpond 9y agoThe Single Truth about software development is that there is only one deliverable: working software.
- mmcnl 9y agoI would take that one step further and say the only deliverable is satisfied customers.
- zwischenzug 9y agoI would take that further and say the only deliverable is profit. Or more VC :)
- emodendroket 9y agoThe style he describes as RAD is really enjoyable to work in. No constant scheduled meetings and unread documents; just use your common sense and write something.
- zyxzevn 9y agoI like the car analogy, and I had additions to it with software-components in mind: engine-choices representing data storage: very big engines. for big ships. very small ones, for lawnmower. self-made ones, smoking. gear-choices representing libraries: single gear gear from 1 to 1000. No neutral and no reverse. chassis choices (language choice): rusty old ones, hard to work with, but lasts. new shiny ones, overpriced and falls apart. locks = safety: one key fits all (all doors and all cars). different key for each lock, can be bypassed by opening window. keypad with password, written on the back-side of the car. interior = user interface: millions of choices, all flashy and shiny, each part looks totally different. Unusable steering. break (software stability): no breaks. when used causes blue-screen outside (program output): all parts have different sizes. holes covered with duck-tape and rust covered with shiny paint. Enjoy the ride!
- richie5um 9y agoReally liked this article. I liken software methodologies to cooking recipes. On one end of the scale you have a highly skilled and experienced chef, who can craft a great (requested) dish with a variety of ingredients and equipment. Knowing how to adapt and what to add and when. This works well, but requires a heavy cost (experience) and is, from the outside, difficult to follow and control. On the other end of the scale you have fast food. A strict process and fixed ingredients that ‘just works’. Those involved in cooking have very little knowledge of why, and are almost certainly not able to adapt. This works well for scenarios where you exactly know and control both the input and output. A skilled chef can ‘codify’ their recipes, but this results in a specific description of what to do, and not why. Thus marginalising the adaptability. In my experience, most software engineers are trying to attain the skilled chef role. While most _managers_ want the fast-food style predictability. With the ‘process’ being the battleground. The best process is where you - a complete multi-disciplinary team - make as quick iterations as possible, learning as you go. But, always knowing where you are trying to get to (and why!) and constantly adjusting accordingly. Good luck out there.
- hshhbd 9y agoI really enjoyed reading this and I strongly agree there isn't an easy task at all. Depends on the business, people, existing processes, it's up to us to find what really works. I had some thoughts based on my recent experience and went into a mood to write something quite long on my LinkedIn https://www.linkedin.com/pulse/waterfall-scrum-kanban-scumban-whats-next-richard-shenghua-he https://www.linkedin.com/pulse/waterfall-scrum-kanban-scumba... if anyone interested.
- hshhbd 9y agoSorry, had a typo in the previous link, please use this one https://www.linkedin.com/pulse/waterfall-scrum-kanban-scrumban-whats-next-richard-shenghua-he/ https://www.linkedin.com/pulse/waterfall-scrum-kanban-scrumb...
- jeff6845 9y agoI've not found a methodology yet that can displace smart decisions. Methodologies organize projects to help further smart decisions and avoid dumb ones, but smart decisions and skill still dictate the success or failure of projects. In the end, the people are still more important than the methodology, but I've encountered a mentality a few times where people believed a good methodology will fix all. I've seen companies keep changing methodologies looking for success with largely the same staff, and key decision makers, but yet the results are the same. Insanity, anyone?
- z3t4 9y agoThe fiction I like best right now is "continuous delivery": You release often, push to production several times per day, one working branch, always ready to deploy. Features being in development are behind flags. Regression tests help a lot, eg, when you make an update or fix a bug you make an automatic test that checks if the new thing works, or repeats the steps that caused the bug and makes sure it's no longer there.
- shoover 9y agoI’m surprised that Don Reinertsen’s book The Principles of Product Development Flow doesn’t come up in these discussions. Rich Hickey mentioned it once after a conference years ago and that’s the only time I’ve heard it mentioned. I’ve since found a few blog posts responding to conference talks (of which there are several on YouTube from around 2015), but not many. I think people simply don’t know about it. I wrote it down at the time and finally digested it over the past several months. It’s an incredible book that I think would resonate with many of the commenters here. I think of Reinertsen’s approach as industrial agile, in the most honorable sense of industrial. There is theory and some math. There is heavy emphasis on project economics as a means of making complex trade-offs and a basis for training, deputizing, and expecting employees at all levels of an organization to make better decisions on the fly as part of doing their jobs. Lessons are drawn from many fields, such as network communications, queueing theory, and the Marines. Contrasts are drawn with manufacturing to explain exactly why some of the techniques we adopt do not apply in product development but where many techniques from kanban systems do apply. The book is organized around hundreds of principles given as tools, to be considered and applied as needed to your specific situation. Annoyingly, he refuses to get into implementation. At first I thought he was saving pages, but the more I think about I realize the seeming omission was necessary genius. Digging into even one example implementation would result in the death of the message and framework as people copied that and get dogmatic about a methodology all over again. The whole presentation is beautifully constructed and tight. It puts a perspective and language on things in a way that makes me hopeful and interested in project management, similar to how Rich Hickey’s talks raise my understanding, vocabulary, and aspirations for software engineering. I would welcome anyone involved in prioritizing commercial software projects to read and discuss this book (and document their implementations).