7 ms·

The consultants have taken over "Agile" No, what's happened is that BigCorp has decided to adopt Agile and won't adopt anything without a lot of instruction bo

The consultants have taken over "Agile" No, what's happened is that BigCorp has decided to adopt Agile and won't adopt anything without a lot of instruction books and role descriptions. The consultants, at least the ones who know what they're doing, have been trying to fight this all along. It's the industry itself that continues to over-engineer everything it touches, from Javascript frameworks to development processes. What we keep finding is that you can't keep people from themselves, that is, if you don't give somebody a 47-page instruction book full of diagrams and processes they want, somebody else will. I really, really wish the problem with Agile was simply the consultants. Instead, as we've seen with process after process, we keep screwing it up, feeling that the more of something we have, the better it must be. Consultants, tool vendors, trainers, and the like? They're just adjusting their marketing pitch to what people want to hear.

sidlls 9y ago
It isn't just a "BigCorp" organizational issue. You touched on it, but gently: developers share some blame here. There's a real anti-engineering bias in this industry. I mean engineering as it's practiced in other contexts. There is a process of design and analysis that is done before build and implementation starts that in this industry is considered wasteful or work to be done by lesser mortals who aren't smart enough to code. That's a big reason why there's all this over-engineering of everything.
- jerf 9y ago
  I am honestly having trouble parsing through how you simultaneously accuse engineers of being too lazy to use process and then agreeing that there is too much process. I don't engineer as other engineering disciplines do, because my world is not like other engineering disciplines. If architects could push a button and see a skyscraper built in five minutes, then push another button to tear it down and build it with a slightly tweaked again, for basically no cost, often while clients are actually living in the building (!), you'd see them doing a lot less up-front planning from them, too. Stop having "real engineering" envy. As the characteristics of software are added to other engineering fields, they start engineering more like us, because that's the right answer. No, we are not in fact specially stupid or lazy, which is exactly the same mistake as thinking we are specially smart or industrious. We're neither, and we do the things we do for reasons every bit as good as the reasons for their processes.
  - sidlls 9y ago
    I'm not accusing anyone of being lazy, rather inappropriately arrogant and consequently ignorant. Also, I don't agree there's too much process. I think there's too much bad process that isn't recognized as either bad or process. I strongly disagree with this: "As the characteristics of software are added to other engineering fields, they start engineering more like us, because that's the right answer." It's the wrong answer, and we get things like these recent Tesla and Uber incidents because of this attitude.
    - jerf 9y ago
      "recent Tesla and Uber incidents because of this attitude." That is not what I'm talking about; that's software engineers operating in non-software fields without taking in the appropriate expertise. There's nothing special about software there, excepting perhaps that because software is everywhere our field has greater exposure to that mistake, but I've read all sorts of stories where people didn't respect the expertise of people who came before them and made grave errors as a result. That's not even limited to engineering. (In fact it's a well-known common mistake of youth across the board....) Ironically, the automotive industry is actually the one I had in mind when I wrote that, because I live in Michigan, and personally know some automotive engineers, both on the design and testing sides. We've had several conversations over the years as to how their design processes have changed now that they can plug designs into the computers and do thing like finite element analysis in the computer before they build anything physically. In the 1980s, booking the crash chamber was really hard, because it was in constant use. Now it's easy to get into. They can't close it, because the real automotive engineers know they need the validation of reality, but they do many fewer crash tests than they used to. Because of the introduction of more of this software into their process, they're doing a lot less of the up-front planning that you might think they are doing, which they used to do, in favor of more exploratory engineering and experimentation in the computer before making anything physical. And you can see the results in modern cars... if you know how to look, anyhow. A lot of it is invisible to the end consumer [1], and some of the engineering gains get somewhat eaten up by additional requirements for safety and such, but a modern car is not just "sorta nicer than they used to be", they are orders of magnitude more sophisticated than they were 30 or 40 years ago, and a lot of that is a direct impact of the fact that they are now able to adopt more software-engineering-like processes into some parts of their engineering. If it weren't for that power, we'd never have cost-effective cars that also meet the safety and emissions requirements. [1]: "This car is quieter on the road because this strut is here instead of 1.5 cm closer to the front, which is why it has this weird bend in it." Not even an exaggeration. If you look around, you'll also see these weird bits of gluey stuff stuck to a piece of metal or something that doesn't seem to even be gluing something. Those are often for noise or vibration. Some of them are empirically discovered during testing, some during simulation. There's all kinds of little details like that that you don't notice if you don't know what you're looking for.
- GiorgioG 9y ago
  > There's a real anti-engineering bias in this industry. What we do is hardly real engineering in most cases. I don't mean medical or transportation systems. In BigCorp, most of us write line of business applications, which will generally speaking not directly kill anyone - unlike designing/engineering bridges, roof systems, etc.
  - subway 9y ago
    I've watched the "my product isn't safety critical" line of thinking utterly fuck some companies. Any recommendations on which of your projects I can keep an eye on?
    - GiorgioG 9y ago
      I've done work for organizations where the practices ranged from extremely strict processes, to no process at all (from requirements gathering to devops.) I feel my best experiences were always in places that had some process that was developed over time, that had management in place that was willing to experiment with changes to said processes. But the way Agile is bought into at most big companies these days...it feels like management just wants to be lazy: "follow the process because hey that's what these smart guys say worked over in org X" even if it's a waste of time/resources/etc.
  - zbentley 9y ago
    Part of the reason this becomes so dangerous is myopia. If I'm "just building an internal issue tracker", sure, I don't need to spend ages securing it . . . until someone else exposes it to the open internet because that's the only way to make it available to offshore partners (which in turn is because someone in networking made the--from their perspective reasonable--decision not to pay for the setup of a VPN because it was expensive). Each person is making more or less reasonable decisions, but the way they interact with the rest of the whole is what causes disasters and inflicts real harm on people's lives. A question I'm interested in is: "real" engineering has the same issue of scale and potential myopia. Consider the guy mixing concrete at the batch plant on a tight schedule who, faced with an impending shortage of aggregate, mixes a little less into his mud--he does this so the trucks lining up to fill won't have to wait for the other truck bringing more aggregate. Now, his concrete was headed for a bridge, which is genuinely structurally compromised by the lack of aggregate. Do things like this happen in other fields of engineering with anywhere near the same frequency they happen in software? Why or why not? (In the concrete example--and, to be honest, this is just 'cause I like talking about concrete--there is no "right" answer given the shortage. "Fatten" the mud by lacking aggregate? Structural issues. Let trucks wait? Jobs with unfinished pours may have premature cementing, which, unless their contractors are willing to bite the "tear down the partial pour; it's too cemented--wreck it, rebuild the forms, start over" bullet which pretty much always ends up pointed at them and not the batch plant, leads to . . . you guessed it, structural issues).
- jdmichal 9y ago
  > There is a process of design and analysis that is done before build and implementation starts that in this industry is considered wasteful or work to be done by lesser mortals who aren't smart enough to code. That's called "backlog refinement" in scrum. And it's supposed to be done by the engineers.
- commandlinefan 9y ago
  You make it sound as though I could do all of this up-front design and analysis work, and it would all be time well spent, and it would save time and effort in the long run, and it would produce a better end-product and the customers and my boss would be happier with the results, but I'm not doing that because... I'm just difficult that way? I'm a lazy, coddled, spoiled, petulant developer who'd rather play with his toys than put on his big-boy pants and do some real work? Pardon me while I bristle with indignation here. First of all, under the repressive regime of "agile" methodologies which demand that I: say exactly what I'm going to do and exactly how long it's going to take and provide ongoing evidence that I did exactly what I was going to do and that it took exactly how long I said it was going to take, have something to demo every single morning to prove that I wasn't wasting the whole day goofing around on Facebook, something open-ended like design is an impossible pipe dream. What does a "design" even look like? A word document? A visio diagram? A powerpoint? In this micromanaged, "I know that you programmers can accomplish any task of any difficulty of any level of lack of definition in any compressed timeline I can think of, but I know you're not doing it because you're a bunch of overeducated spoiled brats" management environment, we've gotten the message loud and clear that we had better do whatever produces what appears to be working software as fast as possible, regardless of how inefficient or anachronistic it might be because in spite of the supposed "talent shortage", every one of us is one angry boss away from unemployment and a line of skeptical prospective employers who don't care how much education or experience you have, but do you know Angular 5? Because we only use Angular 5.
  - TheCoelacanth 9y ago
    WTF kind of Agile process is that? I've never heard of anyone having daily demos. The processes I've seen schedule them for every 2-3 weeks. If you're referring stand-up meetings, the people involved in those aren't supposed to be managers, they are supposed to be the people who you are directly working with.
    - DanielBMarkham 9y ago
      Sure. You can have daily sprints if you like --- if you know what you're doing. The rule with frameworks is this: the more time you spend thinking and screwing around with the framework, the more you're either doing it wrong or it's hurting more than helping. Scrum/Agile should just work, just like your C compiler just works. It's a tool that operates almost invisibly in the background while you make stuff people want. If it takes a long time to work with it, or you spend a lot of time fussing with it or arguing over whether you're doing it right or not? You've got something backwards somewhere. People end up with long sprints and tons of meetings either usually because they don't understand what they're doing or they're doing it the wrong way. Personally I prefer week or less sprints. Anything else tends to grow a bunch of meeting BS over time that's non-productive.
jerf 9y ago
It seems to me one of the human irrationalities we don't hear much about is the widespread belief (proved by actions) that process and bureaucracy is free, and therefore, when analyzing a new process, it is not necessary to consider what the costs of the process are. Once you set the cost incorrectly to zero, it only takes some very vague, theoretical, handwaved benefits to tip apparent value of the process positive, and that is a very low bar. So there is an enormous bias in humans for instituting processes. We almost never consciously get rid of them, either, because even after we've been paying the price of processes for years, somehow we still don't account for them as a cost for that specific process. They're just the cost of doing business or something like that, vague ambient costs that don't attach to anything specific somehow, mysteriously. Something that nobody can get a cognitive handle on and start thinking about clearly. You'll note I haven't limited this in scope to software engineering at all. It's not hopeless, though, because people can be educated (for lack of a better word) into properly accounting for costs. For some people it can be a "why didn't I realize this years ago" sort of revelation. If you manage to start communicating in this way properly, it's also a bus that engineering and the business management can start communicating with each other productively. They don't want to hear about the details of your technical debt, and you probably don't want to hear about the details of how they are refinancing their debts or whatever; you need to trust them to drive the business and they need to be able to trust you to offer them solid cost/benefit analyses of technical decisions, and offer them the correct suite of options (some they may not have considered), and while that may not bring heaven on Earth, at least these two traditionally disparate worlds can be communicating based on cost/benefit terms.
deleted 9y ago
[deleted]

sidlls 9y ago

It isn't just a "BigCorp" organizational issue. You touched on it, but gently: developers share some blame here. There's a real anti-engineering bias in this industry. I mean engineering as it's practiced in other contexts. There is a process of design and analysis that is done before build and implementation starts that in this industry is considered wasteful or work to be done by lesser mortals who aren't smart enough to code. That's a big reason why there's all this over-engineering of everything.

jerf 9y ago
I am honestly having trouble parsing through how you simultaneously accuse engineers of being too lazy to use process and then agreeing that there is too much process. I don't engineer as other engineering disciplines do, because my world is not like other engineering disciplines. If architects could push a button and see a skyscraper built in five minutes, then push another button to tear it down and build it with a slightly tweaked again, for basically no cost, often while clients are actually living in the building (!), you'd see them doing a lot less up-front planning from them, too. Stop having "real engineering" envy. As the characteristics of software are added to other engineering fields, they start engineering more like us, because that's the right answer. No, we are not in fact specially stupid or lazy, which is exactly the same mistake as thinking we are specially smart or industrious. We're neither, and we do the things we do for reasons every bit as good as the reasons for their processes.
- sidlls 9y ago
  I'm not accusing anyone of being lazy, rather inappropriately arrogant and consequently ignorant. Also, I don't agree there's too much process. I think there's too much bad process that isn't recognized as either bad or process. I strongly disagree with this: "As the characteristics of software are added to other engineering fields, they start engineering more like us, because that's the right answer." It's the wrong answer, and we get things like these recent Tesla and Uber incidents because of this attitude.
  - jerf 9y ago
    "recent Tesla and Uber incidents because of this attitude." That is not what I'm talking about; that's software engineers operating in non-software fields without taking in the appropriate expertise. There's nothing special about software there, excepting perhaps that because software is everywhere our field has greater exposure to that mistake, but I've read all sorts of stories where people didn't respect the expertise of people who came before them and made grave errors as a result. That's not even limited to engineering. (In fact it's a well-known common mistake of youth across the board....) Ironically, the automotive industry is actually the one I had in mind when I wrote that, because I live in Michigan, and personally know some automotive engineers, both on the design and testing sides. We've had several conversations over the years as to how their design processes have changed now that they can plug designs into the computers and do thing like finite element analysis in the computer before they build anything physically. In the 1980s, booking the crash chamber was really hard, because it was in constant use. Now it's easy to get into. They can't close it, because the real automotive engineers know they need the validation of reality, but they do many fewer crash tests than they used to. Because of the introduction of more of this software into their process, they're doing a lot less of the up-front planning that you might think they are doing, which they used to do, in favor of more exploratory engineering and experimentation in the computer before making anything physical. And you can see the results in modern cars... if you know how to look, anyhow. A lot of it is invisible to the end consumer [1], and some of the engineering gains get somewhat eaten up by additional requirements for safety and such, but a modern car is not just "sorta nicer than they used to be", they are orders of magnitude more sophisticated than they were 30 or 40 years ago, and a lot of that is a direct impact of the fact that they are now able to adopt more software-engineering-like processes into some parts of their engineering. If it weren't for that power, we'd never have cost-effective cars that also meet the safety and emissions requirements. [1]: "This car is quieter on the road because this strut is here instead of 1.5 cm closer to the front, which is why it has this weird bend in it." Not even an exaggeration. If you look around, you'll also see these weird bits of gluey stuff stuck to a piece of metal or something that doesn't seem to even be gluing something. Those are often for noise or vibration. Some of them are empirically discovered during testing, some during simulation. There's all kinds of little details like that that you don't notice if you don't know what you're looking for.
GiorgioG 9y ago
> There's a real anti-engineering bias in this industry. What we do is hardly real engineering in most cases. I don't mean medical or transportation systems. In BigCorp, most of us write line of business applications, which will generally speaking not directly kill anyone - unlike designing/engineering bridges, roof systems, etc.
- subway 9y ago
  I've watched the "my product isn't safety critical" line of thinking utterly fuck some companies. Any recommendations on which of your projects I can keep an eye on?
  - GiorgioG 9y ago
    I've done work for organizations where the practices ranged from extremely strict processes, to no process at all (from requirements gathering to devops.) I feel my best experiences were always in places that had some process that was developed over time, that had management in place that was willing to experiment with changes to said processes. But the way Agile is bought into at most big companies these days...it feels like management just wants to be lazy: "follow the process because hey that's what these smart guys say worked over in org X" even if it's a waste of time/resources/etc.
- zbentley 9y ago
  Part of the reason this becomes so dangerous is myopia. If I'm "just building an internal issue tracker", sure, I don't need to spend ages securing it . . . until someone else exposes it to the open internet because that's the only way to make it available to offshore partners (which in turn is because someone in networking made the--from their perspective reasonable--decision not to pay for the setup of a VPN because it was expensive). Each person is making more or less reasonable decisions, but the way they interact with the rest of the whole is what causes disasters and inflicts real harm on people's lives. A question I'm interested in is: "real" engineering has the same issue of scale and potential myopia. Consider the guy mixing concrete at the batch plant on a tight schedule who, faced with an impending shortage of aggregate, mixes a little less into his mud--he does this so the trucks lining up to fill won't have to wait for the other truck bringing more aggregate. Now, his concrete was headed for a bridge, which is genuinely structurally compromised by the lack of aggregate. Do things like this happen in other fields of engineering with anywhere near the same frequency they happen in software? Why or why not? (In the concrete example--and, to be honest, this is just 'cause I like talking about concrete--there is no "right" answer given the shortage. "Fatten" the mud by lacking aggregate? Structural issues. Let trucks wait? Jobs with unfinished pours may have premature cementing, which, unless their contractors are willing to bite the "tear down the partial pour; it's too cemented--wreck it, rebuild the forms, start over" bullet which pretty much always ends up pointed at them and not the batch plant, leads to . . . you guessed it, structural issues).
jdmichal 9y ago
> There is a process of design and analysis that is done before build and implementation starts that in this industry is considered wasteful or work to be done by lesser mortals who aren't smart enough to code. That's called "backlog refinement" in scrum. And it's supposed to be done by the engineers.
commandlinefan 9y ago
You make it sound as though I could do all of this up-front design and analysis work, and it would all be time well spent, and it would save time and effort in the long run, and it would produce a better end-product and the customers and my boss would be happier with the results, but I'm not doing that because... I'm just difficult that way? I'm a lazy, coddled, spoiled, petulant developer who'd rather play with his toys than put on his big-boy pants and do some real work? Pardon me while I bristle with indignation here. First of all, under the repressive regime of "agile" methodologies which demand that I: say exactly what I'm going to do and exactly how long it's going to take and provide ongoing evidence that I did exactly what I was going to do and that it took exactly how long I said it was going to take, have something to demo every single morning to prove that I wasn't wasting the whole day goofing around on Facebook, something open-ended like design is an impossible pipe dream. What does a "design" even look like? A word document? A visio diagram? A powerpoint? In this micromanaged, "I know that you programmers can accomplish any task of any difficulty of any level of lack of definition in any compressed timeline I can think of, but I know you're not doing it because you're a bunch of overeducated spoiled brats" management environment, we've gotten the message loud and clear that we had better do whatever produces what appears to be working software as fast as possible, regardless of how inefficient or anachronistic it might be because in spite of the supposed "talent shortage", every one of us is one angry boss away from unemployment and a line of skeptical prospective employers who don't care how much education or experience you have, but do you know Angular 5? Because we only use Angular 5.
- TheCoelacanth 9y ago
  WTF kind of Agile process is that? I've never heard of anyone having daily demos. The processes I've seen schedule them for every 2-3 weeks. If you're referring stand-up meetings, the people involved in those aren't supposed to be managers, they are supposed to be the people who you are directly working with.
  - DanielBMarkham 9y ago
    Sure. You can have daily sprints if you like --- if you know what you're doing. The rule with frameworks is this: the more time you spend thinking and screwing around with the framework, the more you're either doing it wrong or it's hurting more than helping. Scrum/Agile should just work, just like your C compiler just works. It's a tool that operates almost invisibly in the background while you make stuff people want. If it takes a long time to work with it, or you spend a lot of time fussing with it or arguing over whether you're doing it right or not? You've got something backwards somewhere. People end up with long sprints and tons of meetings either usually because they don't understand what they're doing or they're doing it the wrong way. Personally I prefer week or less sprints. Anything else tends to grow a bunch of meeting BS over time that's non-productive.

jerf 9y ago

It seems to me one of the human irrationalities we don't hear much about is the widespread belief (proved by actions) that process and bureaucracy is free, and therefore, when analyzing a new process, it is not necessary to consider what the costs of the process are. Once you set the cost incorrectly to zero, it only takes some very vague, theoretical, handwaved benefits to tip apparent value of the process positive, and that is a very low bar. So there is an enormous bias in humans for instituting processes. We almost never consciously get rid of them, either, because even after we've been paying the price of processes for years, somehow we still don't account for them as a cost for that specific process. They're just the cost of doing business or something like that, vague ambient costs that don't attach to anything specific somehow, mysteriously. Something that nobody can get a cognitive handle on and start thinking about clearly. You'll note I haven't limited this in scope to software engineering at all. It's not hopeless, though, because people can be educated (for lack of a better word) into properly accounting for costs. For some people it can be a "why didn't I realize this years ago" sort of revelation. If you manage to start communicating in this way properly, it's also a bus that engineering and the business management can start communicating with each other productively. They don't want to hear about the details of your technical debt, and you probably don't want to hear about the details of how they are refinancing their debts or whatever; you need to trust them to drive the business and they need to be able to trust you to offer them solid cost/benefit analyses of technical decisions, and offer them the correct suite of options (some they may not have considered), and while that may not bring heaven on Earth, at least these two traditionally disparate worlds can be communicating based on cost/benefit terms.

deleted 9y ago

[deleted]