5 ms·

A few years ago I worked on a project where my company was contracted to build a small piece of a much larger system by IBM. IBM was managing many different con

by seibelj 10y ago

A few years ago I worked on a project where my company was contracted to build a small piece of a much larger system by IBM. IBM was managing many different contractors. We needed a way to "securely" show a picture, aka not using the pre-installed image showing app. So IBM spent tens of thousands of dollars to build a new Android app that displayed a picture passed to it via an intent. I told our product guy that I could build it in an hour for free, but he said it wasn't our problem. In our twice-weekly status calls, we would ask about when the "image showing app" would arrive, and we heard all about the minutiae of building that app, from getting a PM, to finding a partner in India, to QA, to issues with the app, back to the contractor, and finally to us, where it was buggy as hell and never worked right. Completely mind boggling, and very enlightening about how IBM (and I'm sure other big vendors) operate.

sliverstorm 10y ago
Presumably that kind of process rigor is necessary in massive, high-scope projects, and the fallout we see is chiefly from the inability to apply a more lightweight process for tiny projects?
- bazqux2 10y ago
  It's counter productive for big projects as well. The Too Big To Fail attitudes to process cover all manor of sins and corruptions to the point the projects is guaranteed to fail. See the Joint Strike Fighter for the worlds single most expensive example of this.
- devishard 10y ago
  I'd buy that logic if it produced low defect rates and high uptime, but it doesn't. When we used SilverPop earlier this year it was down about 5% of the time and wrong 5% of the time, and even the basic way the interfaces we had to integrate with worked was a mess. I don't think their process is heavy because it's rigorous, I think it's heavy because it's a reflection of their corporate structure (which is large).
- g8oz 10y ago
  I'd call it process theater rather than process rigor.
  - AJ007 10y ago
    This is hardly limited to IBM or software development. Almost any business that relies on human to human "sales" is at risk of devolving in to a deceptive mess of complexity things that have nothing to do with delivering the desired product or service. That is dangerous because eventually competitors will be lapping you at a tiny fraction of the price.
  - dragonwriter 10y ago
    I don't disagree with the description "process theater", but its a common part of the overhead of enterprise contracting (often as a result of the demands of the customer, and often central to the compensation structure), and even a bigger part of the overhead of enterprise contracting that involves subcontracting. "Securing revenue through enterprise development contracts" is a different discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)
- justicezyx 10y ago
  This seems more like "spaghetti process". Process meant to dissect problem to the right size for predictable execution. Right size does not translate to always the same dissection for vastly different sized problems. For a tool as simple as the parent post describes, what IBM does is not process rigor.
- blakeyrat 10y ago
  The problem is I've worked with companies with a lot of "process rigor" (as you phrase it), which in theory should result in bug-free quality software. But it doesn't. The process rigor encourages checking stuff off of lists, but discourages the little ad-hoc fixes that make a product good. Nobody's going to fix the minor bug in the date entry field if it takes 4 hours of "process" to write the 4 minutes of code.
  - freehunter 10y ago
    The goal of it is not to result in bug-free software, the goal is to result in completely traceable responsibility. It's to protect the company. I work for IBM and believe me, the process and paperwork gets super tedious, but I understand why. The number of times I've had a client threaten to sue for breach of contract is staggering. If we didn't have a ton of process and paperwork showing a trail of evidence and every communication that led us to the final result, we'd be out of business so damn fast. Unfortunately some people really like to take unfair advantage of their vendors. Let me give you an example. Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted. The real event was more complicated, but for the sake of anonymity, let's say they wanted a weekly report as a bar graph, and when we delivered, they realized a pie chart made more sense. Now, being the good consultant I am, I switched the bar graph to a pie chart. It wasn't in the statement of work, but it only took a couple of hours and I like to keep my clients happy. After they got the pie chart, they said they liked the format but they wanted it switched from a weekly to a monthly report. Sure, it only takes a couple of hours. And then they wanted the monthly report, but the data resolution broken out into weeks instead of days. Only a couple of hours. And then they wanted the name of the report changed. A couple more hours. Bear in mind, this is all 100% custom development, from scratch. Eventually we ran over budget and over time doing these minor tasks, because what the client asked for wasn't what they actually wanted. They demanded we keep going until they were happy. They hadn't signed off on this work being completed, so they claimed it wasn't done according to the contract. A few weeks later we got a letter from their legal department, and in return we sent them the statement of work with everything checked off, and all of the emails we had sent back and forth between us and the client, showing where they acknowledged the scope was changing and we had said we would try our best, but couldn't guarantee anything other than what was in the SOW. We also showed where they refused to sign an extension of the project timeline and refused to sign a SOW change to formally add the new scope to the project. And that was the end of that story. Every step of the way when something changed, even if we discussed it on the phone, we had a follow-up email to confirm. Everything we did got documented, every change was clearly spelled out as to what we changed and what was the original agreed-upon scope. Every scope change was approved by the PM and sometimes, one of my managers. And at the end of the project, we deliver all of this (sometimes hundreds of pages) in a nice, neat report detailing to the client what they hired us for and what we actually did. It takes forever and it doesn't guarantee the end result will be perfect. What it guarantees is that if we get sued, we have all the paperwork proving we did exactly what was agreed to in the contract.
pg_is_a_butt 10y ago
"partner in India".... done. if you worked in india for next to nothing, would you see your clients as people who you wanted to help, or people who wanted to hurt others (others who charge fair prices for quality goods, and pay local taxes) to exploit you? exploiting them back is fair moral play. sell them shit and they'll turn around to hire you to fix the shit. fix the shit with more shit and you've got yourself a career. you're all idiots.
merpnderp 10y ago
There's a name for software built with that kind of process and quality. Enterprise.
stonogo 10y ago
I'm not defending IBM here, because god knows I've had my experiences with their juggernaut, but in this case you are not considering the big picture. If IBM had let you donate this component, there would be: 1) financial implications (you were paid to develop a solution. we, the customer, have discovered that you were donated time & code. did you disclose your potential status as receiver-of-charity during your initial bid? do we need to hold a recompete because other bidders did not know to use free labor in their bid? how much code was donated? does this reduce the average wage of developers on the project below minimum acceptable standards? can you prove it?) 2) product liability implications (this free code you used is not working properly. people are dying or becoming impoverished as a result. we are going to the donor-developer for doing this, his employer for allowing it, and you for failing to provide sufficiently rigorous integration testing.) 3) support implications (your free code broke, and nobody can get ahold of the rando who wrote it, and the company he worked for at the time disavows responsibility. everyone is going to get sued.) 4) due diligence implications (did you use this code because it was free or because it was correct? can you prove it? was this a cost-cutting measure that may have deprived me, the customer, of a superior product? can you prove it? don't answer me; answer my legal team.) I know it sounds extreme, but I've seen each of these things occur, at differing scales, over the years, and the reason companies like IBM exist is to deliver defensible solutions. This is not startup-land; it is not sufficient (or even, often, necessary) to deliver a working product -- the product must be able to withstand a Congressional investigation. You don't have to prove it correct, but you do have to prove you adhered to legislative guidelines intended to control the process. This is also the reason behind the headline regarding IBM's "reputation." Nobody at IBM gives a shit what any of us think about their technological prowess or even their project management skills. IBM's reputation is an ability to operate at a massive scale without getting nailed to the wall by some government or another. If they get their ears pinned back in this census situation, the takeaway is not "IBM tech sucks" or "IBM PM sucks" -- the takeaway is "IBM is not a safe company for politicians to hire"... and for an IBM, that is fatal. Again, I agree the whole system by which IBM operates is disgusting and wasteful, but to compete in this space you must become this space, as it were.
- seibelj 10y ago
  I don't think you're wrong, it's just amusing that something so simple became such a gigantic mess when put through the IBM process. It's food for thought.
semi-extrinsic 10y ago
> I could build it in an hour for free The big problem is that it's not beneficial for IBM/Oracle/Accenture/$consultancy to quickly deliver a stable solution that works well. They make much more money by having seven layers of middle management committees vote together a crappy solution that will need five years of support/upgrades and then a replacement. In other industries, like construction, the same fundamental problem exists, but there human lives are at stake, standards bodies have been formed to prevent it, and non-software engineers aren't outsorced to people in India who have as little knowledge as they have pride in their workmanship. If a contractor built apartment buildings the way IBM and friends build software, they'd be sued out of existence. I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all? It happens relatively often with software consultancies, usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice.
- eumoria 10y ago
  > usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice Also something to consider in a circumstance like public transport stuff is that the multiple approval levels and meddling happens on the Government side as well. Like a perfect storm of bureaucratic bullshit that ends up making an enormous bloated mess that does everything but what was originally intended.
- jeffbush 10y ago
  "I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all?" Construction projects go over budget and are late all the time. Remember the big dig? https://en.wikipedia.org/wiki/Big_Dig https://en.wikipedia.org/wiki/Big_Dig The Sydney Opera house was 10 years late and 1475% over budget: https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_and_cost https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_... Anyone who's had renovations done to their house can also probably sympathize.
  - erichocean 10y ago

by seibelj 10y ago

sliverstorm 10y ago

Presumably that kind of process rigor is necessary in massive, high-scope projects, and the fallout we see is chiefly from the inability to apply a more lightweight process for tiny projects?

bazqux2 10y ago
It's counter productive for big projects as well. The Too Big To Fail attitudes to process cover all manor of sins and corruptions to the point the projects is guaranteed to fail. See the Joint Strike Fighter for the worlds single most expensive example of this.
devishard 10y ago
I'd buy that logic if it produced low defect rates and high uptime, but it doesn't. When we used SilverPop earlier this year it was down about 5% of the time and wrong 5% of the time, and even the basic way the interfaces we had to integrate with worked was a mess. I don't think their process is heavy because it's rigorous, I think it's heavy because it's a reflection of their corporate structure (which is large).
g8oz 10y ago
I'd call it process theater rather than process rigor.
- AJ007 10y ago
  This is hardly limited to IBM or software development. Almost any business that relies on human to human "sales" is at risk of devolving in to a deceptive mess of complexity things that have nothing to do with delivering the desired product or service. That is dangerous because eventually competitors will be lapping you at a tiny fraction of the price.
- dragonwriter 10y ago
  I don't disagree with the description "process theater", but its a common part of the overhead of enterprise contracting (often as a result of the demands of the customer, and often central to the compensation structure), and even a bigger part of the overhead of enterprise contracting that involves subcontracting. "Securing revenue through enterprise development contracts" is a different discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)
justicezyx 10y ago
This seems more like "spaghetti process". Process meant to dissect problem to the right size for predictable execution. Right size does not translate to always the same dissection for vastly different sized problems. For a tool as simple as the parent post describes, what IBM does is not process rigor.
blakeyrat 10y ago
The problem is I've worked with companies with a lot of "process rigor" (as you phrase it), which in theory should result in bug-free quality software. But it doesn't. The process rigor encourages checking stuff off of lists, but discourages the little ad-hoc fixes that make a product good. Nobody's going to fix the minor bug in the date entry field if it takes 4 hours of "process" to write the 4 minutes of code.
- freehunter 10y ago
  The goal of it is not to result in bug-free software, the goal is to result in completely traceable responsibility. It's to protect the company. I work for IBM and believe me, the process and paperwork gets super tedious, but I understand why. The number of times I've had a client threaten to sue for breach of contract is staggering. If we didn't have a ton of process and paperwork showing a trail of evidence and every communication that led us to the final result, we'd be out of business so damn fast. Unfortunately some people really like to take unfair advantage of their vendors. Let me give you an example. Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted. The real event was more complicated, but for the sake of anonymity, let's say they wanted a weekly report as a bar graph, and when we delivered, they realized a pie chart made more sense. Now, being the good consultant I am, I switched the bar graph to a pie chart. It wasn't in the statement of work, but it only took a couple of hours and I like to keep my clients happy. After they got the pie chart, they said they liked the format but they wanted it switched from a weekly to a monthly report. Sure, it only takes a couple of hours. And then they wanted the monthly report, but the data resolution broken out into weeks instead of days. Only a couple of hours. And then they wanted the name of the report changed. A couple more hours. Bear in mind, this is all 100% custom development, from scratch. Eventually we ran over budget and over time doing these minor tasks, because what the client asked for wasn't what they actually wanted. They demanded we keep going until they were happy. They hadn't signed off on this work being completed, so they claimed it wasn't done according to the contract. A few weeks later we got a letter from their legal department, and in return we sent them the statement of work with everything checked off, and all of the emails we had sent back and forth between us and the client, showing where they acknowledged the scope was changing and we had said we would try our best, but couldn't guarantee anything other than what was in the SOW. We also showed where they refused to sign an extension of the project timeline and refused to sign a SOW change to formally add the new scope to the project. And that was the end of that story. Every step of the way when something changed, even if we discussed it on the phone, we had a follow-up email to confirm. Everything we did got documented, every change was clearly spelled out as to what we changed and what was the original agreed-upon scope. Every scope change was approved by the PM and sometimes, one of my managers. And at the end of the project, we deliver all of this (sometimes hundreds of pages) in a nice, neat report detailing to the client what they hired us for and what we actually did. It takes forever and it doesn't guarantee the end result will be perfect. What it guarantees is that if we get sued, we have all the paperwork proving we did exactly what was agreed to in the contract.

pg_is_a_butt 10y ago

"partner in India".... done. if you worked in india for next to nothing, would you see your clients as people who you wanted to help, or people who wanted to hurt others (others who charge fair prices for quality goods, and pay local taxes) to exploit you? exploiting them back is fair moral play. sell them shit and they'll turn around to hire you to fix the shit. fix the shit with more shit and you've got yourself a career. you're all idiots.

merpnderp 10y ago

There's a name for software built with that kind of process and quality. Enterprise.

stonogo 10y ago

I'm not defending IBM here, because god knows I've had my experiences with their juggernaut, but in this case you are not considering the big picture. If IBM had let you donate this component, there would be: 1) financial implications (you were paid to develop a solution. we, the customer, have discovered that you were donated time & code. did you disclose your potential status as receiver-of-charity during your initial bid? do we need to hold a recompete because other bidders did not know to use free labor in their bid? how much code was donated? does this reduce the average wage of developers on the project below minimum acceptable standards? can you prove it?) 2) product liability implications (this free code you used is not working properly. people are dying or becoming impoverished as a result. we are going to the donor-developer for doing this, his employer for allowing it, and you for failing to provide sufficiently rigorous integration testing.) 3) support implications (your free code broke, and nobody can get ahold of the rando who wrote it, and the company he worked for at the time disavows responsibility. everyone is going to get sued.) 4) due diligence implications (did you use this code because it was free or because it was correct? can you prove it? was this a cost-cutting measure that may have deprived me, the customer, of a superior product? can you prove it? don't answer me; answer my legal team.) I know it sounds extreme, but I've seen each of these things occur, at differing scales, over the years, and the reason companies like IBM exist is to deliver defensible solutions. This is not startup-land; it is not sufficient (or even, often, necessary) to deliver a working product -- the product must be able to withstand a Congressional investigation. You don't have to prove it correct, but you do have to prove you adhered to legislative guidelines intended to control the process. This is also the reason behind the headline regarding IBM's "reputation." Nobody at IBM gives a shit what any of us think about their technological prowess or even their project management skills. IBM's reputation is an ability to operate at a massive scale without getting nailed to the wall by some government or another. If they get their ears pinned back in this census situation, the takeaway is not "IBM tech sucks" or "IBM PM sucks" -- the takeaway is "IBM is not a safe company for politicians to hire"... and for an IBM, that is fatal. Again, I agree the whole system by which IBM operates is disgusting and wasteful, but to compete in this space you must become this space, as it were.

seibelj 10y ago
I don't think you're wrong, it's just amusing that something so simple became such a gigantic mess when put through the IBM process. It's food for thought.

semi-extrinsic 10y ago

> I could build it in an hour for free The big problem is that it's not beneficial for IBM/Oracle/Accenture/$consultancy to quickly deliver a stable solution that works well. They make much more money by having seven layers of middle management committees vote together a crappy solution that will need five years of support/upgrades and then a replacement. In other industries, like construction, the same fundamental problem exists, but there human lives are at stake, standards bodies have been formed to prevent it, and non-software engineers aren't outsorced to people in India who have as little knowledge as they have pride in their workmanship. If a contractor built apartment buildings the way IBM and friends build software, they'd be sued out of existence. I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all? It happens relatively often with software consultancies, usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice.

eumoria 10y ago
> usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice Also something to consider in a circumstance like public transport stuff is that the multiple approval levels and meddling happens on the Government side as well. Like a perfect storm of bureaucratic bullshit that ends up making an enormous bloated mess that does everything but what was originally intended.
jeffbush 10y ago
"I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all?" Construction projects go over budget and are late all the time. Remember the big dig? https://en.wikipedia.org/wiki/Big_Dig https://en.wikipedia.org/wiki/Big_Dig The Sydney Opera house was 10 years late and 1475% over budget: https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_and_cost https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_... Anyone who's had renovations done to their house can also probably sympathize.
- erichocean 10y ago