6 ms·

If I am running a business that is looking at what projects to fund for the next year, and one of those projects is a software project with an unknowable time t

by cpitman 10y ago

If I am running a business that is looking at what projects to fund for the next year, and one of those projects is a software project with an unknowable time to completion vs (for example) opening a new store, with a well known time to complete, why would I ever pick the software project? It has a completely unknown ROI! If you cannot even give me probabilities, the lottery is a more guaranteed investment! Software estimation is required for real business to ever consider software projects. The key is to embrace uncertainty, and instead of estimating that task X will take exactly Y days, estimate a range of possibilities. I like the three point system, estimating a best case, worst case, and most likely case. It explicitly asks you to think about the long tail for when things go wrong. Then use statistical methods to combine estimates and create estimates for multiple confidence levels. This is all covered in a great book, "Software Estimation: Demystifying the Black Art" (https://read.amazon.com/kp/embed?asin=B00JDMPOVQ&asin=B00JDMPOVQ&preview=newtab&linkCode=kpe&ref_=cm_sw_r_kb_dp_kE5KxbAPDNJB5 https://read.amazon.com/kp/embed?asin=B00JDMPOVQ&asin=B00JDM...). It's not that long a read, and it goes over the problems covered by this blog post but doesn't throw it's hands up in the air and say estimation is impossible.

je42 10y ago
Estimation is not impossible. However, 100% confidence in your estimates is impossible.
- Leszek 10y ago
  100% confidence in any estimate is impossible, that's what makes it an estimate.
  - je42 10y ago
    Exactly, but if you communicate it to a "standard" manager/boss it usually becomes a deadline.
jsprogrammer 10y ago
>If I am running a business that is looking at what projects to fund for the next year, and one of those projects is a software project with an unknowable time to completion vs (for example) opening a new store, with a well known time to complete, why would I ever pick the software project? It has a completely unknown ROI! Because the benefits of the software project would far outweigh the benefits of a new store? With the completion of the software project you could open an order of magnitude more stores? Completion of the software project will increase the profitability of existing stores such that they make more than a new store at existing (hypothetical, since the store is not open yet) profitability?
- cpitman 10y ago
  And that would be great, as long as doing the software project doesn't take several orders of magnitude more time. You have to be able to estimate the project with some confidence interval to know if it is worth it. If opening a store takes 6-9 months, and we have a software project that can make all of my 100 stores ~10% more efficient that needs 2-4 years, then the software project looks a lot more feasible! (ignoring that calendar time is not exactly cost, and that we may need shorter term wins to stay in business) The problem is if, when asked for an estimate, we say "I don't know and I never will know until I'm done". Now I don't know if stores will be more efficient 2 weeks or 20 years from now.
  - jsprogrammer 10y ago
    Too bad, so sad. How can you estimate something that isn't done? You can throw some words out there, but they are likely meaningless. If you are operating solely on predictive ROI, then maybe investing in ambitious software projects is not for your business. >Now I don't know if stores will be more efficient 2 weeks or 20 years from now. That is your problem, not the software's or of the people creating the software.
markbnj 10y ago
>> If I am running a business that is looking at what projects to fund for the next year, and one of those projects is a software project with an unknowable time to completion vs (for example) opening a new store, with a well known time to complete, why would I ever pick the software project? Maybe because you're trying to innovate? Will opening a new store dramatically change your company's market outlook? Will it open up entirely new markets for products you don't yet have? Will it solve new problems for people? Milton Hershey offered his first chocolate bar in 1900. It took him a further five years to perfect the manufacturing processes that produced the Hershey's Kiss. None of those processes, and none of the machinery to execute them existed when he started. He had to invent them. Today, 110 or so years later, the company produces around 60 million of them a day which it ships all over the world. Probably he should have just opened another store. No offense, but people who think the way you do about risk are exactly the reason large companies lose their ability to innovate at all.
- lj3 10y ago
  > Maybe because you're trying to innovate? > No offense, but people who think the way you do about risk are exactly the reason large companies lose their ability to innovate at all. Exactly. This is why the startup ecosystem continues to thrive even after the age of the big IPO is over. Startups are akin to R&D for larger companies who don't want to (or can't) take the risk to try something new.
- cpitman 10y ago
  I'm not really making the argument your reacting to. Software projects can be estimated, so it is possible to understand what their ROI is. I'm saying that, because software can be estimated using rigorous technique instead of hunches, that we can continue to justify developing software. Even if what you are doing is truly research that is completely open ended, there still have to be controls placed on it. Budget/grants/runway will run out at some point.
  - maxaf 10y ago
    I'm not a believer in what you're saying, but I dearly want to believe, in no small part because there are significant organizational rewards to be reaped for believing in estimates and acting the part. What are these rigorous techniques you speak of? I seriously, honestly want to know.
atemerev 10y ago
There are other kind of businesses where estimation is hard. Producing movies, for example. Even if you cast established actors and limit yourself to low-risk sequels of successful franchises, chances of box office bomb are substantial. Book publishing would be even better example; thankfully, writing a book is dirt cheap comparing to developing software or making movies. Pharmaceutic research is absolutely unpredictable — but big pharma companies are usually quite profitable, like software companies (their research/production patterns are similar). New software development is _always_ innovation (because, if your problem is already solved by existing software, it is way cheaper to buy this software instead — as cost of production is close to zero compared to the cost of development). Innovation is _always_ unpredictable. Building the business on innovation is therefore a hard problem.
- TheOtherHobbes 10y ago
  It's fairly easy to estimate the time needed for a novel or a textbook. The only possible reasons for a holdup in (say) a tech book is if the product is in beta and big changes happen after writing but before publication. Producing a movie isn't rocket science either. But if anything it's more complex than most software projects, because you have to arrange the finance, hire hundreds of people spread across tens of teams, move people and stuff around, design and build things, knock things down, add a lot of stuff in post - which may need custom software - and so on. And that's just the footage. You also have to market and distribute the movie worldwide, work out deals for residuals on various media, and so on. And everyone has to be paid on time, near as, or work stops. Estimating profit is a completely different job, unrelated to keeping the machine running from start to finish. It's barely a footnote. Studios are mostly not bad at it, but they do sometimes get it very wrong. Designing a CRUD website that scales is nowhere close in terms of complexity. But here's an interesting thing: movies only get finished because there's a hierarchy of talent and delegation. The director/producer can hire an expert or a ready-made team to handle part of the project, and delegate that entire element to them. From then on the producer/director needs to do some oversight, but the details (should) "just work." Startups don't work like this. Startups hire everyone on a permanent contract basis, often as individuals, try to build them into teams internally, often on the basis that anyone sub-senior is pretty much interchangeable with everyone else at the same level, and generally delegation doesn't happen in the same way. There's really no reason why teams of programmers couldn't get together, set up service companies, and hire themselves out as consultant solution providers. After a few successes a team should have it made. This doesn't happen. I think that cultural difference is very interesting.
humanrebar 10y ago
> ...but doesn't throw it's hands up in the air and say estimation is impossible. It is impossible to know for sure. It's a bet. It's also impossible to say precisely what the DOW will be at next Thursday. But you can make a guess with confidence intervals, long or short the appropriate index, and make some money. The problem is that "real business" people want impossibly precise estimates for software projects in particular. If software estimates were more like medical prognoses or financial reports, they would include confidence intervals and risk assessments, and we would be having another conversation.
pyb 10y ago
That's precisely why so few companies who can produce good software, are also proficient in some other area. Successful software development mostly happens when it's the main focus of the organisation. Apple used to be the prominent exception, but their software's quality seems to have decreased in recent times.
FollowSteph3 10y ago
It's not as simple as you make it to be. How long do you think it would take to walk from San Francisco to Los Angeles? I guarantee you that your max will not be high enough. Try it and then read the following article and let's see how close you came http://www.michaelrwolfe.com/2013/10/19/50/ http://www.michaelrwolfe.com/2013/10/19/50/ Just because you can't always have an accurate estimate doesn't mean it's now worth it. Also remember that estimates are based on resources. We were able to go to the moon in a decade but a lot of work was done before and the budget to get there was much higher than what most software development efforts ever get relatively speaking. It might not be going to the moon but that doesn't mean some projects aren't very complex either.
- FollowSteph3 10y ago
  Actually another great example is a new pharmaceutical drug. Homw can you accurately predict that effort. If something is complex and or newer sometimes it's very hard if not impossible to predict.
- cpitman 10y ago
  That article is more like asking someone who has never programmed before how long it will take to do a project. If you are estimating something that you have truly never done and have no experience in, then it's going to be more difficult. But if you at least acknowledge that, you can take steps against it like lowering your own confidence level in your estimates (this feature will take 4-8 weeks, and I'm only 25% confident the actual timeline is between those). The issue is more when you are over confident, which it sounds like no one here is! I'm not an avid hiker, but I've done enough weekend hikes to know that 10 miles per day can be pushing it for me on some terrain. I wouldn't ask someone who has never hiked ever before to estimate how long a hike would take. I wouldn't ask a programmer to estimate how long it takes to construct a bridge.
  - FollowSteph3 10y ago
    Have you ever tried to sell something to management with a 25% confidence level with an estimate that is at least 2x (4-8 weeks). You've just basically said your estimate is anywhere from a week to half a year, which basically means you have no estimate ;) In terms of never having done it before, this is true of most software projects. If you're building the same house from the same plans then this is the same as say installing a Wordpress blog, easy to estimate. But if you were to ask how long it would take to build Wordpress from scratch, even having a Wordpress sample app, it would be very hard. Not only that but there are a lot of assumptions. Is it for a handful of visitors or are we talking a million visitor blog? Ignoring that, let's go back to the pharma example. How long do you estimate developing a new drug will take? It's not the first drug your company has developed. Anything beyond a week is hard to estimate. Back to your estimate of 25% within 4-8 weeks, this would never be accepted because it could mean anything. The worse part, I an tell you right now, even if new functionality or changes are added, you will be elf to 4-8 weeks, the 25% will be ignored. And in most cases management will hold you to 4 weeks because this is what was sold to their bosses.
freddie_mercury 10y ago
To generate an ROI requires two inputs. Businesses also need to create estimates of the return. How often do you see that happen?
- cpitman 10y ago
  It is honestly all over the board. I've worked with clients that had estimated the return, in many cases because it was very easy to do. Migrating from an expensive proprietary product (IBM MQ) to a much cheaper open source product (Red Hat AMQ) when you have 1000s of brokers and yearly subscription costs makes it very easy to calculate the savings. Another client was working with us to create a new product line from scratch. As part of that, they had to estimate the return on the product, the budget it would take, then justify that to their owners. I've also had clients where they are not great at this. They are also invariably the ones with poor scope control and overall immature project management.
slavik81 10y ago
How did your store opening estimate get created in the first place? Making a good estimate takes time and it has an unknown ROI. Especially for the first store.
geon 10y ago
> instead of estimating that task X will take exactly Y days, estimate a range of possibilities. That's pretty much the definition of "estimate".

by cpitman 10y ago

je42 10y ago

Estimation is not impossible. However, 100% confidence in your estimates is impossible.

Leszek 10y ago
100% confidence in any estimate is impossible, that's what makes it an estimate.
- je42 10y ago
  Exactly, but if you communicate it to a "standard" manager/boss it usually becomes a deadline.

jsprogrammer 10y ago

>If I am running a business that is looking at what projects to fund for the next year, and one of those projects is a software project with an unknowable time to completion vs (for example) opening a new store, with a well known time to complete, why would I ever pick the software project? It has a completely unknown ROI! Because the benefits of the software project would far outweigh the benefits of a new store? With the completion of the software project you could open an order of magnitude more stores? Completion of the software project will increase the profitability of existing stores such that they make more than a new store at existing (hypothetical, since the store is not open yet) profitability?

cpitman 10y ago
And that would be great, as long as doing the software project doesn't take several orders of magnitude more time. You have to be able to estimate the project with some confidence interval to know if it is worth it. If opening a store takes 6-9 months, and we have a software project that can make all of my 100 stores ~10% more efficient that needs 2-4 years, then the software project looks a lot more feasible! (ignoring that calendar time is not exactly cost, and that we may need shorter term wins to stay in business) The problem is if, when asked for an estimate, we say "I don't know and I never will know until I'm done". Now I don't know if stores will be more efficient 2 weeks or 20 years from now.
- jsprogrammer 10y ago
  Too bad, so sad. How can you estimate something that isn't done? You can throw some words out there, but they are likely meaningless. If you are operating solely on predictive ROI, then maybe investing in ambitious software projects is not for your business. >Now I don't know if stores will be more efficient 2 weeks or 20 years from now. That is your problem, not the software's or of the people creating the software.

markbnj 10y ago

>> If I am running a business that is looking at what projects to fund for the next year, and one of those projects is a software project with an unknowable time to completion vs (for example) opening a new store, with a well known time to complete, why would I ever pick the software project? Maybe because you're trying to innovate? Will opening a new store dramatically change your company's market outlook? Will it open up entirely new markets for products you don't yet have? Will it solve new problems for people? Milton Hershey offered his first chocolate bar in 1900. It took him a further five years to perfect the manufacturing processes that produced the Hershey's Kiss. None of those processes, and none of the machinery to execute them existed when he started. He had to invent them. Today, 110 or so years later, the company produces around 60 million of them a day which it ships all over the world. Probably he should have just opened another store. No offense, but people who think the way you do about risk are exactly the reason large companies lose their ability to innovate at all.

lj3 10y ago
> Maybe because you're trying to innovate? > No offense, but people who think the way you do about risk are exactly the reason large companies lose their ability to innovate at all. Exactly. This is why the startup ecosystem continues to thrive even after the age of the big IPO is over. Startups are akin to R&D for larger companies who don't want to (or can't) take the risk to try something new.
cpitman 10y ago
I'm not really making the argument your reacting to. Software projects can be estimated, so it is possible to understand what their ROI is. I'm saying that, because software can be estimated using rigorous technique instead of hunches, that we can continue to justify developing software. Even if what you are doing is truly research that is completely open ended, there still have to be controls placed on it. Budget/grants/runway will run out at some point.
- maxaf 10y ago
  I'm not a believer in what you're saying, but I dearly want to believe, in no small part because there are significant organizational rewards to be reaped for believing in estimates and acting the part. What are these rigorous techniques you speak of? I seriously, honestly want to know.

atemerev 10y ago

There are other kind of businesses where estimation is hard. Producing movies, for example. Even if you cast established actors and limit yourself to low-risk sequels of successful franchises, chances of box office bomb are substantial. Book publishing would be even better example; thankfully, writing a book is dirt cheap comparing to developing software or making movies. Pharmaceutic research is absolutely unpredictable — but big pharma companies are usually quite profitable, like software companies (their research/production patterns are similar). New software development is _always_ innovation (because, if your problem is already solved by existing software, it is way cheaper to buy this software instead — as cost of production is close to zero compared to the cost of development). Innovation is _always_ unpredictable. Building the business on innovation is therefore a hard problem.

TheOtherHobbes 10y ago
It's fairly easy to estimate the time needed for a novel or a textbook. The only possible reasons for a holdup in (say) a tech book is if the product is in beta and big changes happen after writing but before publication. Producing a movie isn't rocket science either. But if anything it's more complex than most software projects, because you have to arrange the finance, hire hundreds of people spread across tens of teams, move people and stuff around, design and build things, knock things down, add a lot of stuff in post - which may need custom software - and so on. And that's just the footage. You also have to market and distribute the movie worldwide, work out deals for residuals on various media, and so on. And everyone has to be paid on time, near as, or work stops. Estimating profit is a completely different job, unrelated to keeping the machine running from start to finish. It's barely a footnote. Studios are mostly not bad at it, but they do sometimes get it very wrong. Designing a CRUD website that scales is nowhere close in terms of complexity. But here's an interesting thing: movies only get finished because there's a hierarchy of talent and delegation. The director/producer can hire an expert or a ready-made team to handle part of the project, and delegate that entire element to them. From then on the producer/director needs to do some oversight, but the details (should) "just work." Startups don't work like this. Startups hire everyone on a permanent contract basis, often as individuals, try to build them into teams internally, often on the basis that anyone sub-senior is pretty much interchangeable with everyone else at the same level, and generally delegation doesn't happen in the same way. There's really no reason why teams of programmers couldn't get together, set up service companies, and hire themselves out as consultant solution providers. After a few successes a team should have it made. This doesn't happen. I think that cultural difference is very interesting.

humanrebar 10y ago

> ...but doesn't throw it's hands up in the air and say estimation is impossible. It is impossible to know for sure. It's a bet. It's also impossible to say precisely what the DOW will be at next Thursday. But you can make a guess with confidence intervals, long or short the appropriate index, and make some money. The problem is that "real business" people want impossibly precise estimates for software projects in particular. If software estimates were more like medical prognoses or financial reports, they would include confidence intervals and risk assessments, and we would be having another conversation.

pyb 10y ago

That's precisely why so few companies who can produce good software, are also proficient in some other area. Successful software development mostly happens when it's the main focus of the organisation. Apple used to be the prominent exception, but their software's quality seems to have decreased in recent times.

FollowSteph3 10y ago

It's not as simple as you make it to be. How long do you think it would take to walk from San Francisco to Los Angeles? I guarantee you that your max will not be high enough. Try it and then read the following article and let's see how close you came http://www.michaelrwolfe.com/2013/10/19/50/ http://www.michaelrwolfe.com/2013/10/19/50/ Just because you can't always have an accurate estimate doesn't mean it's now worth it. Also remember that estimates are based on resources. We were able to go to the moon in a decade but a lot of work was done before and the budget to get there was much higher than what most software development efforts ever get relatively speaking. It might not be going to the moon but that doesn't mean some projects aren't very complex either.

FollowSteph3 10y ago
Actually another great example is a new pharmaceutical drug. Homw can you accurately predict that effort. If something is complex and or newer sometimes it's very hard if not impossible to predict.
cpitman 10y ago
That article is more like asking someone who has never programmed before how long it will take to do a project. If you are estimating something that you have truly never done and have no experience in, then it's going to be more difficult. But if you at least acknowledge that, you can take steps against it like lowering your own confidence level in your estimates (this feature will take 4-8 weeks, and I'm only 25% confident the actual timeline is between those). The issue is more when you are over confident, which it sounds like no one here is! I'm not an avid hiker, but I've done enough weekend hikes to know that 10 miles per day can be pushing it for me on some terrain. I wouldn't ask someone who has never hiked ever before to estimate how long a hike would take. I wouldn't ask a programmer to estimate how long it takes to construct a bridge.
- FollowSteph3 10y ago
  Have you ever tried to sell something to management with a 25% confidence level with an estimate that is at least 2x (4-8 weeks). You've just basically said your estimate is anywhere from a week to half a year, which basically means you have no estimate ;) In terms of never having done it before, this is true of most software projects. If you're building the same house from the same plans then this is the same as say installing a Wordpress blog, easy to estimate. But if you were to ask how long it would take to build Wordpress from scratch, even having a Wordpress sample app, it would be very hard. Not only that but there are a lot of assumptions. Is it for a handful of visitors or are we talking a million visitor blog? Ignoring that, let's go back to the pharma example. How long do you estimate developing a new drug will take? It's not the first drug your company has developed. Anything beyond a week is hard to estimate. Back to your estimate of 25% within 4-8 weeks, this would never be accepted because it could mean anything. The worse part, I an tell you right now, even if new functionality or changes are added, you will be elf to 4-8 weeks, the 25% will be ignored. And in most cases management will hold you to 4 weeks because this is what was sold to their bosses.

freddie_mercury 10y ago

To generate an ROI requires two inputs. Businesses also need to create estimates of the return. How often do you see that happen?

cpitman 10y ago
It is honestly all over the board. I've worked with clients that had estimated the return, in many cases because it was very easy to do. Migrating from an expensive proprietary product (IBM MQ) to a much cheaper open source product (Red Hat AMQ) when you have 1000s of brokers and yearly subscription costs makes it very easy to calculate the savings. Another client was working with us to create a new product line from scratch. As part of that, they had to estimate the return on the product, the budget it would take, then justify that to their owners. I've also had clients where they are not great at this. They are also invariably the ones with poor scope control and overall immature project management.

slavik81 10y ago

How did your store opening estimate get created in the first place? Making a good estimate takes time and it has an unknown ROI. Especially for the first store.

geon 10y ago

> instead of estimating that task X will take exactly Y days, estimate a range of possibilities. That's pretty much the definition of "estimate".