4 ms·

An analogy I like to use: "Here's a book of sudoku / crossword puzzles. How long would it take you to solve all of them?" Estimating this is similar to estimat

by biot 13y ago

An analogy I like to use: "Here's a book of sudoku / crossword puzzles. How long would it take you to solve all of them?" Estimating this is similar to estimating a software system. You've seen puzzles like them before, but you don't know the difficulty level of each or which ones have tricky clues that cause you to rack your brain trying to find the solution. It also nicely illustrates the 10x effect: some puzzle solvers breeze through them while others take forever. If you could break a software system down into a rigorous formal specification that could be precisely estimated, it would be possible to use that specification to build it automatically. Then that shifts the estimation up the ladder of abstraction to one where you estimate the time to create the rigorous formal specification.

jacques_chester 13y ago
Suppose you're asked to estimate a single sudoku problem. That's basically impossible, because n=1. In small samples from a large population, variation is enormous. But what if I ask you to estimate the time for 100 sudoku problems? For 1000? What if, after 2 years of solving sudoku problems full time I ask you for your estimate? What if I develop simple statistical models based on the number of squares that are already filled in? Already it has become possible to improve your estimate, to make it more accurate. The value of an estimate is balanced on the line between the cost of its uncertainty and the cost of its development. Over time, for any class of problem, the cost of developing estimates falls.
- jbrechtel 13y ago
  Definitely true that estimates can be improved from some hypothetical floor of quality.. Estimates still come with a cost in themselves though. They're not an end, merely a means. So we should consider alternatives. An alternative approach I've heard from some teams, though haven't had the pleasure of trying myself, is to break your work into similarly sized small chunks and measure team speed over time in terms of these chunks. This sounds a lot like relative point based estimation often seen on agile teams. However, the difference is that once you establish this sizing rhythm during analysis (with the help of estimation early on to hone in on sizing) then you skip estimation altogether. I suspect this needs to be calibrated from time to time ("our stories have gotten too big") and exceptions need to be made (e.g. "there's just no way to split this work up and keep it releseable") but those scenarios are exceptions, not the norm. A world without estimates sounds great to me.
  - jacques_chester 13y ago
    > However, the difference is that once you establish this sizing rhythm during analysis (with the help of estimation early on to hone in on sizing) then you skip estimation altogether. I'd argue that really what you're doing now is estimating the size of any given chunk. Chunks that are estimated to be too big are then decomposed. But you still make some judgement about how big each chunk is before you actually queue it up. > A world without estimates sounds great to me. Most people estimate, they just don't think of it as estimating. Similarly, most people can instinctively perform differential calculus; they just think of it as, for example, "catching a ball". I think that you're right that estimates are never ends in themselves and that their net value needs to be considered. In a sense, you perform a meta-estimate of an estimate's cost and return before doing one. Is it worth developing parametric models of when you will be home for dinner? Probably not, dinner will be cold and moldy by the time you get to it.
- brudgers 13y ago
  "What if I develop simple statistical models based on the number of squares that are already filled in?" Better to just have the experience to look at a Sudoku and have a good feeling for how difficult it is - it's not just the number of filled in squares but their locations and their values. After solving 1000's of Sudoku problems, experience will allow one to see the relevant patterns or rather have a feel for them when looking at a particular Sudoku. The process is sped up if one starts off estimating Sudoku solution times from day one and then compares those estimates to actual solution times. A longer way to developing meaningful estimates (and I prefer meaningful over accurate, since a good estimate can still be very wrong), is to avoid estimating one's time as an end in itself.
  - jacques_chester 13y ago
    > A longer way to developing meaningful estimates (and I prefer meaningful over accurate, since a good estimate can still be very wrong), is to avoid estimating one's time as an end in itself. Could you elaborate? What would one estimate on a book of sudoku problems?
    - brudgers 13y ago
      Who is the audience for the estimate? An estimate of how long it will take me to solve 1000 Sudoku for recreation will not have much meaning to a client who just wants the solution. What are the resources that will be brought to the task? An estimate for solving 1000 Sudoku with a staff of professional programmers will be different from one premised upon bribing a dining hall full of seniors with coffee, juice and donuts down at the retirement village. What is the unit of estimation? Calendar days until delivery is different from man-hours because I spend so much time on HN. The value of an estimate is the integrity and relevance of the process by which it was generated not its accuracy. Accuracy is a fortunate side effect - and then only sometimes. The key is that accuracy correlates to experience at making estimates, and rationalizing the failure to go through the process because past estimates have been inaccurate is the long road I mentioned.
- zeteo 13y ago
  > What if I develop simple statistical models based on the number of squares that are already filled in? What if you were unable to? NP-hard problems are weird like that. What if the underlying statistical distribution had random variance or some other strange features that make traditional statistical models useless? Maybe the issue is that we're trying to use intuitions developed with everyday problems that are essentially linear ("mow the lawn", "drive to the store") in places where they're not appropriate. Unlike usual everyday tasks, NP-hard problems (such as Sudoku or coding) have no known "royal road" - sometimes you just have to try all possible solutions.
  - jacques_chester 13y ago
    NP-hard problems often have "good enough" solutions that don't take heat-death time to solve. Consider the TSP. You can wait until the last proton decays and tie up all matter and energy in the universe. Or you can use an ant colony optimiser and accept that you will probably only come to within a few percent of the best possible solution. Oh noes! And so it is with estimation. We can build an ever-more-elaborate estimate that will consecutively shave fractions of a percentage point of precision and accuracy. Or we can accept that they are estimates, that they are meant to be imprecise and that accuracy is a range, not a number.
mathattack 13y ago
Good analogy. But I would add "this book and the next one that I create" and "you can have help from whoever happens to be with you" In the best case there is still a little "unknown unknown" about both the scope and team. The project manager can tighten things but you can never completely get rid of it.

by biot 13y ago

jacques_chester 13y ago

Suppose you're asked to estimate a single sudoku problem. That's basically impossible, because n=1. In small samples from a large population, variation is enormous. But what if I ask you to estimate the time for 100 sudoku problems? For 1000? What if, after 2 years of solving sudoku problems full time I ask you for your estimate? What if I develop simple statistical models based on the number of squares that are already filled in? Already it has become possible to improve your estimate, to make it more accurate. The value of an estimate is balanced on the line between the cost of its uncertainty and the cost of its development. Over time, for any class of problem, the cost of developing estimates falls.

jbrechtel 13y ago
Definitely true that estimates can be improved from some hypothetical floor of quality.. Estimates still come with a cost in themselves though. They're not an end, merely a means. So we should consider alternatives. An alternative approach I've heard from some teams, though haven't had the pleasure of trying myself, is to break your work into similarly sized small chunks and measure team speed over time in terms of these chunks. This sounds a lot like relative point based estimation often seen on agile teams. However, the difference is that once you establish this sizing rhythm during analysis (with the help of estimation early on to hone in on sizing) then you skip estimation altogether. I suspect this needs to be calibrated from time to time ("our stories have gotten too big") and exceptions need to be made (e.g. "there's just no way to split this work up and keep it releseable") but those scenarios are exceptions, not the norm. A world without estimates sounds great to me.
- jacques_chester 13y ago
  > However, the difference is that once you establish this sizing rhythm during analysis (with the help of estimation early on to hone in on sizing) then you skip estimation altogether. I'd argue that really what you're doing now is estimating the size of any given chunk. Chunks that are estimated to be too big are then decomposed. But you still make some judgement about how big each chunk is before you actually queue it up. > A world without estimates sounds great to me. Most people estimate, they just don't think of it as estimating. Similarly, most people can instinctively perform differential calculus; they just think of it as, for example, "catching a ball". I think that you're right that estimates are never ends in themselves and that their net value needs to be considered. In a sense, you perform a meta-estimate of an estimate's cost and return before doing one. Is it worth developing parametric models of when you will be home for dinner? Probably not, dinner will be cold and moldy by the time you get to it.
brudgers 13y ago
"What if I develop simple statistical models based on the number of squares that are already filled in?" Better to just have the experience to look at a Sudoku and have a good feeling for how difficult it is - it's not just the number of filled in squares but their locations and their values. After solving 1000's of Sudoku problems, experience will allow one to see the relevant patterns or rather have a feel for them when looking at a particular Sudoku. The process is sped up if one starts off estimating Sudoku solution times from day one and then compares those estimates to actual solution times. A longer way to developing meaningful estimates (and I prefer meaningful over accurate, since a good estimate can still be very wrong), is to avoid estimating one's time as an end in itself.
- jacques_chester 13y ago
  > A longer way to developing meaningful estimates (and I prefer meaningful over accurate, since a good estimate can still be very wrong), is to avoid estimating one's time as an end in itself. Could you elaborate? What would one estimate on a book of sudoku problems?
  - brudgers 13y ago
    Who is the audience for the estimate? An estimate of how long it will take me to solve 1000 Sudoku for recreation will not have much meaning to a client who just wants the solution. What are the resources that will be brought to the task? An estimate for solving 1000 Sudoku with a staff of professional programmers will be different from one premised upon bribing a dining hall full of seniors with coffee, juice and donuts down at the retirement village. What is the unit of estimation? Calendar days until delivery is different from man-hours because I spend so much time on HN. The value of an estimate is the integrity and relevance of the process by which it was generated not its accuracy. Accuracy is a fortunate side effect - and then only sometimes. The key is that accuracy correlates to experience at making estimates, and rationalizing the failure to go through the process because past estimates have been inaccurate is the long road I mentioned.
zeteo 13y ago
> What if I develop simple statistical models based on the number of squares that are already filled in? What if you were unable to? NP-hard problems are weird like that. What if the underlying statistical distribution had random variance or some other strange features that make traditional statistical models useless? Maybe the issue is that we're trying to use intuitions developed with everyday problems that are essentially linear ("mow the lawn", "drive to the store") in places where they're not appropriate. Unlike usual everyday tasks, NP-hard problems (such as Sudoku or coding) have no known "royal road" - sometimes you just have to try all possible solutions.
- jacques_chester 13y ago
  NP-hard problems often have "good enough" solutions that don't take heat-death time to solve. Consider the TSP. You can wait until the last proton decays and tie up all matter and energy in the universe. Or you can use an ant colony optimiser and accept that you will probably only come to within a few percent of the best possible solution. Oh noes! And so it is with estimation. We can build an ever-more-elaborate estimate that will consecutively shave fractions of a percentage point of precision and accuracy. Or we can accept that they are estimates, that they are meant to be imprecise and that accuracy is a range, not a number.

mathattack 13y ago

Good analogy. But I would add "this book and the next one that I create" and "you can have help from whoever happens to be with you" In the best case there is still a little "unknown unknown" about both the scope and team. The project manager can tighten things but you can never completely get rid of it.