3 ms·

Very important article. Software engineering also has a lot of fat-tailed distributions that can masquerade as normal distributions. Our intuition tends to assu

by jerf 3y ago

Very important article. Software engineering also has a lot of fat-tailed distributions that can masquerade as normal distributions. Our intuition tends to assume normal distributions to one degree or another and we must learn when to disregard it. One example that is both very important to software engineers and will hit close to home is effort estimations. It is a super fat-tailed distribution. The problem is that a lot of tools managers will use to try to make schedules implicitly assume that errors will essentially be Gaussian. I can draw this fancy Gantt chart because OK, sure, task 26 may be a couple of days late, but maybe task 24 will be a couple of days early. As we all know from real life, tasks are effectively never early, and it's completely expected that at least one task from any non-trivial project will be budgeted for 2 weeks in the early planning stage but actually take 4 months. Managers are driven batty by this because intuitively this feels like a 7 or 8 sigma event... but in real life, it isn't, it's so predictable that I just predicted it in your project, sight unseen, even though I know nothing about your project other than the fact it is "nontrivial". But there's a ton of places in software engineering, especially at any kind of scale, you must learn to discard Gaussian assumptions and deal with fatter tailed distributions. I get a lot of mileage out of the ideas in this essay. It's hard to give a really concrete guide to how to do that, but it's a very valuable skill if you can figure out how to learn it.

prometheus76 3y ago
I have been working in custom steel fabrication for most of my career, and our distributions are all fat-tail distributions. The main trick we've learned is to focus on and obsess over outliers at every stage of moving through our shop. If something has been sitting for more than three days, it's brought up in every meeting until it's moving again. The hardest part to communicate to accountant types and management types (especially if they come to our company from a manufacturing background) is that our system is and always will be a chaotic system, and therefore inherently unpredictable. The reason we are a chaotic system is because the base element in our system is a person, and people have inherently chaotic productivity output. Another major contributor to the chaos is the Pareto distribution among workers of productivity. When scheduling plans a task to take 10 hours, what they can't account for is that if employee A does it, it will take 4 hours, and if employee B does it, it will take 25 hours. I could go on and on with other layers of complexity that create long fat-tail distributions for us, but you get the point.
angarg12 3y ago
It might sound silly when spoken out loud, but I found the following example quite enlightening about this phenomenon: imagine that you want to drive to the airport, and you need to estimate how long it will take you to get there. You can simply measure how long it will take you to drive by dividing average speed by distance. If you actually drove there many times and measured your trip length, most of the time you would end up being later, sometimes horribly so. Why is that? The explanation is actually trivial: no matter how well things work, there is an absolute minimum time required to get to the airport. On the other hand, there are many things that might delay your trip: heavy traffic, closed roads, accidents... So, on the aggregate, there are very few things that can "go well", and they will reduce your trip time only a bit. On the other hand there are many things that can go wrong and make your trip last much, much longer. That's a sort of intuitive explanation of why things like software estimates are fat-tailed instead of normal.
- taeric 3y ago
  Ironically, that anecdote also paints a good reason to sample the data on how long it takes to get there. If you check your own history for how long it takes to get somewhere, it is probably far more predictable than you'd expect. Of course, this also explains why software estimates are hard. We are all too often estimating things we haven't done before.
- clove 3y ago
  >Why is that? The explanation is actually trivial: no matter how well things work, there is an absolute minimum time required to get to the airport. On the other hand, there are many things that might delay your trip: heavy traffic, closed roads, accidents... Perhaps an easier explanation is that within driving the course you will encounter several Poisson processes, and the waiting time for a Poisson process is an exponential distribution.
bsder 3y ago
> The problem is that a lot of tools managers will use to try to make schedules implicitly assume that errors will essentially be Gaussian. No, the fundamental problem is that someone who provides a genuinely accurate schedule always loses against someone who provides a politically palatable schedule. Until you fix this, scheduling will always be inaccurate.
- _nalply 3y ago
  This someone is perhaps not affected by a missed schedule as badly as others. It is rational to provide too optimistic a schedule. And if something goes amiss, find a political solution. One is to find a scapegoat.
- jerf 3y ago
  That is a true problem, but an orthogonal one to what I am talking about. Gantt charts subtly, but deeply, work in the assumption that task failures will be Gaussian. There are many problems with the whole "I can schedule the world months in advance" in the software world, even these two aren't all of them. There's a reason it isn't just sort of a failure, but such a comprehensive failure, across decades, across all scales of projects, or, rather, there's a lot of reasons.

by jerf 3y ago

prometheus76 3y ago

I have been working in custom steel fabrication for most of my career, and our distributions are all fat-tail distributions. The main trick we've learned is to focus on and obsess over outliers at every stage of moving through our shop. If something has been sitting for more than three days, it's brought up in every meeting until it's moving again. The hardest part to communicate to accountant types and management types (especially if they come to our company from a manufacturing background) is that our system is and always will be a chaotic system, and therefore inherently unpredictable. The reason we are a chaotic system is because the base element in our system is a person, and people have inherently chaotic productivity output. Another major contributor to the chaos is the Pareto distribution among workers of productivity. When scheduling plans a task to take 10 hours, what they can't account for is that if employee A does it, it will take 4 hours, and if employee B does it, it will take 25 hours. I could go on and on with other layers of complexity that create long fat-tail distributions for us, but you get the point.

angarg12 3y ago

It might sound silly when spoken out loud, but I found the following example quite enlightening about this phenomenon: imagine that you want to drive to the airport, and you need to estimate how long it will take you to get there. You can simply measure how long it will take you to drive by dividing average speed by distance. If you actually drove there many times and measured your trip length, most of the time you would end up being later, sometimes horribly so. Why is that? The explanation is actually trivial: no matter how well things work, there is an absolute minimum time required to get to the airport. On the other hand, there are many things that might delay your trip: heavy traffic, closed roads, accidents... So, on the aggregate, there are very few things that can "go well", and they will reduce your trip time only a bit. On the other hand there are many things that can go wrong and make your trip last much, much longer. That's a sort of intuitive explanation of why things like software estimates are fat-tailed instead of normal.

taeric 3y ago
Ironically, that anecdote also paints a good reason to sample the data on how long it takes to get there. If you check your own history for how long it takes to get somewhere, it is probably far more predictable than you'd expect. Of course, this also explains why software estimates are hard. We are all too often estimating things we haven't done before.
clove 3y ago
>Why is that? The explanation is actually trivial: no matter how well things work, there is an absolute minimum time required to get to the airport. On the other hand, there are many things that might delay your trip: heavy traffic, closed roads, accidents... Perhaps an easier explanation is that within driving the course you will encounter several Poisson processes, and the waiting time for a Poisson process is an exponential distribution.

bsder 3y ago

> The problem is that a lot of tools managers will use to try to make schedules implicitly assume that errors will essentially be Gaussian. No, the fundamental problem is that someone who provides a genuinely accurate schedule always loses against someone who provides a politically palatable schedule. Until you fix this, scheduling will always be inaccurate.

_nalply 3y ago
This someone is perhaps not affected by a missed schedule as badly as others. It is rational to provide too optimistic a schedule. And if something goes amiss, find a political solution. One is to find a scapegoat.
jerf 3y ago
That is a true problem, but an orthogonal one to what I am talking about. Gantt charts subtly, but deeply, work in the assumption that task failures will be Gaussian. There are many problems with the whole "I can schedule the world months in advance" in the software world, even these two aren't all of them. There's a reason it isn't just sort of a failure, but such a comprehensive failure, across decades, across all scales of projects, or, rather, there's a lot of reasons.