4 ms·

"Survivorship bias" is a favourite go-to for contrarians, but it's not a strong effect and when you see something surprising way out at the tail end of a distri

"Survivorship bias" is a favourite go-to for contrarians, but it's not a strong effect and when you see something surprising way out at the tail end of a distribution, attributing it to survivorship bias is a lost opportunity to learn something important. When something survives 2x longer than expected, it might be survivorship bias. When it survives 10x, 100x, or 1000x longer than expected, you should probably look closer and see what material or operational differences might be responsible, because there's going to be something worth learning. Even if what you learn is just that any incandescent lightbulb can have a greatly extended life if you run it at below-spec brightness.

giantrobot 5y ago
When you have an outlier like this bulb you can't make any good statistical judgements about it unless it was part of a controlled experiment. This bulb is a sample size of one. We might roughly know some details about its original construction and early operational life but not enough to really draw conclusions from. The only way to get more details about it will be a postmortem examination of its physical qualities. Due to its age we have little knowledge of its specific construction details. Were the factory specs followed exactly (or even recorded) or was some of the fabrication machinery tuned by the operator? Did this bulb end up with thicker than normal glass or filament? What was the mortality rate of other bulbs in its batch? Does the fire station have better (or worse) than typical wiring? Did it have bad wiring early in the bulb's life to unintentionally under-load it? Sometimes extreme outliers are just extreme outliers. A postmortem examination of the bulb might tell you it had a manufacturing defect giving it thicker than typical filament that combined with babying it for the past 50 years extended its life. Sorry it's not just being contrarian. Outliers are outliers. Just because something is an outlier in a distribution doesn't automatically mean there's some hidden truth of the universe to be revealed.
kragen 5y ago
> When something survives 2x longer than expected, it might be survivorship bias. When it survives 10x, 100x, or 1000x longer than expected, you should probably look closer You're implicitly assuming a normal distribution or something like it, but it's easy for a lognormal distribution to span multiple orders of magnitude like that, and the Weibull distribution commonly used in reliability modeling for this kind of thing can also be heavy-tailed if k > 1. A simple physical example of how this could arise would be if the filament diameter of a batch of bulbs was normally distributed spanning a factor of 2, resulting in filament resistances spanning a factor of 4, resulting in filament powers spanning that same factor of 4, resulting in filament powers per unit area of filament surface spanning a factor of 2 (since the thickest filaments have the lowest resistance and thus the highest power at a fixed voltage; it'd be 8x instead on a constant-current source), resulting in filament Stefan-Boltzmann temperatures varying by about 19% (1.19 is the fourth root of 2), which works out to a temperature difference of about 200 K for filament temperatures of about 1000 K. That's probably (handwaving here) enough temperature difference for about an order of magnitude difference in vapor pressure, which (handwaving wildly at this point) might mean an order of magnitude difference in filament evaporation rate. Also, typically there are hotspots in a filament where it's thinner than the rest, which evaporate more rapidly because they're hotter, which makes them get thinner faster, making them even hotter. So the dominating factor in filament lifetime (at a given voltage) might not be how thick it is but how smooth it is. So, you might find out something interesting. Or you might just find that this bulb had an unusually thick or smooth filament. (Carbon has 1000x higher vapor pressure than tungsten, though, which is relevant here: https://www.powerstream.com/vapor-pressure.htm https://www.powerstream.com/vapor-pressure.htm.)
- jbay808 5y ago
  > You're implicitly assuming a normal distribution I'm not really making that assumption. Instead, I'm pointing out that learning this particular bulb survived due to "unusually think or smooth filament compared with other bulbs" is indeed something interesting you can learn by inspecting the survivor, and that might be something you can apply for process quality improvement. Basically, I'm advocating the point of view that if you are trying to improve lightbulb, or say, ball bearing quality (rather than just model it), then it's useful to look beyond the fact that the Weibull distribution is long-tailed. A wide distribution is a reflection of ignorance and thus a learning opportunity; the long tails are still caused by something. Yes, from a given batch of ball bearings, some might last 10x or 100x longer than others under the same loading conditions. But if you put those super-long-lasting bearings under a microscope, you might observe real, important differences compared with others from the same batch. The balls might all be rounder or more identically-sized compared with their cousins. (You can do the inspection before the lifetime test if you want to be extra fair). But it's even more important when you don't expect a long-tailed distribution to begin with; for example, if you have a linear damage model. In this case, if you control for filament diameter, resistance, or brightness, and still see that one bulb is a mysterious outlier, you might want to inspect the joints or run the gas in the bulb through an analyzer to see if there are any other surprises.
  - kragen 5y ago
    Agreed!

giantrobot 5y ago

When you have an outlier like this bulb you can't make any good statistical judgements about it unless it was part of a controlled experiment. This bulb is a sample size of one. We might roughly know some details about its original construction and early operational life but not enough to really draw conclusions from. The only way to get more details about it will be a postmortem examination of its physical qualities. Due to its age we have little knowledge of its specific construction details. Were the factory specs followed exactly (or even recorded) or was some of the fabrication machinery tuned by the operator? Did this bulb end up with thicker than normal glass or filament? What was the mortality rate of other bulbs in its batch? Does the fire station have better (or worse) than typical wiring? Did it have bad wiring early in the bulb's life to unintentionally under-load it? Sometimes extreme outliers are just extreme outliers. A postmortem examination of the bulb might tell you it had a manufacturing defect giving it thicker than typical filament that combined with babying it for the past 50 years extended its life. Sorry it's not just being contrarian. Outliers are outliers. Just because something is an outlier in a distribution doesn't automatically mean there's some hidden truth of the universe to be revealed.

kragen 5y ago

> When something survives 2x longer than expected, it might be survivorship bias. When it survives 10x, 100x, or 1000x longer than expected, you should probably look closer You're implicitly assuming a normal distribution or something like it, but it's easy for a lognormal distribution to span multiple orders of magnitude like that, and the Weibull distribution commonly used in reliability modeling for this kind of thing can also be heavy-tailed if k > 1. A simple physical example of how this could arise would be if the filament diameter of a batch of bulbs was normally distributed spanning a factor of 2, resulting in filament resistances spanning a factor of 4, resulting in filament powers spanning that same factor of 4, resulting in filament powers per unit area of filament surface spanning a factor of 2 (since the thickest filaments have the lowest resistance and thus the highest power at a fixed voltage; it'd be 8x instead on a constant-current source), resulting in filament Stefan-Boltzmann temperatures varying by about 19% (1.19 is the fourth root of 2), which works out to a temperature difference of about 200 K for filament temperatures of about 1000 K. That's probably (handwaving here) enough temperature difference for about an order of magnitude difference in vapor pressure, which (handwaving wildly at this point) might mean an order of magnitude difference in filament evaporation rate. Also, typically there are hotspots in a filament where it's thinner than the rest, which evaporate more rapidly because they're hotter, which makes them get thinner faster, making them even hotter. So the dominating factor in filament lifetime (at a given voltage) might not be how thick it is but how smooth it is. So, you might find out something interesting. Or you might just find that this bulb had an unusually thick or smooth filament. (Carbon has 1000x higher vapor pressure than tungsten, though, which is relevant here: https://www.powerstream.com/vapor-pressure.htm https://www.powerstream.com/vapor-pressure.htm.)

jbay808 5y ago
> You're implicitly assuming a normal distribution I'm not really making that assumption. Instead, I'm pointing out that learning this particular bulb survived due to "unusually think or smooth filament compared with other bulbs" is indeed something interesting you can learn by inspecting the survivor, and that might be something you can apply for process quality improvement. Basically, I'm advocating the point of view that if you are trying to improve lightbulb, or say, ball bearing quality (rather than just model it), then it's useful to look beyond the fact that the Weibull distribution is long-tailed. A wide distribution is a reflection of ignorance and thus a learning opportunity; the long tails are still caused by something. Yes, from a given batch of ball bearings, some might last 10x or 100x longer than others under the same loading conditions. But if you put those super-long-lasting bearings under a microscope, you might observe real, important differences compared with others from the same batch. The balls might all be rounder or more identically-sized compared with their cousins. (You can do the inspection before the lifetime test if you want to be extra fair). But it's even more important when you don't expect a long-tailed distribution to begin with; for example, if you have a linear damage model. In this case, if you control for filament diameter, resistance, or brightness, and still see that one bulb is a mysterious outlier, you might want to inspect the joints or run the gas in the bulb through an analyzer to see if there are any other surprises.
- kragen 5y ago
  Agreed!