5 ms·

Lindy’s Law is an absolute gem, that I'm keeping. If we don't understand the fundamental limits to any particular kind of trend, our default assumption should

by btilly 5mo ago

Lindy’s Law is an absolute gem, that I'm keeping. If we don't understand the fundamental limits to any particular kind of trend, our default assumption should be that it will continue for about as long as it has gone on already. We can, in fact, easily put a confidence interval on this. With 90% odds we're not in the first 5% of the trend, or the last 5% of the trend. Therefore it will probably go on between 1/19th longer, and 19 times longer. With a median of as long as it has gone on so far. This is deeply counterintuitive. When we expect something to last a finite time, every year it goes on, brings us a year closer to when it stops. But every year that it goes on properly brings the expectation that it will go on for a year longer still. We're looking at a trend. We believe that it will be finite. Our intuition for that is that every year spent, is a year closer to the end. But our expectation becomes that every year spent, means that it will last yet another year more! How can we apply that? A simple way is stocks. How long should we expect a rapidly growing company, to continue growing rapidly?

jerf 5mo ago
It's an interesting idea, and it may be something that could be mathematically justified, but I do think this is an abuse of Lindy's Law in the absence of such a justification. Per Wikipedia [1]: "The Lindy effect applies to non-perishable items, like books, those that do not have an "unavoidable expiration date"." And later in the article you can see the mathematical formulation which says the law holds for things with a Pareto distribution [2]. I'd want to see some sort of good analysis that "the life span of exponential growth curves" is drawn from some Pareto distribution. I don't think it's completely out of the question. But I'm also nowhere near confident enough that it is a true statement to casually apply Lindy's Law to it. [1]: https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect [2]: https://en.wikipedia.org/wiki/Pareto_distribution https://en.wikipedia.org/wiki/Pareto_distribution
- btilly 5mo ago
  The analysis in the article explains why it applies to any phenomena that we might be able to notice. The argument given is the same as the one that I first ran across, not by that name, in https://www.nature.com/articles/363315a0 https://www.nature.com/articles/363315a0. https://en.wikipedia.org/wiki/Doomsday_argument https://en.wikipedia.org/wiki/Doomsday_argument claims that it was a rediscovery of something that was hypothesized a decade article. I hadn't tried to give it a name, or thought to apply it outside of that context. As for the mathematical qualms, I'm a big believer in not letting formal mathematical technicalities get in the way of adopting an effective heuristic. And the heuristic reasoning here is compelling enough that I would like to adopt it.
  - tsimionescu 5mo ago
    The argument sounds nice, but it's just wrong. It only works if most processes you're going to encounter that you know nothing about happen to be Lindy processes. If most processes happening around you that you know nothing about are not of that type, then the argument fails.
    - BurningFrog 5mo ago
      I understood it as if you know absolutely nothing about a process, your best guess is that it's half done. I don't even think there are any "genuine" Lindy processes. What would those look like? Are they always half done?
      - seanhunter 5mo ago
        > I understood it as if you know absolutely nothing about a process, your best guess is that it's half done. That is the argument that is being made, but that only holds if the process is drawn from an underlying Pareto distribution with epsilon > 1[1]. As a counterexample, I’m jetlagged and disorientated. I go to sleep and wake up. It’s light outside but I don’t know the time. What’s the best guess of the time of day? By the “Lindy law” the best guess is that the process of daytime is halfway done so if I’m half-way through the day, my best guess is it’s noon. Clearly that’s not the best guess that could be made. The distribution of times I might wake up is heavily skewed towards the morning, so the best guess is going to be some time in the morning. Now you might argue that we don’t know absolutely nothing about the cycle of the day and night and that’s true. But we also don’t know absolutely nothing about any of the examples in TFA either. The point is, the times of day I might wake up are not drawn from a pareto distribution with the right parameters so the Lindy Law heuristic completely fails. In TFA the author gives no justification for why the remaining lifespan of the exponential growth of AI might be drawn from such a distribution either, so there’s no reason to think the heuristic will be accurate in that case either. [1] From https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect. epsilon = 1 + 1/p where p is the parameter of the conditional expectation E[T-t|T>t] = p t. So only things with p positive but finite exhibit this effect. If p is negative then the best guess is going to be that the lifetime of the thing will end immediately because we’re already past the expected lifetime, and if p is infinite then the thing will never end so all finite guesses about its length are equally bad. So whether half-way is a good heuristic depends entirely on the underlying process and you’d need to demonstrate that the majority of things have positive p for half-way to be the best guess. That’s far from clear.
- tomjakubowski 5mo ago
  People who correctly cite the Lindy effect won't look like people who correctly cite the Lindy effect.
- riknos314 5mo ago
  > those that do not have an "unavoidable expiration date"." Try avoiding the heat death of the universe /s
LPisGood 5mo ago
This is the exact same heuristic used in CPU scheduling. We expect fresh processes to terminate quickly and long running processes to last for a while longer.
skybrian 5mo ago
You can do that but you're laundering ignorance into precise-seeming mathematics. Better to just say "we're probably somewhere in the middle, not at the beginning or end" and leave it at that. Calling a peak is hard.
- btilly 5mo ago
  You speak about laundering ignorance into precise-seeming mathematics as if it was a bad thing. But that's the entire idea of Bayesian reasoning. Which has proven to be surprisingly effective in a wide range of domains. I'm all for quantifying my ignorance, and using it as an outside view to help guide my expectations. Read the book Superforecasting to understand how effective forecasters use an outside view to adjust their inside view, to allow them to forecast things more precisely.
cortesoft 5mo ago
I feel like Lindy's law doesn't work for things whose observation is partly controlled by the thing itself. For example, take something like a fad or trend; they don't have a hard end date like human lifespan, so it should follow Lindy's law. However, the likelihood, on average across the population, that you observe a trend is going to be higher at the end of a trend lifecycle than at the beginning. This is baked into the definition - more and more people hear about a trend over time, so the largest quantity of observers will be at the end of the lifecycle, when the popularity reaches its peak. In other words, if you are a random person, finding out about a trend likely means it is near the end rather than the middle.
- mike_hock 5mo ago
  Similarly, if you are a random person being alive, it likely means that the world population is near its peak and extinction is at hand, or at least the start of a permanent decline. We have at least global warming and impending WW3, so that line of reasoning seems to work.
- ccppurcell 5mo ago
  Well it only works when there is no information at all apart from the past frequency. It's the solution to the tank problem. You know that the enemy number their tanks as they're produced. You capture a tank and know its number, N. What's the best guess about how many tanks the enemy has produced so far? As a pure mathematical model with no other details, the best guess is 2N. Of course in reality you have some ideas about how long it takes to make a tank, how many resources the enemy has etc. Analogously you have information about the way trends develop.
tsimionescu 5mo ago
While this is very fun as a mathematical exercise, it's completely irrelevant as a real tool for getting a better understanding of unknown processes in the real world. The law only applies for certain types of processes, and is completely wrong for other types (e.g. a human who has lived 50 years may live 50 more, but one who has lived 100 years will certainly not live 100 more). So the question becomes: what type of process are you looking at? And that turns out to be exactly the question you started with: is there a fundamental limit to this growth curve, or not.
- jfjfnfnttbtg 5mo ago
  > The law only applies for certain types of processes Did you even read the post? It’s an estimate in the context where you have zero information on which to base an accurate estimate. The author’s point is that if you’re making a different estimate you need to actually say what information is informing that. Human lifespan is obviously not a case where we have zero information, so what is your point in bringing that up?
  - t43562 5mo ago
    The lifespan argument shows how important it is to have more than zero information.
    - btilly 5mo ago
      Yes, it is valuable to have more than zero information. But often we don't have the information that we wish. Even more often, the information that we have leads us to a story, that severely misleads us. Reminding ourselves of the zero information version of the story, can be an antidote to being mislead that way. Therefore it is valuable to know how to make the most out of zero information. And if we have information, to think about exactly why it leads to a different conclusion.
      - t43562 5mo ago
        The argument is that it's not very valuable because it can be incredibly wrong and the priority is to get more information. "And we have information to think" - then we don't have zero information right?
throwawayk7h 5mo ago
Closely related is Laplace's Rule of Succession[1], which basically says that (in lieu of other information), the odds of something happening next time go down the more times in a row that it doesn't happen (and vice versa). So for example, the longer a time bomb ticks, the less likely it is to go off any time soon. (Assuming the timer isn't visible.) :) [1] https://en.wikipedia.org/wiki/Rule_of_succession https://en.wikipedia.org/wiki/Rule_of_succession
coldtea 5mo ago
>If we don't understand the fundamental limits to any particular kind of trend, our default assumption should be that it will continue for about as long as it has gone on already. We can, in fact, easily put a confidence interval on this. With 90% odds we're not in the first 5% of the trend, or the last 5% of the trend. Therefore it will probably go on between 1/19th longer, and 19 times longer. With a median of as long as it has gone on so far. People would confidently cite Lindy's law all the way near the end of a trend. Nothing would stop a Roman saying that just before the Fall. We don't always need to "understand the fundamental limits" to a trend to see where it's going. Just to observe more than a random blind guess about them. I also wouldn't trust the "see how much we're improving" benchmarks of a trillion dollar pre-IPO industry to begin with.
robberat7 5mo ago
I am sure Lindy's law is not a general law. Because i have a completely opposite example- How many years can we go by without having a passenger plane crash? as the number of years with no crash increases the chance of something going wrong increases with it. If a crash happened last week, you'd have some sort of sense that okay, it's not going to happen the week after. I can't comment on the causal mechanism, but maybe the collective of people working in the flight industry become more aware after a crash and do their job well.

by btilly 5mo ago

jerf 5mo ago

It's an interesting idea, and it may be something that could be mathematically justified, but I do think this is an abuse of Lindy's Law in the absence of such a justification. Per Wikipedia [1]: "The Lindy effect applies to non-perishable items, like books, those that do not have an "unavoidable expiration date"." And later in the article you can see the mathematical formulation which says the law holds for things with a Pareto distribution [2]. I'd want to see some sort of good analysis that "the life span of exponential growth curves" is drawn from some Pareto distribution. I don't think it's completely out of the question. But I'm also nowhere near confident enough that it is a true statement to casually apply Lindy's Law to it. [1]: https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect [2]: https://en.wikipedia.org/wiki/Pareto_distribution https://en.wikipedia.org/wiki/Pareto_distribution

btilly 5mo ago
The analysis in the article explains why it applies to any phenomena that we might be able to notice. The argument given is the same as the one that I first ran across, not by that name, in https://www.nature.com/articles/363315a0 https://www.nature.com/articles/363315a0. https://en.wikipedia.org/wiki/Doomsday_argument https://en.wikipedia.org/wiki/Doomsday_argument claims that it was a rediscovery of something that was hypothesized a decade article. I hadn't tried to give it a name, or thought to apply it outside of that context. As for the mathematical qualms, I'm a big believer in not letting formal mathematical technicalities get in the way of adopting an effective heuristic. And the heuristic reasoning here is compelling enough that I would like to adopt it.
- tsimionescu 5mo ago
  The argument sounds nice, but it's just wrong. It only works if most processes you're going to encounter that you know nothing about happen to be Lindy processes. If most processes happening around you that you know nothing about are not of that type, then the argument fails.
  - BurningFrog 5mo ago
    I understood it as if you know absolutely nothing about a process, your best guess is that it's half done. I don't even think there are any "genuine" Lindy processes. What would those look like? Are they always half done?
    - seanhunter 5mo ago
      > I understood it as if you know absolutely nothing about a process, your best guess is that it's half done. That is the argument that is being made, but that only holds if the process is drawn from an underlying Pareto distribution with epsilon > 1[1]. As a counterexample, I’m jetlagged and disorientated. I go to sleep and wake up. It’s light outside but I don’t know the time. What’s the best guess of the time of day? By the “Lindy law” the best guess is that the process of daytime is halfway done so if I’m half-way through the day, my best guess is it’s noon. Clearly that’s not the best guess that could be made. The distribution of times I might wake up is heavily skewed towards the morning, so the best guess is going to be some time in the morning. Now you might argue that we don’t know absolutely nothing about the cycle of the day and night and that’s true. But we also don’t know absolutely nothing about any of the examples in TFA either. The point is, the times of day I might wake up are not drawn from a pareto distribution with the right parameters so the Lindy Law heuristic completely fails. In TFA the author gives no justification for why the remaining lifespan of the exponential growth of AI might be drawn from such a distribution either, so there’s no reason to think the heuristic will be accurate in that case either. [1] From https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect. epsilon = 1 + 1/p where p is the parameter of the conditional expectation E[T-t|T>t] = p t. So only things with p positive but finite exhibit this effect. If p is negative then the best guess is going to be that the lifetime of the thing will end immediately because we’re already past the expected lifetime, and if p is infinite then the thing will never end so all finite guesses about its length are equally bad. So whether half-way is a good heuristic depends entirely on the underlying process and you’d need to demonstrate that the majority of things have positive p for half-way to be the best guess. That’s far from clear.
tomjakubowski 5mo ago
People who correctly cite the Lindy effect won't look like people who correctly cite the Lindy effect.
riknos314 5mo ago
> those that do not have an "unavoidable expiration date"." Try avoiding the heat death of the universe /s

LPisGood 5mo ago

This is the exact same heuristic used in CPU scheduling. We expect fresh processes to terminate quickly and long running processes to last for a while longer.

skybrian 5mo ago

You can do that but you're laundering ignorance into precise-seeming mathematics. Better to just say "we're probably somewhere in the middle, not at the beginning or end" and leave it at that. Calling a peak is hard.

btilly 5mo ago
You speak about laundering ignorance into precise-seeming mathematics as if it was a bad thing. But that's the entire idea of Bayesian reasoning. Which has proven to be surprisingly effective in a wide range of domains. I'm all for quantifying my ignorance, and using it as an outside view to help guide my expectations. Read the book Superforecasting to understand how effective forecasters use an outside view to adjust their inside view, to allow them to forecast things more precisely.

cortesoft 5mo ago

I feel like Lindy's law doesn't work for things whose observation is partly controlled by the thing itself. For example, take something like a fad or trend; they don't have a hard end date like human lifespan, so it should follow Lindy's law. However, the likelihood, on average across the population, that you observe a trend is going to be higher at the end of a trend lifecycle than at the beginning. This is baked into the definition - more and more people hear about a trend over time, so the largest quantity of observers will be at the end of the lifecycle, when the popularity reaches its peak. In other words, if you are a random person, finding out about a trend likely means it is near the end rather than the middle.

mike_hock 5mo ago
Similarly, if you are a random person being alive, it likely means that the world population is near its peak and extinction is at hand, or at least the start of a permanent decline. We have at least global warming and impending WW3, so that line of reasoning seems to work.
ccppurcell 5mo ago
Well it only works when there is no information at all apart from the past frequency. It's the solution to the tank problem. You know that the enemy number their tanks as they're produced. You capture a tank and know its number, N. What's the best guess about how many tanks the enemy has produced so far? As a pure mathematical model with no other details, the best guess is 2N. Of course in reality you have some ideas about how long it takes to make a tank, how many resources the enemy has etc. Analogously you have information about the way trends develop.

tsimionescu 5mo ago

While this is very fun as a mathematical exercise, it's completely irrelevant as a real tool for getting a better understanding of unknown processes in the real world. The law only applies for certain types of processes, and is completely wrong for other types (e.g. a human who has lived 50 years may live 50 more, but one who has lived 100 years will certainly not live 100 more). So the question becomes: what type of process are you looking at? And that turns out to be exactly the question you started with: is there a fundamental limit to this growth curve, or not.

jfjfnfnttbtg 5mo ago
> The law only applies for certain types of processes Did you even read the post? It’s an estimate in the context where you have zero information on which to base an accurate estimate. The author’s point is that if you’re making a different estimate you need to actually say what information is informing that. Human lifespan is obviously not a case where we have zero information, so what is your point in bringing that up?
- t43562 5mo ago
  The lifespan argument shows how important it is to have more than zero information.
  - btilly 5mo ago
    Yes, it is valuable to have more than zero information. But often we don't have the information that we wish. Even more often, the information that we have leads us to a story, that severely misleads us. Reminding ourselves of the zero information version of the story, can be an antidote to being mislead that way. Therefore it is valuable to know how to make the most out of zero information. And if we have information, to think about exactly why it leads to a different conclusion.
    - t43562 5mo ago
      The argument is that it's not very valuable because it can be incredibly wrong and the priority is to get more information. "And we have information to think" - then we don't have zero information right?

throwawayk7h 5mo ago

Closely related is Laplace's Rule of Succession[1], which basically says that (in lieu of other information), the odds of something happening next time go down the more times in a row that it doesn't happen (and vice versa). So for example, the longer a time bomb ticks, the less likely it is to go off any time soon. (Assuming the timer isn't visible.) :) [1] https://en.wikipedia.org/wiki/Rule_of_succession https://en.wikipedia.org/wiki/Rule_of_succession

coldtea 5mo ago

>If we don't understand the fundamental limits to any particular kind of trend, our default assumption should be that it will continue for about as long as it has gone on already. We can, in fact, easily put a confidence interval on this. With 90% odds we're not in the first 5% of the trend, or the last 5% of the trend. Therefore it will probably go on between 1/19th longer, and 19 times longer. With a median of as long as it has gone on so far. People would confidently cite Lindy's law all the way near the end of a trend. Nothing would stop a Roman saying that just before the Fall. We don't always need to "understand the fundamental limits" to a trend to see where it's going. Just to observe more than a random blind guess about them. I also wouldn't trust the "see how much we're improving" benchmarks of a trillion dollar pre-IPO industry to begin with.

robberat7 5mo ago

I am sure Lindy's law is not a general law. Because i have a completely opposite example- How many years can we go by without having a passenger plane crash? as the number of years with no crash increases the chance of something going wrong increases with it. If a crash happened last week, you'd have some sort of sense that okay, it's not going to happen the week after. I can't comment on the causal mechanism, but maybe the collective of people working in the flight industry become more aware after a crash and do their job well.