7 ms·
The Lindy Effect
- scoofy 3y ago“Tried and true”
- coke12 3y agoSayings about how old things are surprisingly effective: "Don't knock it till you try it" "Chesterton's fence" "Reinventing the wheel" Any others?
- lovecg 3y ago“if it ain’t broke, don’t fix it” “better the devil you know than the devil you don’t” “old is gold” “stood the test of time”
- treerock 3y agoNot sure it fits, but I like The Gods of Copy Book Headings https://en.m.wikipedia.org/wiki/The_Gods_of_the_Copybook_Headings https://en.m.wikipedia.org/wiki/The_Gods_of_the_Copybook_Hea...
- kawhah 3y agoThe opposite of Chesterton's fence is "Chesterton's facehugger". Sometimes, when you discover something bad, the best strategy is to immediately try to remove it as quickly as possible, before spending any energy trying to find out why it appeared.
- atleastoptimal 3y agoThere's a subset of this I think about sometimes, not sure if it's actually a thing or been looked at/is valid. Let's say I'm waiting for a package to arrive and it's late. As each minute passes, does the chance of it coming the very next minute increase or decrease? One could say it increases because since it will eventually come and I have passed all the previous minutes they haven't come, then the remaining minutes until it comes comprise a smaller percentage of the all the remaining minutes, increasing the chance each minute. However up to a certain point the chance may decrease, as in, if it is "late" in a substantial sense, whatever caused it being something that delayed it a significant amount of time increases as time elapses, as the chances of it being something small that caused the lateness decreases as each minute passes.
- kqr 3y agoAnd you can model this through two survival functions, one thin-tailed and one heavy-tailed. As time increases, whatever your initial probability of "package will be very late" it will be dwarfed by the posterior after evidence of a long waiting time. It's the exact same technique used in https://two-wrongs.com/forecasting-covid-19-variants.html https://two-wrongs.com/forecasting-covid-19-variants.html except applied to "packages that may be late" instead.
- codetrotter 3y ago> it will eventually come It might also never come. Sometimes packages just disappear in transit. Maybe it lost its label. Maybe it got broken. Maybe someone stole it. Etc. My experience with packages goes like: - Most often things I order in my own country or from a neighbouring country arrive exactly on the expected day, or at least the next business day after that. This makes sense because shorter travel has less opportunity for things to go wrong. - Things from far away have their estimate set for like 60 days from now, because no one knows what boat it will end up etc. Sometimes these things show up quickly. Most of the time it takes a while, but still it arrives before the long estimate date. Sometimes they show up a bit late. - If they are very delayed then in both cases it means the package is just gone and will never arrive. So I inform the platform that the package was lost in transit and they refund the purchase.
- mlsu 3y agoThis is a sort of transformation of the bathtub curve.
- 3np 3y agoReplace it with waiting for a bus and you have a classic textbook example for teaching Poisson process! https://en.wikipedia.org/wiki/Poisson_point_process https://en.wikipedia.org/wiki/Poisson_point_process https://stats.stackexchange.com/questions/122722/please-explain-the-waiting-paradox https://stats.stackexchange.com/questions/122722/please-expl...
- dmurray 3y agoIt's specifically not a Poisson process, which assumes that the chance of a bus arriving at any given time is constant. A bus arrival has some kind of multimodal distribution, where the bus either comes on time or very late, with some noise in each case.
- DantesKite 3y ago> One implication is that even things which are becoming less robust over time can display the Lindy effect. Interesting. And somewhat counterintuitive.
- riffic 3y agohttps://www.google.com/search?q=lindy+site:https://news.ycombinator.com https://www.google.com/search?q=lindy+site:https://news.ycom...
- marttt 3y agoI like to use the Lindy effect as an everyday heuristic, quite often for choosing what books to read. That is: if a book has been highly relevant, say, for the past 20 years, it may have considerable relevance 20 years from today as well. Conclusion: I will think about buying that physical book for my home library, so my children would have a carefully-filtered collection of "classics" from many eras by the time they reach young adulthood. As a result, I almost never buy (or often even read) current-day bestsellers. Usually, one is not in a hurry with reading a book; also, new books can be really well marketed these days, which always makes me cautious. It seems to make more sense to read a book 5-10 years after it was first released. The Lindy effect does seem to hold true when I think about e.g. SICP, K&R's C book -- or Kernighan et al's 1988 AWK book for which the authors release a second edition in October this year! [1] Not to mention classic books in fiction or philosophy. Or cultural memes and ideas -- as a rough heuristic, the Lindy effect does often seem to hold true with these as well. It is a fun and extremely useful thinking tool for longer-term everyday prognostics. 1: https://www.awk.dev/ https://www.awk.dev/
- adamnemecek 3y ago[flagged]
- kqr 3y agoBy what metric? The Lindy effect is concerned with survival, which is affected to a large degree by, in this case, memetic fitness, which these books have in spades.
- adamnemecek 3y agoBy the quality of the book.
- kimi 3y agoI believe K&R is a wonderful introduction to C - "wonderful" as in "let's open the editor and try this". It's well written, simple, and you are left with the feeling that someone just gave you a Jedi weapon. It took a while for us as a community to realize that such a Jedi weapon can easily slice your fingers off, because before the internet the worst that would happen is "it crashed again" but that was kind of normal.
- golol 3y agoIt seems like the Lindy effect for a positive real number valued ramdom variable X, is the following identity E[X | X >= a] = 2 * a
- nonrandomstring 3y agoIt's more likely described by the non-linear portion corresponding to the first stage of the Bathtub Curve [0] we use in engineering. [0] https://en.wikipedia.org/wiki/Bathtub_curve https://en.wikipedia.org/wiki/Bathtub_curve Most complex but faulted things fail very early.
- sieste 3y ago> E[X | X >= a] = 2 * a Is it possible to derive the maximum entropy probability distribution subject to that constraint?
- golol 3y agoLet f: R -> R be the probability density function of X. It fulfills int f(x) dx = 1 and we furthermore assume that f is differentiable. We can write the identity as int_a^oo x f(x) dx / int_a^oo f(x) dx = 2 a. We rewrite this as int_a^oo x f(x) dx = 2 a int_a^oo f(x) dx. Taking the derivative with respect to a on both sides yields - a f(a) = 2 int_a^oo f(x) dx + 2 a (- f(a)). Taking another derivative yields - f(a) - a f'(a) = - 2 f(a) - 2 f(a) - 2 a f'(a). We rewrite this as f'(a) = - 3 f(a) / a, and furthermore f'(a) / f(a) = - 3 / a. We can write this as (ln ○ f)'(a) = - 3 / a. As far as I can tell, we can not find a solution on all of [0, oo). Let's assume that failure is impossible up to an early time t. This means that we assume f(x) = 0 for all x < t, and that possibly f(t) > 0, meaning we have a jump discontinuity in the probability density at x = t. We integrate from t to x int_t^x (ln ○ f)'(a) da = int_t^x - 3 / a da. This evaluates to ln(f(x)) = ln(f(t)) - int_t^x 3 / a da = ln(f(t)) - 3 ln(x) + 3 ln(t) = ln(f(t)) - ln(x^3) + ln(t^3) = ln(f(t) t^3 / x^3). In particular f(x) = f(t) t^3 / x^3. Right now f(t) is still a free parameter, but recall that 1 = int f(x) dx = int_t^oo f(t) t^3 / x^3 dx = f(t) t^3 [- 1 / (2 oo^2) + 1 / (2 t^2)] = f(t) t / 2 has to hold, so we find that f(t) = 2 / t and f(x) = 2 t^2 / x^3. So these are the probability densities that fulfill the "modified" Lindy effect, where failure is impossible before time t > 0. I haven't double checked this so maybe there is a mistake, but I'm quite convinced that this line of reasoning leads to a unique distribution for each t > 0 and impossibility when t = 0.
- stakhanov 3y agoHere is one way, I use the Lindy effect as a mental model: When picking a tech stack for a software investment and you want a reasonable expectation that it will still be actively maintained 10 years from now, you need to go back in time by 10 years. I’ve found this 10 years time horizon a useful one to work with, based on version histories of various bits of software I build upon. For example, for Python, this takes me back to version 3.4, before type decorators and many other things hit the scene that I disapprove of. So now, I test all my code with both version 3.4 and the newest version (3.11). I won’t use language features from 3.11 that weren’t already there in 3.4, and I won’t use code in 3.4 that breaks or throws deprecation warnings in 3.11. I also apply this test to dependencies, paying close attention to what happens if I try to get the newest version of some Python library running on version 3.4 of the interpreter or a 10-year-old version of the library on version 3.11 of the interpreter. This means my code is engineered so that it could have been in continuous operation for the last 10 years while running continuous updates on what’s underneath it. And this gives me a reasonable expectation that my code will require only minimal code changes over the next 10 years to keep up with whatever might arise. With the python interpreter itself, it’s remarkably easy to do that. I feel it doesn’t limit my coding in any meaningful way. With other bits of software, including many python libraries that one might depend on, it would be absolutely unworkable. In such a case, I take that as a clear signal not to use the dependency at all. This is a lot of work, but also a good forcing function that prevents me from becoming a dependency hog myself.
- emblaegh 3y agoSounds like there would be barely any up to date dependency that you could use, which in some cases could have known security issues, no?
- stakhanov 3y agoTo clarify: I still run the latest version of all the packages I depend on. For example, the newest version of SSL that you can still use with Python 3.4 has known issues. But I don't use that in production. I use Python 3.11 with an up-to-date SSL, but the code that I've written myself is code that I've also tested on Python 3.4 with outdated SSL. So the subset of features I use is a set of features considered by Python and SSL maintainers as so essential that they've maintained compatibility around them for ten years, and are therefore likely to maintain compatibility around for another ten years. -- Whether that's true no one can know, but applying the Lindy effect mental model would imply that this would be a useful way to think about it. The packages I get to use after applying the above filter are a minority, but, surprisingly, it's not "barely any" either. If you decide to live like this, you'll have less eyecandy in your life and less capability to capitalize on all the latest hotness all the time. But you'll get the job done, pretty much regardless of what the job might be. Your dependency hog colleagues will call you a dinosaur for your choice of tech stack. But when security vulnerabilities happen then, more often than not, there won't be any action needed on your part because, more often than not, you'll discover that you don't depend on the thing that has the vulnerability in the first place. You'll be home for dinner, while those same colleagues try to patch their dependencies, watch their software break, and try to work their way out of dependency hell (presumably by introducing more dependencies).
- vishnugupta 3y agoI really wish that we, software developers, apply Lindy rule while choosing library/framework/language/SaaS tool etc., Lot of developers underestimate the value of collective brain power that has gone into making a system mature, not to talk about community and ecosystem support. I was extremely hesitant to get onboard Golang bandwagon 5-7 years ago. I reluctantly said OK to try it out in a non critical service and all my concerns turned out to be true. I also apply this rule while buying car and choosing a bank. It's been interesting to see all the fintechs who badmouthed incumbent banks now going through exactly same challenges that banks went through and addressed.
- lexoj 3y agoCare to elaborate on the Go issues you encountered? Nowadays Go is very well established, unless you specifically meant that the issues happened 5-7 years ago.
- vishnugupta 3y agoOf the top of my head: 1) Lack of libraries. 2) Little to non-existent tooling & observability support. 3) Lots of challenges around gRPC. That it needed a discovery service such as etcd was a big source of pain. I remember two or three massive downtimes where an adjacent team went through hell for a day just to get back a high throughput system back online. gRPC + etcd was very susceptible to thundering herd problem. 4) Lack of experienced developers who would guide others idiomatic way of doing things. And related, go routines need a different mindset and somewhat a grounds up re-learning. The code would routinely get into deadlock sort of scenarios. I conceded that most of the above are not inherent to Go but to any new system/language. I'm sure they all have been addressed now as the language and ecosystem has matured. That was the point I was trying to make though; don't jump onto a new bandwagon. Of course you want to learn it on your own and experiment that's totally fine but to use it in production is fraught with risk IMO.
- hyperthesis 3y agojava will continue for 28 years; c much longer; and when both are long gone, lisp and fortran will still be here.
- bmacho 3y ago> So something whose lifespan is exponentially distributed (such as a radioactive particle) does not display the Lindy effect. Huh? Is this true? If I had particles with unknown half-times, I think I would expect the older particles to stay around longer, if that makes sense. (It is possible that there is no 'I expect' in the probability theory sense, like in the two envelopes problem.)
- mikewarot 3y agoClearly, my choice to program in Free Pascal/Lazarus, using the old GUI toolkits is vindicated! ;-) In the world of machining, 1940s-1960s American cast iron machine tools (Mills, Lathes, etc) are the best available. Grind / Scrape the bearing surfaces true, update the control systems, and they'll be good to go for at least 2 more generations.