8 ms·
The inspection paradox is everywhere (2015)
- dang 2y agoRelated. Others? The Inspection Paradox is Everywhere: a surprising statistical illusion - https://news.ycombinator.com/item?id=20665234 https://news.ycombinator.com/item?id=20665234 - Aug 2019 (4 comments) Inspection Paradox (2015) - https://news.ycombinator.com/item?id=18342560 https://news.ycombinator.com/item?id=18342560 - Oct 2018 (13 comments)
- twic 2y agoOne i noticed when working with star catalogue data decades ago: most stars are in binary (or higher) systems. But most star systems are single!
- svat 2y agoA comment on the post gives this similar example: > My favorite example (true in many European countries): most families have a single child, but most kids have siblings.
- layer8 2y agoThe link to the “new version” 404s, but this is probably a copy of it: https://medium.com/towards-data-science/the-inspection-paradox-is-everywhere-2ef1c2e9d709 https://medium.com/towards-data-science/the-inspection-parad... Also, this relates to renewal theory: https://en.wikipedia.org/wiki/Renewal_theory#Inspection_paradox https://en.wikipedia.org/wiki/Renewal_theory#Inspection_para...
- lisper 2y ago"Inspection paradox" seems like a misnomer. It has nothing to do with inspection and it isn't a paradox. It's just selection bias [1] or more specifically, sampling bias [2]. In the case of class sizes, are you asking the average class size from the point of view of a student (N per class) or a teacher (1 per class)? Take the extreme case where you have one class with 99 students and one class with one student. From the student's point of view, the average class has 98 students. From the teachers' point of view, the average class has 50. --- [1] https://en.wikipedia.org/wiki/Selection_bias https://en.wikipedia.org/wiki/Selection_bias [2] https://en.wikipedia.org/wiki/Sampling_bias https://en.wikipedia.org/wiki/Sampling_bias
- roenxi 2y agoIt is called that because one of the most common places it turns up is managers/inspectors doing routine work. Typically if a manager checks in on their report they'll be stuck doing something that takes a lot longer than usual. If an inspector inspects ... I dunno, widgets, then the widget will typically be unusual because of selection biases. I'd expect that since this is the usual workplace introduction to the economic value of knowing about the bias it got its name there. A lot of confused people trying to work out why their data makes no sense. The other fun workplace paradox would be if HR ever tries to be data driven, does some metrics over the engineering department and works out that a degree is inversely correlated with any attempted at measure of skill. Fortunately most HR persons are not interested enough in stats to try that approach.
- penr0se 2y agoCould you elaborate more on the inverse correlation between degree and skill? Do you mean that usually people who did not go to university actually went straight to work and had the chance to get more skill as opposed to people with a degree that actually started later?
- roenxi 2y agoSimpsons paradox. People without qualifications have to be obviously competent to be hired to do a job. If someone is clueless they probably slipped in because they got certified somehow (like with a degree). Expect a negative correlation between certification and competence (in the workplace) because the workplace only reliably excludes people who are incompetent and unqualified. So the population sampled is made up of [qualified, competent], [unqualified, competent] and [qualified, incompetent]. And anyone who isn't ready for that will get very confused when they try to work out how much value a degree adds in their pool of programmers. Or any department, really.
- penr0se 2y agoThat makes sense, but I would expect this paradox to vanish (or at least get weaker) as you go higher in the hierarchy of technical positions (i.e. from junior to lead, to senior, to principal etc.). I would expect the workplace to somehow naturally get rid of the incompetent people, so that after a certain point you're only left with [qualified, competent] and [unqualified, competent]
- tshaddox 2y ago> The same effect applies to passenger planes. Airlines complain that they are losing money because so many flights are nearly empty. At the same time passengers complain that flying is miserable because planes are too full. They could both be right. When a flight is nearly empty, only a few passengers enjoy the extra space. But when a flight is full, many passengers feel the crunch. But my complaints about flights being too full is based only on my own experience. I don't think people are upset about airplanes crowdedness because they surveyed a bunch of people and concluded from the survey that airplanes are crowded.
- m463 2y agoEven while they're empty, you don't have enough legroom.
- skirmish 2y agoWhat? You just lie down across three seats and enjoy it.
- kibwen 2y ago> But my complaints about flights being too full is based only on my own experience. I don't think people are upset about airplanes crowdedness because they surveyed a bunch of people and concluded from the survey that airplanes are crowded. This is still falling prey to the error described in the OP. Imagine that 99% of all flights were totally empty, and the remaining 1% of flights were completely full. Despite the fact that 99% of all seats are vacant in this scenario, 100% of all flyers will have the experience of being on exclusively packed flights.
- dullcrisp 2y agoYeah but I think they have a point in that when I book a flight in that scenario—if we don’t require that all the completely empty flights remain completely empty—then I’m going to wind up as the only passenger on one of the (previously) empty flights, rather than ending up on the full one. Unless the empty flights don’t allow bookings.
- fsdkfdsf 2y ago[dead]
- deleted 2y ago[deleted]
- bagels 2y agoIt's been decades since I was on a plane that hasn't been within a few seats of full.
- zootboy 2y agoIt really depends on where (and when) you're going. I've had a decent number of partly full flights going to oddball small airports. But the big hub-to-hub flights tend to be nearly if not completely full.
- mikestew 2y agoWhen 737 Max’s were…having difficulties, Alaska Air offered to reschedule your flight with no penalty. We flew anyway. It was glorious. Many rows with one person. Completely empty rows by the bathrooms. My spouse and I had a three seat row to ourselves. It was flying like it was 1999 again. (And this was Seattle to Orlando, not some puddle jumper to Bumphuck, Nowhere.)
- frotaur 2y agoYes, that's precisely what you would experience according to the paradox described in the post
- bagels 2y agoI wasn't providing a counterexample.
- rqtwteye 2y agoI have this quite a bit when I fly home late at night.
- vivzkestrel 2y agoI am liking the kind of articles on this blog but unfortunately. I cannot find an index page that lists every single post every written with a link to it. Could you kindly add such a page or point me to it?
- svat 2y agoLook at "Blog archive" in the sidebar; you can click on each year/month to expand; it looks something like this: https://imgur.com/a/Q91LC4k https://imgur.com/a/Q91LC4k And you can find more recent posts at his newer blog: https://www.allendowney.com/blog/ https://www.allendowney.com/blog/ where again there are similar links for each month.
- vivzkestrel 2y agothat is a lot of work to find the titles of every article ever written on that website. I am familiar with the archive format that half a dozen blogs use but they really need to come up with something more intuitive from a user experience perspective. I was hoping this OP would put together an index page with every link on his blog ever written. all he gotta do is loop through all the posts on that page and link their titles
- wanderingstan 2y agoThe author left out my favorite example: that you are mathematically more likely to be the slow line at the grocery store (or the slow lane on the highway). It’s not just bad luck! > you actually are, on the average, in the slower lane, because slow lanes are (on average) the ones that have more vehicles in. So you are more likely to be in these lanes than in the faster moving ones which more vehicles are in. https://leightonvw.com/2019/04/04/the-slower-lane-paradox-in-a-nutshell/ https://leightonvw.com/2019/04/04/the-slower-lane-paradox-in...
- hassleblad23 2y agoWow. This is such a fascinating insight. Obvious in hindsight but never occurred to me.
- 325 2y ago> In particular, cars travelling at greater speeds are normally more spread out than slower cars, so that over a given stretch of road there are likely to be more cars in the slower lane How many cars pass a sign over an hour-long period? It should be equal to speed (miles/hour) X density (cars/mile) (say that speed and density are constant over the hour) So more cars in the fast lane will pass that sign than cars in the slow lane will over the hour if speed (fast) / speed (slow) > density (slow) / density (fast)
- xg15 2y agoIsn't this also the cause of the "network illusion"? (I.e., your friends likely have more friends than you have, yet you probably have about as many friends as most people have)