25 ms·

The Waiting Time Paradox, Or, Why Is My Bus Always Late?

345 points by dragly 8y ago

kuu 8y ago
A bit off-topic: How can you integrate a jupyter notebook in a blog post like this one? It looks really nice! Nice article, btw, interesting topic!
- anotheryou 8y ago
  if I'm not mistaken there is a html export function that bakes it in to a static html
  - kuu 8y ago
    That's true! :) Thanks!
amai 8y ago
Is there a "evil" distribution which maximises the waiting time? Or is the Poisson distribution already the theoretical "evil" maximum that a public transport provider can achieve?
- anotheryou 8y ago
  Send all buses at once
  - kirkules 8y ago
    If something like that is an option, just don't ever send any.
    - anotheryou 8y ago
      Well but anything else would equally be bound to the extreme of the rule. "has to come within a 10 minute timeframe" = send one at the beginning, the next one at the end, so always 2 come together.
jobigoud 8y ago
I've encountered the inspection paradox in debates about factory farming and people talking past each other points. If you take the average farm, chances are that it's doing humane farming. But if you take the average animal, it has an overwhelming chance of being in an industrial farm.
- hmmmmmh 8y ago
  Just like if you pick an average human being she probably is poor and black/indian. But average GDP per capita is pretty high worldwide.
varlock 8y ago
Can't believe no one has yet mentioned the PASTA theorem - Poisson Arrivals See Time Averages (https://en.wikipedia.org/wiki/Arrival_theorem#Theorem_for_arrivals_governed_by_a_Poisson_process https://en.wikipedia.org/wiki/Arrival_theorem#Theorem_for_ar...). It is one of the theorems I remember the most from my Queuing Theory classes at the university!
MaxBarraclough 8y ago
This Wikipedia article seems relevant: https://en.wikipedia.org/wiki/Residual_time https://en.wikipedia.org/wiki/Residual_time (From reddit - https://www.reddit.com/r/programming/comments/9s4j58/the_waiting_time_paradox_or_why_is_my_bus_always/e8ntcng/ https://www.reddit.com/r/programming/comments/9s4j58/the_wai... )
nyc111 8y ago
Would not be easier to actually time the actual waiting times as he waited for the bus every day?
- jchw 8y ago
  That would probably make it seem, though, that the buses actually don't arrive (on average) every 10 minutes, since you'd oversample the buses that take longer than 10 minutes.
  - dmurray 8y ago
    That's exactly the phenomenon the article explores.
  - mmt 8y ago
    I'd argue that it's not oversampling at all, but, rather, that the measure of "average bus arrival time" is what's invalid or misleading. After all, the point of the bus arrivals isn't in service of the bus (or driver) but of the passengers. Observed average wait time at each bus stop is a better measure. The even better measure would be average wait time weighted by number of passengers [1]. [1] which is tougher to measure empirically, or even model, than just average wait time for that one person, since it requires counting passengers boarding, not just bus arrival times.
    - hammock 8y ago
      That's a nice idea but ignores all the people sitting in their offices or homes, choosing to go or not go out of their places down to the bus stop. Better to consider each bus stop as an asset to invest in, the more valuable it is, the more people you can serve.
      - hammock 8y ago
        @mmt to clarify, you seem to be treating bus stops independent of alternative means of transportation. Measuring the average wait time of people at the bus stop is not enough: there are people who chose to ride a bike today instead of waiting at the bus stop, because of what happened to them yesterday at the bus stop.
stephengillie 8y ago
OneBusAway is surprisingly accurate, at least in my experience. Google Maps has very good transit support too. One reason buses are late is because a bus must travel a circuit. Cars provide linear transportation, so the delay can only happen in the direction of your travel. Since buses run a circuit, they are impacted by delays in the direction opposite of your travel as well. Your bus might be late because the return route has traffic or other delays. Or maybe a drunk or drug user got in a fight with the driver and the police were needed. Or someone in a wheelchair had a problem getting onto the lift.
- QML 8y ago
  Why is modeling required? Can’t we just put a sensor on every bus, and just return the empirical expected time it takes for the next bus to drive to your station given the time of day and day of week?
  - stephengillie 8y ago
    Every bus does have sensors - it's where the data in the post came from. OneBusAway shows the live location of your bus too. The issue here is the deviation between empirical expected time and actual arrival time. Unhandled exceptions abound.
  - wiml 8y ago
    That's how it works (at least the OneBusAway feed in the Seattle area). They started in the 90s with an RFID transponder on bus stops, read by the bus as it went past; more recently they use things like odometry or GPS to feed information into the system. But: > just return the empirical expected time it takes for the next bus There is a world of complexity in "the empirical expected time", there... expected according to what models? Anecdotally, I think it's especially hard to model because any given delay is probably attributable to one or a few specific incidents. This isn't a situation where everything averages out and we can use a nice tractable AWGN model; we're down in the muck and the shot-noise.
- yhoneycomb 8y ago
  > Google Maps has very good transit support too. Where do you live? I'm guessing the bay area? I was carless in DC for a year and Google Maps was ALWAYS wrong about when busses when arrive. My friend recommended an app called Transit which was right about 90% of the time, which was a godsend for me.
gwern 8y ago
The memorylessness of the Poisson process makes the statistical aspect a bit trivial. But here's an interesting variant: how should you update your beliefs while waiting if there is a certain probability that the bus won't come at all? "The Ups and the Downs of the Hope Function in a Fruitless Search", Falk et al 1994: https://www.gwern.net/docs/statistics/bayes/1994-falk https://www.gwern.net/docs/statistics/bayes/1994-falk
taeric 8y ago
Highly recommend reading this to any folks that are just sitting the discussions. The simulations were worth the article on their own. The real world analysis was a great bonus. Anecdotally, i was expecting confirmation bias to be the main culprit. Pleasantly surprised to seei was wrong.
herodotus 8y ago
Nice article. It reminds me of my year living in London, and taking the bus everyday to Imperial College from West End Lane in West Hampstead. There was a stop on both sides of the road - one for the outbound bus, and one for the inbound (the bus went from central London to a terminus and then returned mostly on the same route). Now we did not use schedules - way too inaccurate at rush hour, and the busses there were pretty frequent anyway. But we did expect an even chance of the inbound bus arriving before an outbound one did. My daughter and I became convinced after a while that this was not happening, so we invented a game (which we called "The Game of Life".) When our bus (inbound) arrived first, we added 1 to our score. We subtracted 1 for every outbound bus that passed before ours arrived (there were often more than 1). We realized that the result would be slightly skewed to the negative, but we expected the outcome to be close to 0 over time. Of course it was not. Anyway we extended the game to many statistical situations. For example, you go to the checkout line at the supermarket, and there are N people in front of you. When you get to the front of the line, you count the people behind you - call that M. If M is bigger than N, you scored life points. If it is smaller, you lost some. So you add M-N to your running score, and you get an idea of how lucky you are in life. However, I never followed up with any real analysis, so I enjoyed this article.
- asdkhadsj 8y ago
  Wow, interesting idea! Imagine if (in the future) some item like a phone can detect this information around you, and automatically record it. Forming games ontop of this life data would be weird, neat, fun and sad all at the same time. Imagine seeing a real example of where someone else is just more lucky than you are in stupid but impactful (on your morale) ways. If it didn't seem so tedious to track, I'd love to implement an app to record this info. Unfortunately no one I know would care, and I'm sure I'd get too lazy to keep it accurate. Neat nonetheless, thanks for the cool thoughts :)
  - adrianN 8y ago
    Similar to the recruiters that throw away the top half of the application stack because they don't want unlucky people in their company I could see such data become valuable to some people.
edoo 8y ago
Hah great analysis. One factor with bus' is the schedule is likely planned to minimize early arrivals at the risk of being late more often. Usually when a bus is early it has to sit and wait until its departure time. A late running bus can be more efficient, and if kept until departure time might not ever get a chance to average down the bursts of lateness.
- stormbrew 8y ago
  I don't think this is always true. My city measures buses as being on time at inner stops as being between something like 5 minutes early and 1 minute late (yes, you read that right). Timing stops, where they have to wait, are pretty infrequent (mostly bus terminals). I don't know how common it is but it does exist. And buses perpetually being early means that if you're on time you wait even longer for the next one.
- 8note 8y ago
  a very late bus grabs everyone from each stop, so the next bus ends up early because it skips a bunch of less popular stops
twtw 8y ago
> a Poisson process is a memoryless process that assumes the probability of an arrival is entirely independent of the time since the previous arrival. In reality, a well-run bus system will have schedules deliberately structured to avoid this kind of behavior: buses don't begin their routes at random times throughout the day, but rather begin their routes on a schedule chosen to best serve the transit-riding public. I've never really understood any example involving a poisson process. They always seem to involve bus arrivals or light bulbs burning out, and I can't understand why the memory less property would ever make any sense for these. Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting? What is an actual phenomenon that is well modeled by a poisson process?
- jobigoud 8y ago
  > Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting? I don't think it's saying anything about how long you've been waiting, and you don't know when was the last arrival. It's saying that if you pick a random point on the timeline, the expected wait time doesn't change. That's because by taking a random point you have more chances of landing in a larger stretch of wait time than in a smaller one.
  - twtw 8y ago
    > I don't think it's saying anything about how long you've been waiting, and you don't know when was the last arrival. This is exactly what memorylessness says something about. Your second paragraph isn't unique to Poisson processes, but the author right at the start says that the expected value of the waiting time is the same as the average interarrival time, which indicates Poisson.
- bzbarsky 8y ago
  > What is an actual phenomenon that is well modeled by a poisson process? Radioactive decay. Collisions of fluid molecules. Unstimulated (i.e. not in a laser) photon emission due to electron transitions in an atom. Lots of pretty memoryless stuff going on at the microscopic level.
nyc111 8y ago
> When waiting for a bus that comes on average every 10 minutes, your average waiting time will be 10 minutes. This is very ambiguous. Unless he gives a time frame the numbers do not make sense. Average in a week? Average in a year? This is not how it works in real life. And I cannot accept his premise. My experience tells me that, in New York, when I used to take a bus to work, sometimes the bus was coming as I was walking to the stop; sometimes I would wait a long time. Sometimes not very long. There was no observable bias.
- pieguy 8y ago
  In statistics, "average" often means "expected value". No time frame is specified (although you could consider it an infinite time frame). With a small sample size your actual average might not be 10 minutes, but as your sample size grows, it will tend toward 10 minutes.
  - twtw 8y ago
    > it will tend toward 10 minutes If you are talking about spherical-cow style poisson buses, yeah (that's what the author means by "reasonable assumptions). But as the author concludes, bus arrival times are not well modeled by a poisson process.
mayankkaizen 8y ago
Nice article. Since I just started learning Stats, I wish I could find more of such notebooks. Any recommendations?
ikken 8y ago
This reminds me the bet in the bitcoin community [1]. If on average bitcoin blocks are produced every 10 minutes, and you learn that 5 minutes ago someone found a block, what is the average time you will wait for the next block? It turns out it's 10 minutes, not 5 minutes as you would intuitively think. (it's a memoryless process, so average expected time till block is always the same - 10 minutes - no matter how many blocks were recently found). In other words, when you're waiting for bitcoin transaction to be confirmed and go to check how long ago the most recent block was produced, in order to estimate how soon the next one will come - you're doing it wrong. Even if previous block was found 9 minutes ago, you're average waiting time for the next block is still 10 minutes. [1]. https://www.reddit.com/r/btc/comments/7rs8ko/dr_craig_s_wright_has_refused_to_pay_up_on_a_bet/ https://www.reddit.com/r/btc/comments/7rs8ko/dr_craig_s_wrig...
- FabHK 8y ago
  And a related counterintuitive fact (again, assuming 10 minutes): 1. If you pick a block randomly (uniformly), its average length is 10 minutes. 2. If you pick a point t0 in time randomly (uniformly), the average length of the block you're in is 20 mins (and the average length from t0 to next block is 10 mins, and the average length from previous block to t0 is also 10 mins (and, needless to say, 10+10=20...)).
- kccqzy 8y ago
  Here's an even simpler example: a flip of a fair coin. Suppose this is your first flip, one would intuitively think that there is 50% chance of H and 50% T. Suppose you flipped once and got H. For your second flip, one might intuitively think that since the number of times getting H over the long run is 50% of the total flips, and we already have a flip of H, to "balance it out" the next flip should have a smaller probability of getting H. Instead that is wrong. The next flip still has 50% chance of H. Suppose further that one has performed N flips, all of them H. One might even think that because of the way the geometric distribution works, it is very unlikely for the next flip to be H again. Instead that is wrong. The next flip still has 50% chance of H.
- ikeboy 8y ago
  This is actually wrong. The average expected time till next block is almost never 10 minutes because hashpower goes on and offline all the time. It gets adjusted every 2016 blocks based on historical block timing so that if no changes occur then future blocks would be 10 minutes on average - but changes always happen so this is never accurate. As such, you do learn something by looking at prior block times.
usgroup 8y ago
This is a straightforward consequence of modelling an arrival process as a Poisson distribution with a constant rate of arrival lambda... Go from arrival to cumulative arrivals to time of arrival to recurrence of arrival (next arrival). All are Poisson processes, including the recurrence process, which has a fixed expected value.
nakedrobot2 8y ago
In prague, the trams all run on time - within 2 minutes or less of the posted time. So I think this article is incorrect for this particular context.
- albertgoeswoof 8y ago
  Trams don’t have traffic in the way that buses do, and there are only minor differences in the drivers that could cause anomalies (I.e. you can’t steer a tram the wrong way) so they’re much easier to keep on time.
agumonkey 8y ago
Slightly related, my ghost town had few buses and sparses. I could never rely on printed hours. If I got there 10 min earlier to be sure, I'd still never be sure I'd wait 20 min for nothing because it was 11 min early. Of course half the time if I decide to walk to the next town where buses are many, I'd see all my town buses (both ways) pass me <yell-at-cloud.png> I think it made me completely careless about time, I would just go between stops and take the first one, go with the flow. By experience I'd know the range it would take for me to reach big places around the area. I had a friend who was completely foreign to this mode of thinking, she was very dilligent and fully trusting (although she mostly used trains so a lot less divergence). It reminds me of kid studies about intelligence / wealth ratios. When you're environment is random, you think random. When it's predictable you planify.
ChrisFoster 8y ago
It strikes me that even with a perfectly regular starting schedule, buses might clump together in time because the schedule is probably dynamically unstable. To explain, picking up passengers from a stop costs time and a long time between buses implies a high probability that passengers will be waiting at a given stop. This further adding to the delay and shortens the time to the next bus in the schedule. I'm sure drivers try to actively manage this, but if they didn't I suspect the system would naturally evolve toward pairs of buses leapfrogging each other on long routes.
- jobigoud 8y ago
  I think another confusing factor about that specific example is that bus shouldn't ever start before their schedule. Otherwise you run the risk of a bunch of people missing their bus even though they showed up on time. I think bus, trains and planes can only be late. For example this is an article about a Japanese Train company issuing a public apology for departing 20 seconds early. https://www.bbc.com/news/world-asia-42009839 https://www.bbc.com/news/world-asia-42009839
ezoe 8y ago
I hate the poisson distribution because it completely against the naive instincts of how random behaves.
stornetn 8y ago
Reminds me of a similar article that measured a similar kind of question about the wait times for NYC subways conditional on how long you've been waiting (https://erikbern.com/2016/04/04/nyc-subway-math.html https://erikbern.com/2016/04/04/nyc-subway-math.html). I think it's a pretty safe bet that people who like this post will like this article as well.
akane 8y ago
On a related note, arrival time predictions can be biased early to prevent people from missing buses, which also increases the perception of lateness. https://nextbus.cubic.com/FAQs https://nextbus.cubic.com/FAQs
graycat 8y ago
It's all much easier than that: It's just the Poisson process, e.g., with a nice chapter in E. Cinlar, Introduction to Stochastic Processes. Buses come as arrivals. So bus arrivals are a stochastic arrival process where stochastic just means varying randomly over time where, really, the randomly doesn't mean anything, includes deterministic arrivals, that is, known exactly in advance, but also admits any case of unpredictability. Well, in short, if have a stochastic arrival process with stationary, independent increments, then the arrival process is a Poisson process and there is a number, usually denoted by lambda, so that the times between arrivals are independent, identically distributed random variables with exponential distribution with arrival parameter, the arrival rate, lambda. The stationary means that the probability distribution of the times between arrival does not change over time. The independent increments means that the time from one arrival to the next is independent of all the past history of arrivals. The exponential distribution has the property, easy to verify with simple calculus, that the conditional expectation of the arrival time given that the arrival time is already greater than some number is the same as the expected arrival time. So, net, if bus arrivals form a Poisson process, then the time until the next bus arrives is the same after waiting five minutes as not having waited at all. Cinlar's treatment is nice because it is qualitative, that is, has assumptions that can often be confirmed or believed just intuitively. And we might not believe that bus arrivals meed the assumptions. This subject can continue with, say, hazard curves for equipment failures and a lot more about Poisson processes. E.g., the sum of two independent Poisson processes, say, Red buses and Blue buses, assuming that they are Poisson processes, is also a Poisson process with arrival rate the sum of the Red and Blue arrival rates. If randomly throw away some arrivals, then what is left is also a Poisson process with arrival rate adjusted in the obvious way. In Feller's volume II is the renewal theorem that the sum of independent arrival processes, Poisson or not, with mild assumptions, converges to a Poisson process as the number of processes summed grows. So, if the users of a sufficiently busy Web site act independently with mild assumptions, then the Web site will see arrivals accurately as a Poisson process. The vanilla Poisson process is Geiger counter clicks. There is much more to the pure and applied math and applications of Poisson processes.
PascLeRasc 8y ago
Is the inspection paradox what would happen if you surveyed everyone on how many siblings they had, and every sibling double-counted N-1 times (where N is the number of siblings in their family), inflating the resulting "average number of siblings", or is that something different?
- combatentropy 8y ago
  This is my first exposure to it, but yes, I think so. My paraphrase is: The Inspection Paradox is when you ask someone "in the mix" about the mix. You're only going to get an accurate estimation of the mix by standing outside of the mix. So yes, if you want an accurate count of siblings, you would consult some spreadsheet that just lists how many children each family had. If you go and start asking the families themselves (those "in the mix") then your results will be skewed. I thought the article that this article linked to was also very good, "The Inspection Paradox Is Everywhere," by Allen Downey, http://allendowney.blogspot.com/2015/08/the-inspection-paradox-is-everywhere.html http://allendowney.blogspot.com/2015/08/the-inspection-parad...
torgian 8y ago
I’m glad I live in east Asia. Busses and trains are almost never late

345 points by dragly 8y ago

kuu 8y ago

A bit off-topic: How can you integrate a jupyter notebook in a blog post like this one? It looks really nice! Nice article, btw, interesting topic!

anotheryou 8y ago
if I'm not mistaken there is a html export function that bakes it in to a static html
- kuu 8y ago
  That's true! :) Thanks!

amai 8y ago

Is there a "evil" distribution which maximises the waiting time? Or is the Poisson distribution already the theoretical "evil" maximum that a public transport provider can achieve?

anotheryou 8y ago
Send all buses at once
- kirkules 8y ago
  If something like that is an option, just don't ever send any.
  - anotheryou 8y ago
    Well but anything else would equally be bound to the extreme of the rule. "has to come within a 10 minute timeframe" = send one at the beginning, the next one at the end, so always 2 come together.

jobigoud 8y ago

I've encountered the inspection paradox in debates about factory farming and people talking past each other points. If you take the average farm, chances are that it's doing humane farming. But if you take the average animal, it has an overwhelming chance of being in an industrial farm.

hmmmmmh 8y ago
Just like if you pick an average human being she probably is poor and black/indian. But average GDP per capita is pretty high worldwide.

varlock 8y ago

Can't believe no one has yet mentioned the PASTA theorem - Poisson Arrivals See Time Averages (https://en.wikipedia.org/wiki/Arrival_theorem#Theorem_for_arrivals_governed_by_a_Poisson_process https://en.wikipedia.org/wiki/Arrival_theorem#Theorem_for_ar...). It is one of the theorems I remember the most from my Queuing Theory classes at the university!

MaxBarraclough 8y ago

This Wikipedia article seems relevant: https://en.wikipedia.org/wiki/Residual_time https://en.wikipedia.org/wiki/Residual_time (From reddit - https://www.reddit.com/r/programming/comments/9s4j58/the_waiting_time_paradox_or_why_is_my_bus_always/e8ntcng/ https://www.reddit.com/r/programming/comments/9s4j58/the_wai... )

nyc111 8y ago

Would not be easier to actually time the actual waiting times as he waited for the bus every day?

jchw 8y ago
That would probably make it seem, though, that the buses actually don't arrive (on average) every 10 minutes, since you'd oversample the buses that take longer than 10 minutes.
- dmurray 8y ago
  That's exactly the phenomenon the article explores.
- mmt 8y ago
  I'd argue that it's not oversampling at all, but, rather, that the measure of "average bus arrival time" is what's invalid or misleading. After all, the point of the bus arrivals isn't in service of the bus (or driver) but of the passengers. Observed average wait time at each bus stop is a better measure. The even better measure would be average wait time weighted by number of passengers [1]. [1] which is tougher to measure empirically, or even model, than just average wait time for that one person, since it requires counting passengers boarding, not just bus arrival times.
  - hammock 8y ago
    That's a nice idea but ignores all the people sitting in their offices or homes, choosing to go or not go out of their places down to the bus stop. Better to consider each bus stop as an asset to invest in, the more valuable it is, the more people you can serve.
    - hammock 8y ago
      @mmt to clarify, you seem to be treating bus stops independent of alternative means of transportation. Measuring the average wait time of people at the bus stop is not enough: there are people who chose to ride a bike today instead of waiting at the bus stop, because of what happened to them yesterday at the bus stop.

stephengillie 8y ago

OneBusAway is surprisingly accurate, at least in my experience. Google Maps has very good transit support too. One reason buses are late is because a bus must travel a circuit. Cars provide linear transportation, so the delay can only happen in the direction of your travel. Since buses run a circuit, they are impacted by delays in the direction opposite of your travel as well. Your bus might be late because the return route has traffic or other delays. Or maybe a drunk or drug user got in a fight with the driver and the police were needed. Or someone in a wheelchair had a problem getting onto the lift.

QML 8y ago
Why is modeling required? Can’t we just put a sensor on every bus, and just return the empirical expected time it takes for the next bus to drive to your station given the time of day and day of week?
- stephengillie 8y ago
  Every bus does have sensors - it's where the data in the post came from. OneBusAway shows the live location of your bus too. The issue here is the deviation between empirical expected time and actual arrival time. Unhandled exceptions abound.
- wiml 8y ago
  That's how it works (at least the OneBusAway feed in the Seattle area). They started in the 90s with an RFID transponder on bus stops, read by the bus as it went past; more recently they use things like odometry or GPS to feed information into the system. But: > just return the empirical expected time it takes for the next bus There is a world of complexity in "the empirical expected time", there... expected according to what models? Anecdotally, I think it's especially hard to model because any given delay is probably attributable to one or a few specific incidents. This isn't a situation where everything averages out and we can use a nice tractable AWGN model; we're down in the muck and the shot-noise.
yhoneycomb 8y ago
> Google Maps has very good transit support too. Where do you live? I'm guessing the bay area? I was carless in DC for a year and Google Maps was ALWAYS wrong about when busses when arrive. My friend recommended an app called Transit which was right about 90% of the time, which was a godsend for me.

gwern 8y ago

The memorylessness of the Poisson process makes the statistical aspect a bit trivial. But here's an interesting variant: how should you update your beliefs while waiting if there is a certain probability that the bus won't come at all? "The Ups and the Downs of the Hope Function in a Fruitless Search", Falk et al 1994: https://www.gwern.net/docs/statistics/bayes/1994-falk https://www.gwern.net/docs/statistics/bayes/1994-falk

taeric 8y ago

Highly recommend reading this to any folks that are just sitting the discussions. The simulations were worth the article on their own. The real world analysis was a great bonus. Anecdotally, i was expecting confirmation bias to be the main culprit. Pleasantly surprised to seei was wrong.

herodotus 8y ago

Nice article. It reminds me of my year living in London, and taking the bus everyday to Imperial College from West End Lane in West Hampstead. There was a stop on both sides of the road - one for the outbound bus, and one for the inbound (the bus went from central London to a terminus and then returned mostly on the same route). Now we did not use schedules - way too inaccurate at rush hour, and the busses there were pretty frequent anyway. But we did expect an even chance of the inbound bus arriving before an outbound one did. My daughter and I became convinced after a while that this was not happening, so we invented a game (which we called "The Game of Life".) When our bus (inbound) arrived first, we added 1 to our score. We subtracted 1 for every outbound bus that passed before ours arrived (there were often more than 1). We realized that the result would be slightly skewed to the negative, but we expected the outcome to be close to 0 over time. Of course it was not. Anyway we extended the game to many statistical situations. For example, you go to the checkout line at the supermarket, and there are N people in front of you. When you get to the front of the line, you count the people behind you - call that M. If M is bigger than N, you scored life points. If it is smaller, you lost some. So you add M-N to your running score, and you get an idea of how lucky you are in life. However, I never followed up with any real analysis, so I enjoyed this article.

asdkhadsj 8y ago
Wow, interesting idea! Imagine if (in the future) some item like a phone can detect this information around you, and automatically record it. Forming games ontop of this life data would be weird, neat, fun and sad all at the same time. Imagine seeing a real example of where someone else is just more lucky than you are in stupid but impactful (on your morale) ways. If it didn't seem so tedious to track, I'd love to implement an app to record this info. Unfortunately no one I know would care, and I'm sure I'd get too lazy to keep it accurate. Neat nonetheless, thanks for the cool thoughts :)
- adrianN 8y ago
  Similar to the recruiters that throw away the top half of the application stack because they don't want unlucky people in their company I could see such data become valuable to some people.

edoo 8y ago

Hah great analysis. One factor with bus' is the schedule is likely planned to minimize early arrivals at the risk of being late more often. Usually when a bus is early it has to sit and wait until its departure time. A late running bus can be more efficient, and if kept until departure time might not ever get a chance to average down the bursts of lateness.

stormbrew 8y ago
I don't think this is always true. My city measures buses as being on time at inner stops as being between something like 5 minutes early and 1 minute late (yes, you read that right). Timing stops, where they have to wait, are pretty infrequent (mostly bus terminals). I don't know how common it is but it does exist. And buses perpetually being early means that if you're on time you wait even longer for the next one.
8note 8y ago
a very late bus grabs everyone from each stop, so the next bus ends up early because it skips a bunch of less popular stops

twtw 8y ago

> a Poisson process is a memoryless process that assumes the probability of an arrival is entirely independent of the time since the previous arrival. In reality, a well-run bus system will have schedules deliberately structured to avoid this kind of behavior: buses don't begin their routes at random times throughout the day, but rather begin their routes on a schedule chosen to best serve the transit-riding public. I've never really understood any example involving a poisson process. They always seem to involve bus arrivals or light bulbs burning out, and I can't understand why the memory less property would ever make any sense for these. Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting? What is an actual phenomenon that is well modeled by a poisson process?

jobigoud 8y ago
> Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting? I don't think it's saying anything about how long you've been waiting, and you don't know when was the last arrival. It's saying that if you pick a random point on the timeline, the expected wait time doesn't change. That's because by taking a random point you have more chances of landing in a larger stretch of wait time than in a smaller one.
- twtw 8y ago
  > I don't think it's saying anything about how long you've been waiting, and you don't know when was the last arrival. This is exactly what memorylessness says something about. Your second paragraph isn't unique to Poisson processes, but the author right at the start says that the expected value of the waiting time is the same as the average interarrival time, which indicates Poisson.
bzbarsky 8y ago
> What is an actual phenomenon that is well modeled by a poisson process? Radioactive decay. Collisions of fluid molecules. Unstimulated (i.e. not in a laser) photon emission due to electron transitions in an atom. Lots of pretty memoryless stuff going on at the microscopic level.

nyc111 8y ago

> When waiting for a bus that comes on average every 10 minutes, your average waiting time will be 10 minutes. This is very ambiguous. Unless he gives a time frame the numbers do not make sense. Average in a week? Average in a year? This is not how it works in real life. And I cannot accept his premise. My experience tells me that, in New York, when I used to take a bus to work, sometimes the bus was coming as I was walking to the stop; sometimes I would wait a long time. Sometimes not very long. There was no observable bias.

pieguy 8y ago
In statistics, "average" often means "expected value". No time frame is specified (although you could consider it an infinite time frame). With a small sample size your actual average might not be 10 minutes, but as your sample size grows, it will tend toward 10 minutes.
- twtw 8y ago
  > it will tend toward 10 minutes If you are talking about spherical-cow style poisson buses, yeah (that's what the author means by "reasonable assumptions). But as the author concludes, bus arrival times are not well modeled by a poisson process.

mayankkaizen 8y ago

Nice article. Since I just started learning Stats, I wish I could find more of such notebooks. Any recommendations?

ikken 8y ago

This reminds me the bet in the bitcoin community [1]. If on average bitcoin blocks are produced every 10 minutes, and you learn that 5 minutes ago someone found a block, what is the average time you will wait for the next block? It turns out it's 10 minutes, not 5 minutes as you would intuitively think. (it's a memoryless process, so average expected time till block is always the same - 10 minutes - no matter how many blocks were recently found). In other words, when you're waiting for bitcoin transaction to be confirmed and go to check how long ago the most recent block was produced, in order to estimate how soon the next one will come - you're doing it wrong. Even if previous block was found 9 minutes ago, you're average waiting time for the next block is still 10 minutes. [1]. https://www.reddit.com/r/btc/comments/7rs8ko/dr_craig_s_wright_has_refused_to_pay_up_on_a_bet/ https://www.reddit.com/r/btc/comments/7rs8ko/dr_craig_s_wrig...

FabHK 8y ago
And a related counterintuitive fact (again, assuming 10 minutes): 1. If you pick a block randomly (uniformly), its average length is 10 minutes. 2. If you pick a point t0 in time randomly (uniformly), the average length of the block you're in is 20 mins (and the average length from t0 to next block is 10 mins, and the average length from previous block to t0 is also 10 mins (and, needless to say, 10+10=20...)).
kccqzy 8y ago
Here's an even simpler example: a flip of a fair coin. Suppose this is your first flip, one would intuitively think that there is 50% chance of H and 50% T. Suppose you flipped once and got H. For your second flip, one might intuitively think that since the number of times getting H over the long run is 50% of the total flips, and we already have a flip of H, to "balance it out" the next flip should have a smaller probability of getting H. Instead that is wrong. The next flip still has 50% chance of H. Suppose further that one has performed N flips, all of them H. One might even think that because of the way the geometric distribution works, it is very unlikely for the next flip to be H again. Instead that is wrong. The next flip still has 50% chance of H.
ikeboy 8y ago
This is actually wrong. The average expected time till next block is almost never 10 minutes because hashpower goes on and offline all the time. It gets adjusted every 2016 blocks based on historical block timing so that if no changes occur then future blocks would be 10 minutes on average - but changes always happen so this is never accurate. As such, you do learn something by looking at prior block times.

usgroup 8y ago

This is a straightforward consequence of modelling an arrival process as a Poisson distribution with a constant rate of arrival lambda... Go from arrival to cumulative arrivals to time of arrival to recurrence of arrival (next arrival). All are Poisson processes, including the recurrence process, which has a fixed expected value.

nakedrobot2 8y ago

In prague, the trams all run on time - within 2 minutes or less of the posted time. So I think this article is incorrect for this particular context.

albertgoeswoof 8y ago
Trams don’t have traffic in the way that buses do, and there are only minor differences in the drivers that could cause anomalies (I.e. you can’t steer a tram the wrong way) so they’re much easier to keep on time.

agumonkey 8y ago

Slightly related, my ghost town had few buses and sparses. I could never rely on printed hours. If I got there 10 min earlier to be sure, I'd still never be sure I'd wait 20 min for nothing because it was 11 min early. Of course half the time if I decide to walk to the next town where buses are many, I'd see all my town buses (both ways) pass me <yell-at-cloud.png> I think it made me completely careless about time, I would just go between stops and take the first one, go with the flow. By experience I'd know the range it would take for me to reach big places around the area. I had a friend who was completely foreign to this mode of thinking, she was very dilligent and fully trusting (although she mostly used trains so a lot less divergence). It reminds me of kid studies about intelligence / wealth ratios. When you're environment is random, you think random. When it's predictable you planify.

ChrisFoster 8y ago

It strikes me that even with a perfectly regular starting schedule, buses might clump together in time because the schedule is probably dynamically unstable. To explain, picking up passengers from a stop costs time and a long time between buses implies a high probability that passengers will be waiting at a given stop. This further adding to the delay and shortens the time to the next bus in the schedule. I'm sure drivers try to actively manage this, but if they didn't I suspect the system would naturally evolve toward pairs of buses leapfrogging each other on long routes.

jobigoud 8y ago
I think another confusing factor about that specific example is that bus shouldn't ever start before their schedule. Otherwise you run the risk of a bunch of people missing their bus even though they showed up on time. I think bus, trains and planes can only be late. For example this is an article about a Japanese Train company issuing a public apology for departing 20 seconds early. https://www.bbc.com/news/world-asia-42009839 https://www.bbc.com/news/world-asia-42009839

ezoe 8y ago

I hate the poisson distribution because it completely against the naive instincts of how random behaves.

stornetn 8y ago

Reminds me of a similar article that measured a similar kind of question about the wait times for NYC subways conditional on how long you've been waiting (https://erikbern.com/2016/04/04/nyc-subway-math.html https://erikbern.com/2016/04/04/nyc-subway-math.html). I think it's a pretty safe bet that people who like this post will like this article as well.

akane 8y ago

On a related note, arrival time predictions can be biased early to prevent people from missing buses, which also increases the perception of lateness. https://nextbus.cubic.com/FAQs https://nextbus.cubic.com/FAQs

graycat 8y ago

It's all much easier than that: It's just the Poisson process, e.g., with a nice chapter in E. Cinlar, Introduction to Stochastic Processes. Buses come as arrivals. So bus arrivals are a stochastic arrival process where stochastic just means varying randomly over time where, really, the randomly doesn't mean anything, includes deterministic arrivals, that is, known exactly in advance, but also admits any case of unpredictability. Well, in short, if have a stochastic arrival process with stationary, independent increments, then the arrival process is a Poisson process and there is a number, usually denoted by lambda, so that the times between arrivals are independent, identically distributed random variables with exponential distribution with arrival parameter, the arrival rate, lambda. The stationary means that the probability distribution of the times between arrival does not change over time. The independent increments means that the time from one arrival to the next is independent of all the past history of arrivals. The exponential distribution has the property, easy to verify with simple calculus, that the conditional expectation of the arrival time given that the arrival time is already greater than some number is the same as the expected arrival time. So, net, if bus arrivals form a Poisson process, then the time until the next bus arrives is the same after waiting five minutes as not having waited at all. Cinlar's treatment is nice because it is qualitative, that is, has assumptions that can often be confirmed or believed just intuitively. And we might not believe that bus arrivals meed the assumptions. This subject can continue with, say, hazard curves for equipment failures and a lot more about Poisson processes. E.g., the sum of two independent Poisson processes, say, Red buses and Blue buses, assuming that they are Poisson processes, is also a Poisson process with arrival rate the sum of the Red and Blue arrival rates. If randomly throw away some arrivals, then what is left is also a Poisson process with arrival rate adjusted in the obvious way. In Feller's volume II is the renewal theorem that the sum of independent arrival processes, Poisson or not, with mild assumptions, converges to a Poisson process as the number of processes summed grows. So, if the users of a sufficiently busy Web site act independently with mild assumptions, then the Web site will see arrivals accurately as a Poisson process. The vanilla Poisson process is Geiger counter clicks. There is much more to the pure and applied math and applications of Poisson processes.

PascLeRasc 8y ago

Is the inspection paradox what would happen if you surveyed everyone on how many siblings they had, and every sibling double-counted N-1 times (where N is the number of siblings in their family), inflating the resulting "average number of siblings", or is that something different?

combatentropy 8y ago
This is my first exposure to it, but yes, I think so. My paraphrase is: The Inspection Paradox is when you ask someone "in the mix" about the mix. You're only going to get an accurate estimation of the mix by standing outside of the mix. So yes, if you want an accurate count of siblings, you would consult some spreadsheet that just lists how many children each family had. If you go and start asking the families themselves (those "in the mix") then your results will be skewed. I thought the article that this article linked to was also very good, "The Inspection Paradox Is Everywhere," by Allen Downey, http://allendowney.blogspot.com/2015/08/the-inspection-paradox-is-everywhere.html http://allendowney.blogspot.com/2015/08/the-inspection-parad...

torgian 8y ago

I’m glad I live in east Asia. Busses and trains are almost never late