7 ms·

What happens when you add a new teller? (2008)

78 points by isp 11y ago

isp 11y ago
The related post, "Server utilization: Joel on queuing" (2009), includes a useful visualisation: http://www.johndcook.com/blog/2009/01/30/server-utilization-joel-on-queuing/ http://www.johndcook.com/blog/2009/01/30/server-utilization-...
SocksCanClose 11y ago
The inverse corollary being the essay (I can't find it) about the sociology of lines at Tokyo Disneyland. Question: what happens when there's no line for a specific ride at Tokyo Disneyland? Answer: nobody wants to ride the ride! Second question: what happens when there's a line for a specific ride at Tokyo Disneyland? Answer: everybody starts queuing up for that ride, even if they don't know which ride it is!
- deleted 11y ago
  [deleted]
- derefr 11y ago
  So... have (plainclothes) park staff queue for the empty rides?
  - SocksCanClose 11y ago
    yeah, they actually do!
a_c_s 11y ago
I think this would be far more informative if they started by explaining the single teller case: it isn't obvious how customers that arrive at a rate of slightly less than 1 every 10 minutes and take 10 minutes to serve would have an average waiting time of 5 hours.
- basseq 11y ago
  I think it's non-obvious because it quickly gets into queuing theory. To simplify (and expose a flaw in the case[1]), assume a bank is open 9am to 5pm (8 hours). They'd expect to see 46(.4, but let's round) people in a day. So there's two factors: flow rate and service time. FLOW RATE: Those people aren't coming in the door every 10.3 minutes: they're all coming in at 9:00 before work, 12:00 on their lunch breaks, or 4:45 right before closing. So you'd see 15-20 people all working through the doors within 10-20 minutes of each other. Ouch! This is modeled in queue theory as a Poisson distribution of arrival times. SERVICE TIME: Again, we're dealing with averages and distributions, but this time it's exponential. For every person who comes in with a check to deposit (let's say that takes 3m), there's another extreme case (let's say someone who wants to deposit $2,500 in Canadian pennies). As someone in the back of the line, you have to wait for all customers before you to be served before it's your turn. All of a sudden you're in a 29-person line and waiting 5 hours. THAT math makes sense: 29 people × 10 minutes per person = 4.8 hours. THE FLAW [1] Of course, this queue model is continuous: the bank doesn't open or close (as most banks do). Moreover, arrival times are deterministic: you can model based on a distribution, but you could quickly measure expected arrival times. Process efficiency and queue theory are interesting topics. My favorite case is Toyota's six sigma production line engineers helping a NYC food kitchen cut wait times from 90 minutes to 18 minutes with simple adjustments: http://www.nytimes.com/2013/07/27/nyregion/in-lieu-of-money-toyota-donates-efficiency-to-new-york-charity.html?_r=0 http://www.nytimes.com/2013/07/27/nyregion/in-lieu-of-money-...
  - thaumasiotes 11y ago
    That was depressing. The article details the adjustments Toyota made to the food kitchen's system: > The kitchen, which can seat 50 people, typically opened for dinner at 4 p.m., and when all the chairs were filled, a line would form outside. Mr. Foriest would wait for enough space to open up to allow 10 people in. The average wait time could be up to an hour and a half. > [Toyota] eliminated the 10-at-a-time system, allowing diners to flow in one by one as soon as a chair was free. Talk about low-hanging fruit. This is quite literally a case of cutting waiting times by saying "hey, why don't we just stop telling people they have to wait?"
rcarrigan87 11y ago
2 personal observations at Disney: Wait time clocks are normally inflated by around 15-20% on really popular rides. Generally, if the line splits, the left side is shorter bc more people are right handed and tend to go right. No data to support either claim, just what I've experienced. I'd be really curious to see some of Disney's work in line theory.
- deleted 11y ago
  [deleted]
- tgb 11y ago
  Is it right-handedness or driving on the right side?
thrownaway2424 11y ago
This analogy will not resonate with the under-30 crowd. I realized the other day that my kids don't have firsthand recollection of what "going to the bank" might be.
- rwmj 11y ago
  Queuing up to send your ebay parcels might be a more current example. That's my frequent experience at our local post office anyway.
  - mikeash 11y ago
    How about a supermarket self-checkout line, and the difference between one terminal and two?
- wldcordeiro 11y ago
  Under-30 here, the analogy makes sense, I have to go to my bank somewhat regularly for things like check cashing, transferring funds from accounts to others and getting financial advice. I think you're mistaking people under 30 with people who haven't lived by themselves and handled their own finances which is a subgroup of under-30's.
  - ominous_prime 11y ago
    I'm financially independent, mid 30s, married with kids, and I haven't been in my bank branch in years. I agree that under 30s should be able to relate, but technology is changing how often I physically queue up somewhere.
    - kibibu 11y ago
      Ditto, and my bank doesn't even have branches. Last time I stepped into a physical bank to do actual banking was somewhere around dickity 06.
- Dewie3 11y ago
  I'm pretty sure that I've been to a bank.
cabacon 11y ago
See also a very nice video from ClojureWest about queues in system architectures: https://www.youtube.com/watch?v=1bNOO3xxMc0 https://www.youtube.com/watch?v=1bNOO3xxMc0 The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p85 latency rises, you're already about to stall out.
zw123456 11y ago
Anyone with experience in telecom will immediately recognize this as a traffic engineering problem easily solved with the Erlang formula: http://en.wikipedia.org/wiki/Erlang_(unit) http://en.wikipedia.org/wiki/Erlang_(unit) Not to be confused with the Erlang Language. ErlangC assumes queuing, which is probably applicable to this case.
- OldSchool 11y ago
  Wow, out of 28 comments at the moment, yours is the only one to point this out.
theVirginian 11y ago
Interesting conclusion. No explanation.
marcus_holmes 11y ago
Interesting to note the MBA view: If the one teller is 100% busy serving customers, then that's efficient. The queue is clearly temporary and due to demand spikes If two tellers are idle 70% of the time, that's inefficient and one of them should be relocated. I've had similar discussions with non-tech management about server utilisation (why should we buy another server when we're only using 80% capacity of the ones we've got?)
- vacri 11y ago
  Appeal to their marketing side: excess peak demand represents angry clients who will take their business to the bank across the street, which does have two tellers and no wait time.
instakill 11y ago
A great guide on this can be found: http://ss15-teropa.divshot.io/ http://ss15-teropa.divshot.io/
ScottWhigham 11y ago
It helped me to visualize the problem. I don't know if HN's formatting will kill this but here goes. The idea is that six customers arrive at 9:00, 01, 02, 03, 04, 05, and 06 - then map them out to "What happens when you add a new teller?" I tried to figure out how the author's explanation that adding one teller can go from 5 hours to 3 minutes. Edit - the text version was a disaster here. Here is an image of what I used to try to understand: http://i.imgur.com/HOKgfmd.png http://i.imgur.com/HOKgfmd.png One teller: average wait time is 22.5 minutes for "six users who arrive one minute apart from 9:00 to 9:05" Two tellers: average wait time is 8 minutes. I think to go from "5 hours with one teller" to "Three minutes with two tellers" requires you to space people out more than "one every one minute".

78 points by isp 11y ago

isp 11y ago

The related post, "Server utilization: Joel on queuing" (2009), includes a useful visualisation: http://www.johndcook.com/blog/2009/01/30/server-utilization-joel-on-queuing/ http://www.johndcook.com/blog/2009/01/30/server-utilization-...

SocksCanClose 11y ago

The inverse corollary being the essay (I can't find it) about the sociology of lines at Tokyo Disneyland. Question: what happens when there's no line for a specific ride at Tokyo Disneyland? Answer: nobody wants to ride the ride! Second question: what happens when there's a line for a specific ride at Tokyo Disneyland? Answer: everybody starts queuing up for that ride, even if they don't know which ride it is!

deleted 11y ago
[deleted]
derefr 11y ago
So... have (plainclothes) park staff queue for the empty rides?
- SocksCanClose 11y ago
  yeah, they actually do!

a_c_s 11y ago

I think this would be far more informative if they started by explaining the single teller case: it isn't obvious how customers that arrive at a rate of slightly less than 1 every 10 minutes and take 10 minutes to serve would have an average waiting time of 5 hours.

basseq 11y ago
I think it's non-obvious because it quickly gets into queuing theory. To simplify (and expose a flaw in the case[1]), assume a bank is open 9am to 5pm (8 hours). They'd expect to see 46(.4, but let's round) people in a day. So there's two factors: flow rate and service time. FLOW RATE: Those people aren't coming in the door every 10.3 minutes: they're all coming in at 9:00 before work, 12:00 on their lunch breaks, or 4:45 right before closing. So you'd see 15-20 people all working through the doors within 10-20 minutes of each other. Ouch! This is modeled in queue theory as a Poisson distribution of arrival times. SERVICE TIME: Again, we're dealing with averages and distributions, but this time it's exponential. For every person who comes in with a check to deposit (let's say that takes 3m), there's another extreme case (let's say someone who wants to deposit $2,500 in Canadian pennies). As someone in the back of the line, you have to wait for all customers before you to be served before it's your turn. All of a sudden you're in a 29-person line and waiting 5 hours. THAT math makes sense: 29 people × 10 minutes per person = 4.8 hours. THE FLAW [1] Of course, this queue model is continuous: the bank doesn't open or close (as most banks do). Moreover, arrival times are deterministic: you can model based on a distribution, but you could quickly measure expected arrival times. Process efficiency and queue theory are interesting topics. My favorite case is Toyota's six sigma production line engineers helping a NYC food kitchen cut wait times from 90 minutes to 18 minutes with simple adjustments: http://www.nytimes.com/2013/07/27/nyregion/in-lieu-of-money-toyota-donates-efficiency-to-new-york-charity.html?_r=0 http://www.nytimes.com/2013/07/27/nyregion/in-lieu-of-money-...
- thaumasiotes 11y ago
  That was depressing. The article details the adjustments Toyota made to the food kitchen's system: > The kitchen, which can seat 50 people, typically opened for dinner at 4 p.m., and when all the chairs were filled, a line would form outside. Mr. Foriest would wait for enough space to open up to allow 10 people in. The average wait time could be up to an hour and a half. > [Toyota] eliminated the 10-at-a-time system, allowing diners to flow in one by one as soon as a chair was free. Talk about low-hanging fruit. This is quite literally a case of cutting waiting times by saying "hey, why don't we just stop telling people they have to wait?"

rcarrigan87 11y ago

2 personal observations at Disney: Wait time clocks are normally inflated by around 15-20% on really popular rides. Generally, if the line splits, the left side is shorter bc more people are right handed and tend to go right. No data to support either claim, just what I've experienced. I'd be really curious to see some of Disney's work in line theory.

deleted 11y ago
[deleted]
tgb 11y ago
Is it right-handedness or driving on the right side?

thrownaway2424 11y ago

This analogy will not resonate with the under-30 crowd. I realized the other day that my kids don't have firsthand recollection of what "going to the bank" might be.

rwmj 11y ago
Queuing up to send your ebay parcels might be a more current example. That's my frequent experience at our local post office anyway.
- mikeash 11y ago
  How about a supermarket self-checkout line, and the difference between one terminal and two?
wldcordeiro 11y ago
Under-30 here, the analogy makes sense, I have to go to my bank somewhat regularly for things like check cashing, transferring funds from accounts to others and getting financial advice. I think you're mistaking people under 30 with people who haven't lived by themselves and handled their own finances which is a subgroup of under-30's.
- ominous_prime 11y ago
  I'm financially independent, mid 30s, married with kids, and I haven't been in my bank branch in years. I agree that under 30s should be able to relate, but technology is changing how often I physically queue up somewhere.
  - kibibu 11y ago
    Ditto, and my bank doesn't even have branches. Last time I stepped into a physical bank to do actual banking was somewhere around dickity 06.
Dewie3 11y ago
I'm pretty sure that I've been to a bank.

cabacon 11y ago

See also a very nice video from ClojureWest about queues in system architectures: https://www.youtube.com/watch?v=1bNOO3xxMc0 https://www.youtube.com/watch?v=1bNOO3xxMc0 The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p85 latency rises, you're already about to stall out.

zw123456 11y ago

Anyone with experience in telecom will immediately recognize this as a traffic engineering problem easily solved with the Erlang formula: http://en.wikipedia.org/wiki/Erlang_(unit) http://en.wikipedia.org/wiki/Erlang_(unit) Not to be confused with the Erlang Language. ErlangC assumes queuing, which is probably applicable to this case.

OldSchool 11y ago
Wow, out of 28 comments at the moment, yours is the only one to point this out.

theVirginian 11y ago

Interesting conclusion. No explanation.

marcus_holmes 11y ago

Interesting to note the MBA view: If the one teller is 100% busy serving customers, then that's efficient. The queue is clearly temporary and due to demand spikes If two tellers are idle 70% of the time, that's inefficient and one of them should be relocated. I've had similar discussions with non-tech management about server utilisation (why should we buy another server when we're only using 80% capacity of the ones we've got?)

vacri 11y ago
Appeal to their marketing side: excess peak demand represents angry clients who will take their business to the bank across the street, which does have two tellers and no wait time.

instakill 11y ago

A great guide on this can be found: http://ss15-teropa.divshot.io/ http://ss15-teropa.divshot.io/

ScottWhigham 11y ago

It helped me to visualize the problem. I don't know if HN's formatting will kill this but here goes. The idea is that six customers arrive at 9:00, 01, 02, 03, 04, 05, and 06 - then map them out to "What happens when you add a new teller?" I tried to figure out how the author's explanation that adding one teller can go from 5 hours to 3 minutes. Edit - the text version was a disaster here. Here is an image of what I used to try to understand: http://i.imgur.com/HOKgfmd.png http://i.imgur.com/HOKgfmd.png One teller: average wait time is 22.5 minutes for "six users who arrive one minute apart from 9:00 to 9:05" Two tellers: average wait time is 8 minutes. I think to go from "5 hours with one teller" to "Three minutes with two tellers" requires you to space people out more than "one every one minute".