4 ms·

Alice is impatient

148 points by birdculture 4mo ago

trb 4mo ago
Considering other metrics then p99 for user impact is unwise. All users will at some point experience a <1% request, it's not like half of all users will only send requests what will be under your median latency, some of their requests will hit your worst-case. By focusing on the tail and optimizing worst cases you help users more than by improving your median latency.
- deleted 4mo ago
  [deleted]
- the8472 4mo ago
  If your frontend fires hundreds of requests (which isn't uncommon) then the p99 is merely what most users will experience. Ideally you want cumulative distribution chart that goes up to the max. And then that's just for the requests you measure. If something takes too long the user might do something that cancels the requests which means the backend never completes its response and won't get the time-to-response sample, so you need to account dropped requests too. https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU
  - TiddoLangerak 4mo ago
    This is only true if your latency distribution is fully random, which is rarely the case. More often than not, it's the same small group of users hitting most of the p99 because their accounts are simply more resource intensive.
    - the8472 4mo ago
      Depending on how the system distributes work such users can interfere with random with requests from other users through shared resources, so to that cohort these will look like a random latency distribution.
deleted 4mo ago
[deleted]
rustybolt 4mo ago
This article contains very little substance. Show me the math!
- AgentOrange1234 4mo ago
  Yes I found this very hard to follow. I appreciate expressing ideas in math like E_a[X] as much as the next guy, but there is no definition or even description of what the heck E or E_a or Var(x) even mean, so how is anyone supposed to understand the reasoning here? All I get from this is a claim that experienced latency is different than the mean, which sounds important, but I still have no intuition as to why this is. Which is sad, because Booker's blog is often deeply amazing.
  - NightMKoder 4mo ago
    This is standard statistics terminology - E(X) is https://en.wikipedia.org/wiki/Expected_value https://en.wikipedia.org/wiki/Expected_value . E_a is presumably Alice's perceived expected value. Var(X) is https://en.wikipedia.org/wiki/Variance https://en.wikipedia.org/wiki/Variance . The law of large numbers says the arithmetic average of observations becomes E(X) with enough samples. I'm pretty sure what the author is saying is: E(X) =:= \sum_t(t * P(X = t)) is the definition another important note is P(X^2 = t^2) = P(X = t) - because it's the same distribution. E_a(X) is a bit sloppy, but consider X_a aka Alice's latency "experience" distribution. The argument is: P(X_a = t) = t * P(X = t) / \sum_u(u * P(X = u)) - i.e. scale the probability up by t but make it sum to 1. Then E(X_a) = \sum_t(t * P(X_a = t)) = \sum_t(t * t * P(X = t) / \sum_u(u * P(X = u)) aka E(X^2) / E(X) Then (from wikipedia) Var(X) = E(X^2) - (E(X))^2 And we get E(X_a) = (Var(X) + (E(X))^2) / E(X) = E(X) + Var(X) / E(X)
    - gruberben 4mo ago
      Thanks!
  - canjobear 4mo ago
    I was clicking around in response to this article and found this video that explains the inspection paradox nicely. https://youtu.be/Jd1wNizPjoE https://youtu.be/Jd1wNizPjoE
zaik 4mo ago
Is the formula for E_a[X] trivial? I don't see it immediately...
- kgwgk 4mo ago
  E[X^2] weights each time with the time, giving the square, and the E[X] in the denominator is the normalisation factor (also required to fix the dimensions). Say that there are to different waiting times 1s and 3s, and they happen with probability 50% each. The average waiting time (1/2 1+1/2 3) is 2s. However, 75% of the time we are waiting on a 3s event and only 25% on a 1s event. The weighted average is 2.5s. E[X^2]=1/2 1+1/2 9=5(s^2) is not the right answer, it still has to be divided by E[X]=2(s) to get the correct answer.
perching_aix 4mo ago
I've grown to dislike the typical tail measurements completely. What I usually look at these days is what share of unique users experience an "unacceptable experience" over a measurement period instead. I find it much more inquisitive and visceral, to the extent that p99 now boggles my mind. 2N would be dreadful as an availability figure, yet for UX it's treated very different. So much so that my measurements corroborate exactly that; good UX requires the same many-nines reliability as e.g. DCs, not one or two. I wonder if it's p90 and p99 to blame for the shoddy services we have, in a way. It's pretty hard to argue for fixing something when it's presented as only going wrong 0.5% or less of the time after all. Even if at scale that means most of your users are experiencing it weekly.
- quixoticaxolotl 4mo ago
  How does one measure unique users here in a way different from classic p99? I usually associate p99 with an SLO of some kind, and each request as a "unique user" for the service, so at first it seems like the same thing - measuring p99 with a SLO would say 1% of users are allowed to experience a time longer than our acceptable minimum T, and you're measuring the percentage of requests ("users") experiencing T and trying to keep it below 1% (e.g.). Is the difference more about measuring a request "across services"? That is, the total cumulative p99 across services must be small i.e. linking all requests to a user and then measuring that? Or is the difference elsewhere? If the former: are you taking traces and graphing that? What's your methodology?
  - perching_aix 4mo ago
    I think you got it, but let me maybe lay it out more explicitly with a specific example. I visit HN, that's one request. But I visit HN multiple times a day. So for the operation that serves the homepage, if you took e.g. a past 24hr latency p99 chart, the number of requests analyzed would not be the same as the number of unique users involved in making those requests, potentially drastically so. So you might see a p99 you're comfortable with, and conclude that since only 1% of requests were worse than that, it's fine. In practice though, depending on how "well-trodden" that operation is, you might very well be in a situation where all users experienced at least one such beyond-SLO event that day, since the mapping is many to one. The cross operation version of this is important as well, yes. You can have users experience snags across common flows too for example, same idea. Regarding methodology, it's nothing special, I just rely on user IDs and correlation IDs. It really is just a perspective shift, the underlying data is the same. You can even calculate back the "number of nines needed to get an acceptable UX" using this, as long as the general usage habits are stable. It's just gonna be a lot more nines than two in my experience.
ggm 4mo ago
Interesting you work at Amazon and show how end user experience weights to their pessimal experience. So.. apply that to Amazon design heuristics like author name search on books, and how Amazon return "in the style of" and "not a book but this guy called Charles Dickens makes jigsaws" as high order matches and consider how the end user experience weights to the pessimal yet Amazon can show on average they make more money doing this.. (Understood that engineers and AWS don't influence UX in the storefront or search)
- cowthulhu 4mo ago
  Comments like these seem likely to discourage authors from making more interesting posts about niche topics they specialize in, without actually moving the needle on stuff the commenter is pestering about.
  - ggm 4mo ago
    Yes, has that risk. If I were to neutralise it I'd observe any system which assesses UX will trip over this, the majors are no exception. It serves as a useful reminder tuning isn't optimised solely for your(as a user) benefit.
uberex 4mo ago
> More technically, what’s going on here is the inspection paradox. Alex and Alice don’t experience your latency distribution , they experience a t-weighted version of it Ooh I got pushed in the 2m end of the pool there. What is the intuition? The ten hundred most popular words sort of thing. I am very interested in this article though. At first I assumed it would be about TTFB vs. time to render the page after all those async useEffects have run, but it isn't that this is something else and I am very interested.
- diatone 4mo ago
  AIUI: my intuition is Alex and Alice are points in the distribution. They don’t think about their experience in terms of population statistics. They see their individual latency times, and use that as their sample. If t is low in their experience, great the distribution is low. But for any t that goes high that they observe (which tends to be the case in a skewed distribution such as service latencies), it drags their impression of the distribution up, dominating the shape of that impression.
- mungoman2 4mo ago
  Similarly curious about this. The intuition I extracted: Let’s say we have 10 requests, where 9 of them take 1 second to complete but one that takes 100 seconds. The average time to complete a request is about 10 seconds, but if you experience the requests in series, at any given time you’re much more likely to sit and wait in one of those 100 second requests. So if you imagine a long series of requests from this distribution and place yourself randomly in the series, the average time to completion is just a bit less than 50 seconds. This is what is meant by t-weighted, that events with a large t take a larger place.
  - uberex 4mo ago
    I see it is about a long series of requests. Makes sense. Ill start looking at latency at p99.9 and p99.99 more often now!
- CrazyStat 4mo ago
  Perhaps an easier to intuit version of it is how full airliners are. An airline might report that their flights are on average 60% full, and that might be completely absolutely 100% true. But that's not what passengers experience. If we assume (for convenience) that a plane holds 100 people, when the plane is 20% full then 20 passengers experience that, but when the plane is 100% full then 100 passengers experience that. On average, from a passenger's point of view, the flights are much more than 60% full--it might be 70 or 80%--because a full flight is experienced by more passengers than an empty flight. For a concrete example imagine two flights, one 20% full and one 100% full: the average is 60% from the airline's point of view, but 100 passengers experienced a full flight and only 20 experienced the 20% full flight, so from the passenger's point of view the average is 86.7% full. The same logic applies to outages. If you have an outage that lasts one minute then only a few users will encounter it. If you have an outage that lasts one hour then many more users will encounter that. The longer the outage is, the more likely any given user is to encounter it, so from the user's point of view the "average" outage is much longer than the "true" average where you weight every outage equally. Again we can consider a concrete example: imagine you run a website that gets 100 visitors per minute. You have one outage that lasts 1 minute, then later a second outage that lasts 9 minutes. Your average outage time is 5 minutes. But 100 visitors experienced the 1 minute outage, while 900 visitors experienced the 9 minute outage, so from the point of view of a visitor the average outage is (900*9 + 100*1)/1000 = 8.2 minutes.
deleted 4mo ago
[deleted]
amluto 4mo ago
I found this article to be a very poor explanation of what (I think) it’s trying to say. I think the point the article ought to be making is much better handled entirely separately for request time and for outages. For outages, it goes something like this: if you have a 1 hour outage, and your user notices that outage, they think you had a 1 hour outage. [0] If you do statistics that observe that you also had ten thousand 1 second outages and thus had an MTTR of under two seconds, this does not excuse your 1 hour outage in the slightest. And the longer an outage is, the more likely that any given user interacts with your service during the outage. But the article is oddly caught up in this t-weighting idea, without justification. What does the statement that “Alex and Alice experience E_a[X]” even mean. What’s X? Is it the distribution of request times? If so, then I don’t see the article’s point — if I, a user, sample a bunch of requests, I recover an approximation of X, not X^2. And I really hope that X is not intended to be the distribution of outage lengths because I think the conclusion is just wrong as I alluded to above. Sure, if I happen to sample your service during an outage, the probability that I sample any specific outage is proportional to the length of that outage, but what about all the times that your are (hopefully) not having an outage? What if you have two consecutive outages that are so close to each other in time that I don’t think you recovered? It would be entertaining to make an outage website. I’d pick a distribution over outage lengths. At time 0 I would sample that distribution, get an outage length t, and declare myself down until time t. All requests during that “outage” would report “hey, I’m down, and my outage length is t”. After the “outage” the site would sample again and repeat. This would give the answer in the article. But this is, of course, absurd. [0] To be pedantic, they may not notice the beginning of the outage. This is a constant factor correction.
glitchc 4mo ago
Thank you for writing this article, there's a deep and powerful insight illustrated here: An observer using the system experiences different statistics from the system operator. By extension, taking an average of observer experiences leads to different conclusions from taking an average of system performance. One must not confuse the two when designing systems.
penteract 4mo ago
This is the same calculation behind the observation "you spend much longer in front of red traffic lights than green ones". It's an interesting observation, but it's playing games with the meaning of "mean latency" and I'm not sure this is a very helpful way to look at requests to a web service - slightly slowing the fastest responses to requests would improve your time-weighted mean latency. It's a better metric for looking at outages - instantaneous outages don't affect anyone, and time-weighting correctly discards them. On the other hand, average outage length is a very suspect and gameable metric unless accompanied by uptime %.
reisse 4mo ago
There is a branch of math dedicated to (among other things) truthfully estimating the waiting time, called queueing theory. I wonder why it wasn't mentioned in the article.
pfdietz 4mo ago
This feels similar to how nuclear power is perceived, contrasting deaths per TWh and the long tail effect of a rare but serious accident.
michaelt 4mo ago
> Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. There are also plenty of situations where a service can have a bimodal performance distribution and the impact of that can fall on certain users disproportionately. Imagine a retail website that serves images from a global CDN, with cache misses pulled from a server in the EU. Users who visit our homepage, or look at our bestselling products, get a cache hit from the CDN node close to them, in 50ms. But users who look at our long-tail products get a cache miss - and if they're not near Europe, they'll get a noticeable delay. Hence our mean image load time is 100ms - but a customer browsing an obscure product category for their location can experience markedly worse performance. If Alice is the only person in Costa Rica looking at ski equipment in June, she's going to get a lot of cache misses.
- esperent 4mo ago
  So what's the solution here? Keep the cache artificially warm for even obscure routes?
  - steveBK123 4mo ago
    There may not be a solution in every case, but it's a reminder that dashboards & metrics are no replacement for actually talking to your users. Metrics are at best a proxy for user experience, don't let them be the tail that wags the dog. Like the story Bezos told of his execs claiming call wait times were under 1 minute, so he called the service line from the conference room on the spot and made everyone sit there for 10 minutes waiting to get thru..
    - esperent 4mo ago
      Ok, that's a fun anecdote and I agree it has real world value to think that way. But it doesn't answer my question - this thread is full of people pointing out problems but nobody offering solutions. So I was asking, specifically in the case of an app with a very long tail of cache misses, what's the solution? Do you have to keep potentially millions of routes artificially warm?
mdavid626 4mo ago
I don’t remember any service I used in the last couple of years, where I thought to myself: this service is really fast and responsive. Great experience. Quite the contrary: feels like everything got worse. Sometimes painfully slow, buggy and unreliable.
- tadfisher 4mo ago
  Try the McMaster-Carr website sometime. It's incredible. Expensive, but incredible.
x1798DE 4mo ago
My understanding is a lot of the probability puzzles in Allen Downey's Probably Overthinking It [1] also boil down to similar selection effects (the inspection paradox is definitely in there). There is a lot of cool stuff in that book (and his blog of the same name). 1. https://greenteapress.com/wp/probably-overthinking-it/ https://greenteapress.com/wp/probably-overthinking-it/

148 points by birdculture 4mo ago

trb 4mo ago

Considering other metrics then p99 for user impact is unwise. All users will at some point experience a <1% request, it's not like half of all users will only send requests what will be under your median latency, some of their requests will hit your worst-case. By focusing on the tail and optimizing worst cases you help users more than by improving your median latency.

deleted 4mo ago
[deleted]
the8472 4mo ago
If your frontend fires hundreds of requests (which isn't uncommon) then the p99 is merely what most users will experience. Ideally you want cumulative distribution chart that goes up to the max. And then that's just for the requests you measure. If something takes too long the user might do something that cancels the requests which means the backend never completes its response and won't get the time-to-response sample, so you need to account dropped requests too. https://www.youtube.com/watch?v=lJ8ydIuPFeU https://www.youtube.com/watch?v=lJ8ydIuPFeU
- TiddoLangerak 4mo ago
  This is only true if your latency distribution is fully random, which is rarely the case. More often than not, it's the same small group of users hitting most of the p99 because their accounts are simply more resource intensive.
  - the8472 4mo ago
    Depending on how the system distributes work such users can interfere with random with requests from other users through shared resources, so to that cohort these will look like a random latency distribution.

deleted 4mo ago

[deleted]

rustybolt 4mo ago

This article contains very little substance. Show me the math!

AgentOrange1234 4mo ago
Yes I found this very hard to follow. I appreciate expressing ideas in math like E_a[X] as much as the next guy, but there is no definition or even description of what the heck E or E_a or Var(x) even mean, so how is anyone supposed to understand the reasoning here? All I get from this is a claim that experienced latency is different than the mean, which sounds important, but I still have no intuition as to why this is. Which is sad, because Booker's blog is often deeply amazing.
- NightMKoder 4mo ago
  This is standard statistics terminology - E(X) is https://en.wikipedia.org/wiki/Expected_value https://en.wikipedia.org/wiki/Expected_value . E_a is presumably Alice's perceived expected value. Var(X) is https://en.wikipedia.org/wiki/Variance https://en.wikipedia.org/wiki/Variance . The law of large numbers says the arithmetic average of observations becomes E(X) with enough samples. I'm pretty sure what the author is saying is: E(X) =:= \sum_t(t * P(X = t)) is the definition another important note is P(X^2 = t^2) = P(X = t) - because it's the same distribution. E_a(X) is a bit sloppy, but consider X_a aka Alice's latency "experience" distribution. The argument is: P(X_a = t) = t * P(X = t) / \sum_u(u * P(X = u)) - i.e. scale the probability up by t but make it sum to 1. Then E(X_a) = \sum_t(t * P(X_a = t)) = \sum_t(t * t * P(X = t) / \sum_u(u * P(X = u)) aka E(X^2) / E(X) Then (from wikipedia) Var(X) = E(X^2) - (E(X))^2 And we get E(X_a) = (Var(X) + (E(X))^2) / E(X) = E(X) + Var(X) / E(X)
  - gruberben 4mo ago
    Thanks!
- canjobear 4mo ago
  I was clicking around in response to this article and found this video that explains the inspection paradox nicely. https://youtu.be/Jd1wNizPjoE https://youtu.be/Jd1wNizPjoE

zaik 4mo ago

Is the formula for E_a[X] trivial? I don't see it immediately...

kgwgk 4mo ago
E[X^2] weights each time with the time, giving the square, and the E[X] in the denominator is the normalisation factor (also required to fix the dimensions). Say that there are to different waiting times 1s and 3s, and they happen with probability 50% each. The average waiting time (1/2 1+1/2 3) is 2s. However, 75% of the time we are waiting on a 3s event and only 25% on a 1s event. The weighted average is 2.5s. E[X^2]=1/2 1+1/2 9=5(s^2) is not the right answer, it still has to be divided by E[X]=2(s) to get the correct answer.

perching_aix 4mo ago

I've grown to dislike the typical tail measurements completely. What I usually look at these days is what share of unique users experience an "unacceptable experience" over a measurement period instead. I find it much more inquisitive and visceral, to the extent that p99 now boggles my mind. 2N would be dreadful as an availability figure, yet for UX it's treated very different. So much so that my measurements corroborate exactly that; good UX requires the same many-nines reliability as e.g. DCs, not one or two. I wonder if it's p90 and p99 to blame for the shoddy services we have, in a way. It's pretty hard to argue for fixing something when it's presented as only going wrong 0.5% or less of the time after all. Even if at scale that means most of your users are experiencing it weekly.

quixoticaxolotl 4mo ago
How does one measure unique users here in a way different from classic p99? I usually associate p99 with an SLO of some kind, and each request as a "unique user" for the service, so at first it seems like the same thing - measuring p99 with a SLO would say 1% of users are allowed to experience a time longer than our acceptable minimum T, and you're measuring the percentage of requests ("users") experiencing T and trying to keep it below 1% (e.g.). Is the difference more about measuring a request "across services"? That is, the total cumulative p99 across services must be small i.e. linking all requests to a user and then measuring that? Or is the difference elsewhere? If the former: are you taking traces and graphing that? What's your methodology?
- perching_aix 4mo ago
  I think you got it, but let me maybe lay it out more explicitly with a specific example. I visit HN, that's one request. But I visit HN multiple times a day. So for the operation that serves the homepage, if you took e.g. a past 24hr latency p99 chart, the number of requests analyzed would not be the same as the number of unique users involved in making those requests, potentially drastically so. So you might see a p99 you're comfortable with, and conclude that since only 1% of requests were worse than that, it's fine. In practice though, depending on how "well-trodden" that operation is, you might very well be in a situation where all users experienced at least one such beyond-SLO event that day, since the mapping is many to one. The cross operation version of this is important as well, yes. You can have users experience snags across common flows too for example, same idea. Regarding methodology, it's nothing special, I just rely on user IDs and correlation IDs. It really is just a perspective shift, the underlying data is the same. You can even calculate back the "number of nines needed to get an acceptable UX" using this, as long as the general usage habits are stable. It's just gonna be a lot more nines than two in my experience.

ggm 4mo ago

Interesting you work at Amazon and show how end user experience weights to their pessimal experience. So.. apply that to Amazon design heuristics like author name search on books, and how Amazon return "in the style of" and "not a book but this guy called Charles Dickens makes jigsaws" as high order matches and consider how the end user experience weights to the pessimal yet Amazon can show on average they make more money doing this.. (Understood that engineers and AWS don't influence UX in the storefront or search)

cowthulhu 4mo ago
Comments like these seem likely to discourage authors from making more interesting posts about niche topics they specialize in, without actually moving the needle on stuff the commenter is pestering about.
- ggm 4mo ago
  Yes, has that risk. If I were to neutralise it I'd observe any system which assesses UX will trip over this, the majors are no exception. It serves as a useful reminder tuning isn't optimised solely for your(as a user) benefit.

uberex 4mo ago

> More technically, what’s going on here is the inspection paradox. Alex and Alice don’t experience your latency distribution , they experience a t-weighted version of it Ooh I got pushed in the 2m end of the pool there. What is the intuition? The ten hundred most popular words sort of thing. I am very interested in this article though. At first I assumed it would be about TTFB vs. time to render the page after all those async useEffects have run, but it isn't that this is something else and I am very interested.

diatone 4mo ago
AIUI: my intuition is Alex and Alice are points in the distribution. They don’t think about their experience in terms of population statistics. They see their individual latency times, and use that as their sample. If t is low in their experience, great the distribution is low. But for any t that goes high that they observe (which tends to be the case in a skewed distribution such as service latencies), it drags their impression of the distribution up, dominating the shape of that impression.
mungoman2 4mo ago
Similarly curious about this. The intuition I extracted: Let’s say we have 10 requests, where 9 of them take 1 second to complete but one that takes 100 seconds. The average time to complete a request is about 10 seconds, but if you experience the requests in series, at any given time you’re much more likely to sit and wait in one of those 100 second requests. So if you imagine a long series of requests from this distribution and place yourself randomly in the series, the average time to completion is just a bit less than 50 seconds. This is what is meant by t-weighted, that events with a large t take a larger place.
- uberex 4mo ago
  I see it is about a long series of requests. Makes sense. Ill start looking at latency at p99.9 and p99.99 more often now!
CrazyStat 4mo ago
Perhaps an easier to intuit version of it is how full airliners are. An airline might report that their flights are on average 60% full, and that might be completely absolutely 100% true. But that's not what passengers experience. If we assume (for convenience) that a plane holds 100 people, when the plane is 20% full then 20 passengers experience that, but when the plane is 100% full then 100 passengers experience that. On average, from a passenger's point of view, the flights are much more than 60% full--it might be 70 or 80%--because a full flight is experienced by more passengers than an empty flight. For a concrete example imagine two flights, one 20% full and one 100% full: the average is 60% from the airline's point of view, but 100 passengers experienced a full flight and only 20 experienced the 20% full flight, so from the passenger's point of view the average is 86.7% full. The same logic applies to outages. If you have an outage that lasts one minute then only a few users will encounter it. If you have an outage that lasts one hour then many more users will encounter that. The longer the outage is, the more likely any given user is to encounter it, so from the user's point of view the "average" outage is much longer than the "true" average where you weight every outage equally. Again we can consider a concrete example: imagine you run a website that gets 100 visitors per minute. You have one outage that lasts 1 minute, then later a second outage that lasts 9 minutes. Your average outage time is 5 minutes. But 100 visitors experienced the 1 minute outage, while 900 visitors experienced the 9 minute outage, so from the point of view of a visitor the average outage is (900*9 + 100*1)/1000 = 8.2 minutes.

deleted 4mo ago

[deleted]

amluto 4mo ago

I found this article to be a very poor explanation of what (I think) it’s trying to say. I think the point the article ought to be making is much better handled entirely separately for request time and for outages. For outages, it goes something like this: if you have a 1 hour outage, and your user notices that outage, they think you had a 1 hour outage. [0] If you do statistics that observe that you also had ten thousand 1 second outages and thus had an MTTR of under two seconds, this does not excuse your 1 hour outage in the slightest. And the longer an outage is, the more likely that any given user interacts with your service during the outage. But the article is oddly caught up in this t-weighting idea, without justification. What does the statement that “Alex and Alice experience E_a[X]” even mean. What’s X? Is it the distribution of request times? If so, then I don’t see the article’s point — if I, a user, sample a bunch of requests, I recover an approximation of X, not X^2. And I really hope that X is not intended to be the distribution of outage lengths because I think the conclusion is just wrong as I alluded to above. Sure, if I happen to sample your service during an outage, the probability that I sample any specific outage is proportional to the length of that outage, but what about all the times that your are (hopefully) not having an outage? What if you have two consecutive outages that are so close to each other in time that I don’t think you recovered? It would be entertaining to make an outage website. I’d pick a distribution over outage lengths. At time 0 I would sample that distribution, get an outage length t, and declare myself down until time t. All requests during that “outage” would report “hey, I’m down, and my outage length is t”. After the “outage” the site would sample again and repeat. This would give the answer in the article. But this is, of course, absurd. [0] To be pedantic, they may not notice the beginning of the outage. This is a constant factor correction.

glitchc 4mo ago

Thank you for writing this article, there's a deep and powerful insight illustrated here: An observer using the system experiences different statistics from the system operator. By extension, taking an average of observer experiences leads to different conclusions from taking an average of system performance. One must not confuse the two when designing systems.

penteract 4mo ago

This is the same calculation behind the observation "you spend much longer in front of red traffic lights than green ones". It's an interesting observation, but it's playing games with the meaning of "mean latency" and I'm not sure this is a very helpful way to look at requests to a web service - slightly slowing the fastest responses to requests would improve your time-weighted mean latency. It's a better metric for looking at outages - instantaneous outages don't affect anyone, and time-weighting correctly discards them. On the other hand, average outage length is a very suspect and gameable metric unless accompanied by uptime %.

reisse 4mo ago

There is a branch of math dedicated to (among other things) truthfully estimating the waiting time, called queueing theory. I wonder why it wasn't mentioned in the article.

pfdietz 4mo ago

This feels similar to how nuclear power is perceived, contrasting deaths per TWh and the long tail effect of a rare but serious accident.

michaelt 4mo ago

> Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. There are also plenty of situations where a service can have a bimodal performance distribution and the impact of that can fall on certain users disproportionately. Imagine a retail website that serves images from a global CDN, with cache misses pulled from a server in the EU. Users who visit our homepage, or look at our bestselling products, get a cache hit from the CDN node close to them, in 50ms. But users who look at our long-tail products get a cache miss - and if they're not near Europe, they'll get a noticeable delay. Hence our mean image load time is 100ms - but a customer browsing an obscure product category for their location can experience markedly worse performance. If Alice is the only person in Costa Rica looking at ski equipment in June, she's going to get a lot of cache misses.

esperent 4mo ago
So what's the solution here? Keep the cache artificially warm for even obscure routes?
- steveBK123 4mo ago
  There may not be a solution in every case, but it's a reminder that dashboards & metrics are no replacement for actually talking to your users. Metrics are at best a proxy for user experience, don't let them be the tail that wags the dog. Like the story Bezos told of his execs claiming call wait times were under 1 minute, so he called the service line from the conference room on the spot and made everyone sit there for 10 minutes waiting to get thru..
  - esperent 4mo ago
    Ok, that's a fun anecdote and I agree it has real world value to think that way. But it doesn't answer my question - this thread is full of people pointing out problems but nobody offering solutions. So I was asking, specifically in the case of an app with a very long tail of cache misses, what's the solution? Do you have to keep potentially millions of routes artificially warm?

mdavid626 4mo ago

I don’t remember any service I used in the last couple of years, where I thought to myself: this service is really fast and responsive. Great experience. Quite the contrary: feels like everything got worse. Sometimes painfully slow, buggy and unreliable.

tadfisher 4mo ago
Try the McMaster-Carr website sometime. It's incredible. Expensive, but incredible.

x1798DE 4mo ago

My understanding is a lot of the probability puzzles in Allen Downey's Probably Overthinking It [1] also boil down to similar selection effects (the inspection paradox is definitely in there). There is a lot of cool stuff in that book (and his blog of the same name). 1. https://greenteapress.com/wp/probably-overthinking-it/ https://greenteapress.com/wp/probably-overthinking-it/