8 ms·

> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 mach

> High performance in throughput and latency. In our main data center serving non-China traffic alone, this service handled a peak load of 170k QPS with 40 machines running at 35% CPU usage on NYE 2015. The response time was < 5 ms at 95th percentile, and < 50 ms at the 99th percentile. > Geofence lookups are required on every request from Uber’s mobile apps and must quickly (99th percentile < 100 milliseconds) answer For a 100ms total budget, a ~50ms 99th percentile for a single microservice doesn't sound like something to boast about.

bsaul 7y ago
I don’t read this the way you do. I read it as their service had to answer under 100ms and they made it run under 50
Rapzid 7y ago
With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.
- sokoloff 7y ago
  The latency to human user is what matters. If I get an answer in under 100ms as a user 99% of the time, I don’t care whether 100 or 100K times per day someone else hits that 1% worst case.
  - ses1984 7y ago
    It has nothing to do with your experience being 99% good enough and more to do with the size of the revenue opportunity that could be lost messing up 1% of requests.
  - kevan 7y ago
    As an individual you don't care as long as you aren't in that 1%, but as a customer-obsessed service owner I absolutely care if a bad experience is happening 100k times per day. If you're operating at internet scale you need to look at both percentiles and absolute numbers to assess customer impact. Also keep in mind how percentiles compound when you have more than one service involved in serving a customer request. For example, let's say it takes 5 internal requests to serve an external customer request and each of those services measures latency SLAs at the 99th percentile. The customer request may only finish inside the SLA 95% of the time (99%^5)
  - scottlamb 7y ago
    > The latency to human user is what matters. Agreed, which is why the most surprising part of this article for me was this phrase: "In our main data center serving non-China traffic". The transit latency to the one main datacenter in the (non-China) world is a lot more significant than the server time here. (Over 50 ms just to cross the US; compare with their server-side latency numbers of 95%ile < 5 ms, 99%ile < 50 ms.) If they're serious about latency, their deployment is holding them back much more than their choice of programming language or algorithm. Or maybe the client is not the user's phone, but some other service running within the same datacenter?
- sethammons 7y ago
  I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.
  - lclarkmichalek 7y ago
    Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild. Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.
    - sethammons 7y ago
      And that would be a good signal and worth investigating. Why timeouts? I run a few services doing billions of daily queries. I watch max in addition to percentiles.
    - hobofan 7y ago
      > and almost certainly had timeouts, the "max" would also likely be at the timeout You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.
      - spookthesunset 7y ago
        Sure but now your "100th percentile" would be nothing but the timeout. Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else. .... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.
foobarian 7y ago
If that's 40 bare metal machines... assuming 60ish HT cores, and 35% CPU... that's not that earth shattering. We do that in Java :)
- matt2000 7y ago
  My sense is that most people moving to Go are coming from Node and that explains their excitement about the performance. Not many upsides when you’re already on Java.
  - dashwav 7y ago
    I mean there is a definite performance gain from Java to Go, although I would agree most people probably wouldn't hit it - but if its just as easy to write, might as well use the better language. On top of that Go makes concurrency so much easier to factor in and program with, which in the backend server space is a very important asset over Java. As an interesting addendum, I found this to be a really interesting resource when comparing languages at a very base level, as it is very well written and you can actually look at the source code/ thesis papers for all of the implementations: https://github.com/ixy-languages/ixy-languages https://github.com/ixy-languages/ixy-languages
    - MrBuddyCasino 7y ago
      > I mean there is a definite performance gain from Java to Go I don't think thats true.
      - erik_seaberg 7y ago
        The GC isn't very good yet, but brute-force sequential code can keep more state on the stack where it doesn't matter. (Just please codegen that instead of expecting everyone to read it.)
        marsdepinski 7y ago
        Go's GC is better than most of the Java GCs except the new ZGC and Azul's proprietary one. JIT makes some Java code faster. Ultimately, it's more about programmer expertise in a language rather than the actual language.
sethammons 7y ago
Per machine, that is 4,250k rps. For any http service we write in Go, that lines up with our general sizing estimates before we start work on a generic new service. After we start to hit the service with load, we start profiling and go from there. Sometimes it gets faster and sometimes it gets slower, but it is usually in the right ball park.

bsaul 7y ago

I don’t read this the way you do. I read it as their service had to answer under 100ms and they made it run under 50

Rapzid 7y ago

With that kinda volume the 99th is not interesting at all. Should be looking at the 99.99th.

sokoloff 7y ago
The latency to human user is what matters. If I get an answer in under 100ms as a user 99% of the time, I don’t care whether 100 or 100K times per day someone else hits that 1% worst case.
- ses1984 7y ago
  It has nothing to do with your experience being 99% good enough and more to do with the size of the revenue opportunity that could be lost messing up 1% of requests.
- kevan 7y ago
  As an individual you don't care as long as you aren't in that 1%, but as a customer-obsessed service owner I absolutely care if a bad experience is happening 100k times per day. If you're operating at internet scale you need to look at both percentiles and absolute numbers to assess customer impact. Also keep in mind how percentiles compound when you have more than one service involved in serving a customer request. For example, let's say it takes 5 internal requests to serve an external customer request and each of those services measures latency SLAs at the 99th percentile. The customer request may only finish inside the SLA 95% of the time (99%^5)
- scottlamb 7y ago
  > The latency to human user is what matters. Agreed, which is why the most surprising part of this article for me was this phrase: "In our main data center serving non-China traffic". The transit latency to the one main datacenter in the (non-China) world is a lot more significant than the server time here. (Over 50 ms just to cross the US; compare with their server-side latency numbers of 95%ile < 5 ms, 99%ile < 50 ms.) If they're serious about latency, their deployment is holding them back much more than their choice of programming language or algorithm. Or maybe the client is not the user's phone, but some other service running within the same datacenter?
sethammons 7y ago
I would argue they should be considering max. Percentiles are good for understanding macro trends. But if every request is sacred, max needs to be monitored. Even at four nines, there are a couple dozen requests per second that are greater than what you are observing. How much greater? Who knows, if you are not observing max. Could be minutes in latency.
- lclarkmichalek 7y ago
  Max is pretty horrible, as any comparison requires identical cohort sizes. P99.99 is much better if you want to go wild. Given that the service probably did not have a 100% success rate, and almost certainly had timeouts, the "max" would also likely be at the timeout.
  - sethammons 7y ago
    And that would be a good signal and worth investigating. Why timeouts? I run a few services doing billions of daily queries. I watch max in addition to percentiles.
  - hobofan 7y ago
    > and almost certainly had timeouts, the "max" would also likely be at the timeout You filter out the 504s, just the same way you would analyze it on a per-route basis for those metrics.
    - spookthesunset 7y ago
      Sure but now your "100th percentile" would be nothing but the timeout. Quantifying things like service latency isn't a one-size fits all thing. Every service has its nuances and use cases that make it more meaningful to measure 99%, 99.9%, 99.99% or something else. .... just don't measure average like I've seen naive junior devs do. Average latency is the worst of all metrics to use as it will include all the outliers at the very tippy top of the spectrum and basically render the metric meaningless.

foobarian 7y ago

If that's 40 bare metal machines... assuming 60ish HT cores, and 35% CPU... that's not that earth shattering. We do that in Java :)

matt2000 7y ago
My sense is that most people moving to Go are coming from Node and that explains their excitement about the performance. Not many upsides when you’re already on Java.
- dashwav 7y ago
  I mean there is a definite performance gain from Java to Go, although I would agree most people probably wouldn't hit it - but if its just as easy to write, might as well use the better language. On top of that Go makes concurrency so much easier to factor in and program with, which in the backend server space is a very important asset over Java. As an interesting addendum, I found this to be a really interesting resource when comparing languages at a very base level, as it is very well written and you can actually look at the source code/ thesis papers for all of the implementations: https://github.com/ixy-languages/ixy-languages https://github.com/ixy-languages/ixy-languages
  - MrBuddyCasino 7y ago
    > I mean there is a definite performance gain from Java to Go I don't think thats true.
    - erik_seaberg 7y ago
      The GC isn't very good yet, but brute-force sequential code can keep more state on the stack where it doesn't matter. (Just please codegen that instead of expecting everyone to read it.)
      - marsdepinski 7y ago
        Go's GC is better than most of the Java GCs except the new ZGC and Azul's proprietary one. JIT makes some Java code faster. Ultimately, it's more about programmer expertise in a language rather than the actual language.

sethammons 7y ago

Per machine, that is 4,250k rps. For any http service we write in Go, that lines up with our general sizing estimates before we start work on a generic new service. After we start to hit the service with load, we start profiling and go from there. Sometimes it gets faster and sometimes it gets slower, but it is usually in the right ball park.