17 ms·

Performance Impact of Removing an Out-of-Band Garbage Collector

94 points by jbredeche 8y ago

titzer 8y ago
TLDR: they saved 400 to 1000 cores by switching off the switching off of the GC during requests. The fact that they are running Ruby and are spending 1000 cores on GC is o_O.
- k__ 8y ago
  Isn't Ruby known for these kind of issues?
  - Karunamon 8y ago
    Sadly yes. It’s an absolute joy to code in, but it isn’t even kind of light.
  - nwmcsween 8y ago
    Ruby as a language no, most of the implementations yes.
  - vidarh 8y ago
    In this case it was Github's hack to work around no-longer-existing issues with Ruby's GC that was the issue.... Note that the speedup came from removing their hack in favour of relying on the default behaviour of Ruby 2.4
    - jashmatthews 8y ago
      It was actually Ruby 2.2 from 2014 that made this redundant. 2.2 introduced incremental marking, removing the last long GC phase in MRI Ruby.
- twic 8y ago
  Moreover, "removing OOBGC reduced average response times by about 25%". CPU utilisation is a pretty terrible metric on its own; once you've paid for a CPU, you might as well run it at 100%!
  - majewsky 8y ago
    Not really. Running CPUs at 100% load means you have no breathing room for high-load situations. It also makes them hotter, which may lead to worse performance because of throttling.
    - puzzle 8y ago
      Not 100%, but running them at least 50% is not a bad idea. In these days of containers and QoS scheduling, it's easier to make good use of slack resources. Letting memory sit unused is an even bigger crime! Someone told me that the team at his previous job with a "large cloud provider" based in Seattle was told not to go above 30% CPU usage, at which point they'd buy more hardware. I'm sure not all teams there do things that way, but, coming from Google, 30% (95th percentile over a week) is very low.
      - merb 8y ago
        well if you use k8s you need to have some spare capacity for updates, etc. consider a deployment (deployment app) update it will first create a new container and than kill the old one, which means depending how much memory you request at container creation is the least minimum you would have on spare. if you schedule java apps with at least 1 Gi heap you would at least need 1Gi spare capacity and that is just with one pod/container (it will be worse if you need to do blue/green deploys, since you need the same capacity as your live cluster)
        puzzle 8y ago
        If you're running Kubernetes, you probably have multiple services running (or it's not really worth the complexity). Then whatever slack you have in the cluster can be amortized over all your services, if they all share the same resources (quota). It's also a good idea not to update too many deployments all at the same time. Even then, deployment updates don't necessarily need to surge above their replica count. You can also configure them to terminate X replicas at a time before bringing up new ones. At Google, all teams have Borg quotas, so it's not unusual to max those up by running as many replicas as possible. During updates, Borg does not allow an user to temporarily oversubscribe their quota (unless you're changing replica count and replica footprint at the same time, but that's another fun story), so it will always take down Y tasks first.
- deleted 8y ago
  [deleted]
- vidarh 8y ago
  Those 1000 cores seems to have been equivalent to 10% of their load. The 10% number isn't nearly as shocking.
- stmw 8y ago
  Impressive and interesting, but it is really a great example of how in larger production systems, Garbage Collection turns into "manual memory management" akin to that required to manage malloc() and free() correctly -- little performance traps, ever-changing library behaviors, tuning parameters, etc. (This is not specific to Ruby, the JVM has the same kind of "here is a whole blog post about person-weeks or person-months that went into getting the GC to behave better).
  - tetha 8y ago
    The thing is, the more I work with network based reactive systems, the more a GC seems like an inefficient hack in an application, arising from the fact that we had nothing better at the time it was invented and grew big. I mean a reactive application needs three things: - a fast network connection to persistence. If you can't talk quickly to your database, latency eats your response time no matter what you do. - Maybe some LRU caches across requests, but caches inside an application tend to become a maintenance nightmare quickly. Stales reads are fun, and stalled cache writes are too. - And beyond that, it should be possible to handle most request data by stack allocation/deallocation with references to a cache version if you have a cache with little overhead, depending on your security requirements. I understand the ease of use and simplicity of a GC'd language, especially given the time GC'd languages came around big time. But I've always been wondering if you really need a GC if you think properly about request and object lifetimes.
    - Gibbon1 8y ago
      > handle most request data by stack allocation/deallocation I just write crappy code for microcontrollers. Over 30 years I've seen the amount of ram available for stack grow. From maybe 20-30 bytes to a couple of k. That said it appears to me that stack allocation is severely under utilized for historical reasons. The idea that stack space is a precious resource. Which doesn't fly on machines with 10's of gigabytes of memory. Think programs that put vast quantities of ephemeral objects and short strings on the heap. Using call tree analysis would allow you to put a lot of that on the stack resulting is much better performance and bounded latency.
- Sean1708 8y ago
  > TLDR: they saved 400 to 1000 cores by switching off the switching off of the GC during requests. Isn't that exactly the opposite of what the article is saying? > An OOBGC is not really a Garbage Collector, but more of a technique to use when deciding when to collect garbage in your program. Instead of allowing the GC to run normally, the GC is stopped before processing a web request, then restarted after the response has been sent to the client. Meaning that garbage collection occurs “out of band” of request and response processing. > This graph shows the difference in core utilization before and after OOBGC removal. In other words “number of cores used yesterday” minus “number of cores used today” ... We saw a savings of between 400 and around 1000 cores depending on usage at that point in the day. So it sounds to me that by switching the GC on during requests they saved that many cores.
twic 8y ago
I find this an interesting contrast to Go's move towards a "request-oriented collector": https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ftk-tcRRJcDFANj2VwCB0/edit https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ft... Perhaps a key difference is this point from the Github document: "Since the OOBGC runs the GC after the response is finished, it can cause the web worker to take longer in order to be ready to process the next incoming request. This means that clients can suffer from latency due to queuing wait times." It seems like a little bit more intelligence in the load balancer that sits in front of the application servers would go a long way here. Could it only route requests to servers which have finished garbage collecting and are ready to serve again?
- vidarh 8y ago
  One option is a reversal: Make the worker request a request. Slow, unavailable or failed workers will in effect automatically back off.
  - jashmatthews 8y ago
    Phusion Passenger has some intelligent routing to balance requests so you don't end up with one process idle and one process serving two requests using 2 threads.
    - vidarh 8y ago
      Lots of systems do, but that's requiring lots of logic in the load balancer. Reversal makes this an emergent property of the system.
  - tybit 8y ago
    I’ve been wondering whether this approach would be viable for a while now. Are there any systems that do this? I know solo’s Gloo reverses the flow using NATS but that’s all I’ve seen use this approach.
    - vidarh 8y ago
nneonneo 8y ago
Interesting work, thanks for sharing. Nitpick: > The blue line is CPU utilization for the day the patch went out. You can see a great drop around 15:20. I see a dip at 18:20, not at 15:20 - if there is a dip at 15:20, maybe it would be worth highlighting it.
- irishsultan 8y ago
  I'd assume that it's a timezone issue. The author probably didn't check to see if the labels on the x-axis matched his own timezone and just pointed out where he knew the drop happened, instead of where it actually happened (on his graph).
bluesnowmonkey 8y ago
Of course removing it reduced your CPU consumption. What did it do to p99 request latency? That was the trade-off being made by OOBGC. Does Github use SOA? The more backend services that are involved in handling a user facing request, the more the latency of frontend requests will be dominated by the tail latency of backend services. So in distributed systems it makes much more sense to focus on those tail latencies. Who cares about average latency?
- hinkley 8y ago
  > Who cares about average latency? People who are bad at statistics. So most of us programmers. Seriously though, I don’t think many of us get around to acknowledging that microservices are distributed computing and then consider all of the limitations that go with it. Latency is dominated by the slowest thing you can’t start earlier. So in real distributed computing you see a degree of duplicate effort in order to improve latency. It reduces throughout but increases the utility of the system. When I took CS in college there was a proper class on distributed computing. When I got out I discovered not everybody had one of these (and I think in my program it was an elective. IIRC I chose it instead of kernel design). I had hoped that situation improved but I got tired of asking people disappointing questions about their education so I stopped asking.
  - srean 8y ago
    > People who are bad at statistics. Heh! sadly very true. Many of those who do care about the p95s of an internal service often do it with a sense of chivalry that they have that 5% of the users back. The end user latency percentiles would be quite different from the service side latency, more so if its composed by a "Last_of" or a Max operator. At times I feel tempted to write a library that offers primitives to glue microservice calls such as fan_out, retry, timeout etc but one that can reason about the percentiles of the output from the percentiles of the input. BTW are statistical properties of latencies covered in distributed system courses ? I thought they were mostly about different kinds of guarantees, such as consensus, atleast-once, atmost-once etc etc.
    - 8y ago
chris_wot 8y ago
That's a remarkable author profile on that article.
ksec 8y ago
I am actually surprised GitHub still has OOBGC, which I thought was made redundant in 2014/2015 when Ruby 2.2 had incremental marking, and further improvement in GC 2.3, 2.4 and now 2.5. In 2.6 they are introducing something similar called Sleepy GC https://bugs.ruby-lang.org/issues/14723 https://bugs.ruby-lang.org/issues/14723 There are even better memory default for 2.6 using less memory. ( Or better switch to Jemalloc ) https://bugs.ruby-lang.org/issues/14718 https://bugs.ruby-lang.org/issues/14718 I cant find the thread, but there also a test showing how on Alpine Linux memory and performance were much better for Ruby Apps. Meanwhile we are waiting for Truffle Ruby as a true drop in replacement, that should take lots of Ruby Pain point away. ( Native Image deployment, Better Memory handling and GC, Much faster JIT )

94 points by jbredeche 8y ago

TLDR: they saved 400 to 1000 cores by switching off the switching off of the GC during requests. The fact that they are running Ruby and are spending 1000 cores on GC is o_O.

k__ 8y ago
Isn't Ruby known for these kind of issues?
- Karunamon 8y ago
  Sadly yes. It’s an absolute joy to code in, but it isn’t even kind of light.
- nwmcsween 8y ago
  Ruby as a language no, most of the implementations yes.
- vidarh 8y ago
  In this case it was Github's hack to work around no-longer-existing issues with Ruby's GC that was the issue.... Note that the speedup came from removing their hack in favour of relying on the default behaviour of Ruby 2.4
  - jashmatthews 8y ago
    It was actually Ruby 2.2 from 2014 that made this redundant. 2.2 introduced incremental marking, removing the last long GC phase in MRI Ruby.
twic 8y ago
Moreover, "removing OOBGC reduced average response times by about 25%". CPU utilisation is a pretty terrible metric on its own; once you've paid for a CPU, you might as well run it at 100%!
- majewsky 8y ago
  Not really. Running CPUs at 100% load means you have no breathing room for high-load situations. It also makes them hotter, which may lead to worse performance because of throttling.
  - puzzle 8y ago
    Not 100%, but running them at least 50% is not a bad idea. In these days of containers and QoS scheduling, it's easier to make good use of slack resources. Letting memory sit unused is an even bigger crime! Someone told me that the team at his previous job with a "large cloud provider" based in Seattle was told not to go above 30% CPU usage, at which point they'd buy more hardware. I'm sure not all teams there do things that way, but, coming from Google, 30% (95th percentile over a week) is very low.
    - merb 8y ago
      well if you use k8s you need to have some spare capacity for updates, etc. consider a deployment (deployment app) update it will first create a new container and than kill the old one, which means depending how much memory you request at container creation is the least minimum you would have on spare. if you schedule java apps with at least 1 Gi heap you would at least need 1Gi spare capacity and that is just with one pod/container (it will be worse if you need to do blue/green deploys, since you need the same capacity as your live cluster)
      - puzzle 8y ago
        If you're running Kubernetes, you probably have multiple services running (or it's not really worth the complexity). Then whatever slack you have in the cluster can be amortized over all your services, if they all share the same resources (quota). It's also a good idea not to update too many deployments all at the same time. Even then, deployment updates don't necessarily need to surge above their replica count. You can also configure them to terminate X replicas at a time before bringing up new ones. At Google, all teams have Borg quotas, so it's not unusual to max those up by running as many replicas as possible. During updates, Borg does not allow an user to temporarily oversubscribe their quota (unless you're changing replica count and replica footprint at the same time, but that's another fun story), so it will always take down Y tasks first.
deleted 8y ago
[deleted]
vidarh 8y ago
Those 1000 cores seems to have been equivalent to 10% of their load. The 10% number isn't nearly as shocking.
stmw 8y ago
Impressive and interesting, but it is really a great example of how in larger production systems, Garbage Collection turns into "manual memory management" akin to that required to manage malloc() and free() correctly -- little performance traps, ever-changing library behaviors, tuning parameters, etc. (This is not specific to Ruby, the JVM has the same kind of "here is a whole blog post about person-weeks or person-months that went into getting the GC to behave better).
- tetha 8y ago
  The thing is, the more I work with network based reactive systems, the more a GC seems like an inefficient hack in an application, arising from the fact that we had nothing better at the time it was invented and grew big. I mean a reactive application needs three things: - a fast network connection to persistence. If you can't talk quickly to your database, latency eats your response time no matter what you do. - Maybe some LRU caches across requests, but caches inside an application tend to become a maintenance nightmare quickly. Stales reads are fun, and stalled cache writes are too. - And beyond that, it should be possible to handle most request data by stack allocation/deallocation with references to a cache version if you have a cache with little overhead, depending on your security requirements. I understand the ease of use and simplicity of a GC'd language, especially given the time GC'd languages came around big time. But I've always been wondering if you really need a GC if you think properly about request and object lifetimes.
  - Gibbon1 8y ago
    > handle most request data by stack allocation/deallocation I just write crappy code for microcontrollers. Over 30 years I've seen the amount of ram available for stack grow. From maybe 20-30 bytes to a couple of k. That said it appears to me that stack allocation is severely under utilized for historical reasons. The idea that stack space is a precious resource. Which doesn't fly on machines with 10's of gigabytes of memory. Think programs that put vast quantities of ephemeral objects and short strings on the heap. Using call tree analysis would allow you to put a lot of that on the stack resulting is much better performance and bounded latency.
Sean1708 8y ago
> TLDR: they saved 400 to 1000 cores by switching off the switching off of the GC during requests. Isn't that exactly the opposite of what the article is saying? > An OOBGC is not really a Garbage Collector, but more of a technique to use when deciding when to collect garbage in your program. Instead of allowing the GC to run normally, the GC is stopped before processing a web request, then restarted after the response has been sent to the client. Meaning that garbage collection occurs “out of band” of request and response processing. > This graph shows the difference in core utilization before and after OOBGC removal. In other words “number of cores used yesterday” minus “number of cores used today” ... We saw a savings of between 400 and around 1000 cores depending on usage at that point in the day. So it sounds to me that by switching the GC on during requests they saved that many cores.

twic 8y ago

I find this an interesting contrast to Go's move towards a "request-oriented collector": https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ftk-tcRRJcDFANj2VwCB0/edit https://docs.google.com/document/d/1gCsFxXamW8RRvOe5hECz98Ft... Perhaps a key difference is this point from the Github document: "Since the OOBGC runs the GC after the response is finished, it can cause the web worker to take longer in order to be ready to process the next incoming request. This means that clients can suffer from latency due to queuing wait times." It seems like a little bit more intelligence in the load balancer that sits in front of the application servers would go a long way here. Could it only route requests to servers which have finished garbage collecting and are ready to serve again?

vidarh 8y ago
One option is a reversal: Make the worker request a request. Slow, unavailable or failed workers will in effect automatically back off.
- jashmatthews 8y ago
  Phusion Passenger has some intelligent routing to balance requests so you don't end up with one process idle and one process serving two requests using 2 threads.
  - vidarh 8y ago
    Lots of systems do, but that's requiring lots of logic in the load balancer. Reversal makes this an emergent property of the system.
- tybit 8y ago
  I’ve been wondering whether this approach would be viable for a while now. Are there any systems that do this? I know solo’s Gloo reverses the flow using NATS but that’s all I’ve seen use this approach.
  - vidarh 8y ago

nneonneo 8y ago

Interesting work, thanks for sharing. Nitpick: > The blue line is CPU utilization for the day the patch went out. You can see a great drop around 15:20. I see a dip at 18:20, not at 15:20 - if there is a dip at 15:20, maybe it would be worth highlighting it.

irishsultan 8y ago
I'd assume that it's a timezone issue. The author probably didn't check to see if the labels on the x-axis matched his own timezone and just pointed out where he knew the drop happened, instead of where it actually happened (on his graph).

bluesnowmonkey 8y ago

Of course removing it reduced your CPU consumption. What did it do to p99 request latency? That was the trade-off being made by OOBGC. Does Github use SOA? The more backend services that are involved in handling a user facing request, the more the latency of frontend requests will be dominated by the tail latency of backend services. So in distributed systems it makes much more sense to focus on those tail latencies. Who cares about average latency?

hinkley 8y ago
> Who cares about average latency? People who are bad at statistics. So most of us programmers. Seriously though, I don’t think many of us get around to acknowledging that microservices are distributed computing and then consider all of the limitations that go with it. Latency is dominated by the slowest thing you can’t start earlier. So in real distributed computing you see a degree of duplicate effort in order to improve latency. It reduces throughout but increases the utility of the system. When I took CS in college there was a proper class on distributed computing. When I got out I discovered not everybody had one of these (and I think in my program it was an elective. IIRC I chose it instead of kernel design). I had hoped that situation improved but I got tired of asking people disappointing questions about their education so I stopped asking.
- srean 8y ago
  > People who are bad at statistics. Heh! sadly very true. Many of those who do care about the p95s of an internal service often do it with a sense of chivalry that they have that 5% of the users back. The end user latency percentiles would be quite different from the service side latency, more so if its composed by a "Last_of" or a Max operator. At times I feel tempted to write a library that offers primitives to glue microservice calls such as fan_out, retry, timeout etc but one that can reason about the percentiles of the output from the percentiles of the input. BTW are statistical properties of latencies covered in distributed system courses ? I thought they were mostly about different kinds of guarantees, such as consensus, atleast-once, atmost-once etc etc.
  - 8y ago

chris_wot 8y ago

That's a remarkable author profile on that article.

ksec 8y ago

I am actually surprised GitHub still has OOBGC, which I thought was made redundant in 2014/2015 when Ruby 2.2 had incremental marking, and further improvement in GC 2.3, 2.4 and now 2.5. In 2.6 they are introducing something similar called Sleepy GC https://bugs.ruby-lang.org/issues/14723 https://bugs.ruby-lang.org/issues/14723 There are even better memory default for 2.6 using less memory. ( Or better switch to Jemalloc ) https://bugs.ruby-lang.org/issues/14718 https://bugs.ruby-lang.org/issues/14718 I cant find the thread, but there also a test showing how on Alpine Linux memory and performance were much better for Ruby Apps. Meanwhile we are waiting for Truffle Ruby as a true drop in replacement, that should take lots of Ruby Pain point away. ( Native Image deployment, Better Memory handling and GC, Much faster JIT )