10 ms·

Uber Engineering Tricks of the Trade: Tuning JVM Memory for Large-Scale Services

133 points by commandlinefan 5y ago

asguy 5y ago
“Our Apache Hadoop-based data platform ingests hundreds of petabytes of analytical data with minimum latency and stores it in a data lake built on top of the Hadoop Distributed File System (HDFS).” “Did you just tell me to go fuck myself, Bob?” “I believe I did.”
- pc86 5y ago
  I knew I recognized this from somewhere. http://howfuckedismydatabase.com/nosql/ http://howfuckedismydatabase.com/nosql/ https://news.ycombinator.com/item?id=1636113 https://news.ycombinator.com/item?id=1636113
- bpodgursky 5y ago
  This isn't especially strange or complicated. I guess you're just trying to be witty? But it comes across as pretty ignorant. Yeah if Uber was born today and didn't want to use any on-prem resources they'd use S3, but HDFS is the best alternative to object storage out there, and there's a huge ecosystem of tools around it. If you're running your own datacenter, there's not any serious alternative.
suyash 5y ago
@Uber - have you considered GraalVM ?
- notdang 5y ago
  Will it solve the problem? As I understand the same GC tuning will need to take place.
  - throwaway7783 5y ago
    GP might've meant AOT
    - cellularmitosis 5y ago
      grall removes the need for JIT but does it also remove the need for GC?
      - suyash 5y ago
        almost, it has something called native images
  - chrisseaton 5y ago
    Graal has better escape analysis and so produces less garbage, which is the very best kind of garbage collection! That might be what they meant.
_old_dude_ 5y ago
Tuning GCs like it was 2014 again. It's a little bit sad that they did not hire an expert.
- vajrabum 5y ago
  I don't understand what you're trying to say here. Is the methodology they're using outdated? Or perhaps their approach seems naive? Or are you saying something else?
  - _old_dude_ 5y ago
    The whole section on how GCs work is far from accurate. It only describes how one GC, CMS, works or worked because it had been removed. The permanent generation does not exist anymore since Java 8, etc And i'm far from being an expert in GCs, but the basics are just wrong.
    - viscanti 5y ago
      Maybe they have to use some kind of legacy JVM library that doesn't work on Java 8? It looks like they're a GoLang shop, so I'm not sure why we'd assume their Java stuff is representative of anything other than what they have to do because some useful JVM library for mapping or something needs to be supported.
      - erik_seaberg 5y ago
        Yikes. Fortunately Java is a supported language at Uber, though some teams are more serious about it than others. Scala is also accepted because of Spark.
        viscanti 5y ago
        I guess I misunderstood things then. It's difficult to reconcile why they'd be using a legacy version of Java that required these specific GC tunings if they weren't forced to use it because of libraries that couldn't be replicated or replaced in GoLang. Sounds like it's a much bigger mess there than what I had assumed (I guess I was giving them too much of the benefit of the doubt).
SilurianWenlock 5y ago
I thought the concurrent GC algorithms (zing/azul, g1?) avoided these stop the world events by using concurrency?
- rchiang 5y ago
  This presentation from 2019 gives an idea about some of the GC avg/max runtimes using the NameNode: https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-dynamometer-and-a-case-study-in-namenode-gc https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-dy...
- dharmab 5y ago
  As discussed in the link, these algorithms reduce but do not entirely eliminate STW.
  - _old_dude_ 5y ago
    Recent GC algorithms slow down the application, either by not scheduling coroutines that allocate too much memory or by forcing them to do the marking or the evacuation (helping the GC) when running (or both). With that, STW pause time does not depend on the heap size or on the root set size (the stack size). In practice, it means pause < 1ms, at that point the OS becomes the bottleneck, not the GC. So latency is good but throughput can be reduced by 30%.
crummybowley 5y ago
Just use go.
- domano 5y ago
  I love go and it is my main language, but Uber uses go and clearly has some need for Hadoop. Also the go GC is behind the JVM GCs if you want to really tune it IMHO

“Our Apache Hadoop-based data platform ingests hundreds of petabytes of analytical data with minimum latency and stores it in a data lake built on top of the Hadoop Distributed File System (HDFS).” “Did you just tell me to go fuck myself, Bob?” “I believe I did.”

pc86 5y ago
I knew I recognized this from somewhere. http://howfuckedismydatabase.com/nosql/ http://howfuckedismydatabase.com/nosql/ https://news.ycombinator.com/item?id=1636113 https://news.ycombinator.com/item?id=1636113
bpodgursky 5y ago
This isn't especially strange or complicated. I guess you're just trying to be witty? But it comes across as pretty ignorant. Yeah if Uber was born today and didn't want to use any on-prem resources they'd use S3, but HDFS is the best alternative to object storage out there, and there's a huge ecosystem of tools around it. If you're running your own datacenter, there's not any serious alternative.

suyash 5y ago

@Uber - have you considered GraalVM ?

notdang 5y ago
Will it solve the problem? As I understand the same GC tuning will need to take place.
- throwaway7783 5y ago
  GP might've meant AOT
  - cellularmitosis 5y ago
    grall removes the need for JIT but does it also remove the need for GC?
    - suyash 5y ago
      almost, it has something called native images
- chrisseaton 5y ago
  Graal has better escape analysis and so produces less garbage, which is the very best kind of garbage collection! That might be what they meant.

_old_dude_ 5y ago

Tuning GCs like it was 2014 again. It's a little bit sad that they did not hire an expert.

vajrabum 5y ago
I don't understand what you're trying to say here. Is the methodology they're using outdated? Or perhaps their approach seems naive? Or are you saying something else?
- _old_dude_ 5y ago
  The whole section on how GCs work is far from accurate. It only describes how one GC, CMS, works or worked because it had been removed. The permanent generation does not exist anymore since Java 8, etc And i'm far from being an expert in GCs, but the basics are just wrong.
  - viscanti 5y ago
    Maybe they have to use some kind of legacy JVM library that doesn't work on Java 8? It looks like they're a GoLang shop, so I'm not sure why we'd assume their Java stuff is representative of anything other than what they have to do because some useful JVM library for mapping or something needs to be supported.
    - erik_seaberg 5y ago
      Yikes. Fortunately Java is a supported language at Uber, though some teams are more serious about it than others. Scala is also accepted because of Spark.
      - viscanti 5y ago
        I guess I misunderstood things then. It's difficult to reconcile why they'd be using a legacy version of Java that required these specific GC tunings if they weren't forced to use it because of libraries that couldn't be replicated or replaced in GoLang. Sounds like it's a much bigger mess there than what I had assumed (I guess I was giving them too much of the benefit of the doubt).

SilurianWenlock 5y ago

I thought the concurrent GC algorithms (zing/azul, g1?) avoided these stop the world events by using concurrency?

rchiang 5y ago
This presentation from 2019 gives an idea about some of the GC avg/max runtimes using the NameNode: https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-dynamometer-and-a-case-study-in-namenode-gc https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-dy...
dharmab 5y ago
As discussed in the link, these algorithms reduce but do not entirely eliminate STW.
- _old_dude_ 5y ago
  Recent GC algorithms slow down the application, either by not scheduling coroutines that allocate too much memory or by forcing them to do the marking or the evacuation (helping the GC) when running (or both). With that, STW pause time does not depend on the heap size or on the root set size (the stack size). In practice, it means pause < 1ms, at that point the OS becomes the bottleneck, not the GC. So latency is good but throughput can be reduced by 30%.

crummybowley 5y ago

Just use go.

domano 5y ago
I love go and it is my main language, but Uber uses go and clearly has some need for Hadoop. Also the go GC is behind the JVM GCs if you want to really tune it IMHO