6 ms·

Tuning GCs like it was 2014 again. It's a little bit sad that they did not hire an expert.

vajrabum 5y ago
I don't understand what you're trying to say here. Is the methodology they're using outdated? Or perhaps their approach seems naive? Or are you saying something else?
- _old_dude_ 5y ago
  The whole section on how GCs work is far from accurate. It only describes how one GC, CMS, works or worked because it had been removed. The permanent generation does not exist anymore since Java 8, etc And i'm far from being an expert in GCs, but the basics are just wrong.
  - viscanti 5y ago
    Maybe they have to use some kind of legacy JVM library that doesn't work on Java 8? It looks like they're a GoLang shop, so I'm not sure why we'd assume their Java stuff is representative of anything other than what they have to do because some useful JVM library for mapping or something needs to be supported.
    - erik_seaberg 5y ago
      Yikes. Fortunately Java is a supported language at Uber, though some teams are more serious about it than others. Scala is also accepted because of Spark.
      - viscanti 5y ago
        I guess I misunderstood things then. It's difficult to reconcile why they'd be using a legacy version of Java that required these specific GC tunings if they weren't forced to use it because of libraries that couldn't be replicated or replaced in GoLang. Sounds like it's a much bigger mess there than what I had assumed (I guess I was giving them too much of the benefit of the doubt).
    - _old_dude_ 5y ago
      The problem is that Java 7 is 10 years old, it means that you are missing 10 years of optimization. For GCs that's a lot. GCs before ~2010 have been optimized for throughput more than for latency. Since then, you can choose. Since 10 years, G1 let you set your maximum pause time and you get a huge warning if it miss that target, ZGC or Shenandoah are from the beginning latency first, throughput second.
      - lamontcg 5y ago
        I last tuned Java 6 in about ~2009 and I was a bit shocked at how familiar most of this article sounded... Like your handle, too.
      - acamillo 5y ago
        Any advise on good / recent sources of information about the various GC algorithm in the recent JVM ?
    - vsto 5y ago
      It is Hadoop that requires (mostly) Java 8 and Java 11 (runtime only). https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Ja...
      - tomnipotent 5y ago
        That's for Hadoop 3 - Uber is on CDH 5 last I read, which has min support for Java 7 and it sounds like that's what they're using if the CMS GC was being used (unless it just happened to work better for them in Java 8+).
  - throwdbaaway 5y ago
    Well, at least they take the cake when it comes to running CMS with the largest heap size? It is also quite ironic that they paid for Zing C4, and then came up with "P90 average RPC queue time" as the metric, ignoring Gil Tene's wisdom that you should never average percentiles. Another problematic statement is this: > which in turn decreases request latency and increases throughput Gil Tene wants you to measure p99.99 or pmax, so that you will pay for Zing C4 to get lower tail latency, while sacrificing some throughput. If you end up getting both lower tail latency and more throughput, then something is already very wrong with the existing setup (i.e. the full GC).
StreamBright 5y ago
Using Hadoop like it was 2014 again too. I am surprised how much legacy inefficient crap is lingering around in companies like Uber.
- pavitheran 5y ago
  What should be used instead of Hadoop DFS in 2021?
  - throwaway7783 5y ago
    Oh, these cloud kids... Edit: instant downvotes. Okay, S3/GCS/Azure is the typical answer (egress costs be damned)
    - StreamBright 5y ago
      I guess the same was said about those non-mainframe kids 30 years ago. Tech gets cheaper and better. You can rent CPU time instead of buying a server. It is just that simple.
      - ppf 5y ago
        You mean, like a mainframe? Tech goes round in circles, more like.
      - notyourday 5y ago
        And overpay in orders of magnitude on a 5 year depreciation schedule. And the amount of fun you are going to have getting charged for the S3 API calls and pulling that data out of S3 for processing would make anyone reasonable CFO head spin 360 degrees every minute.
  - StreamBright 5y ago
    Depends on multiple factors: - S3 or compatible would be trivial choice for storage - if on-prem is a must there are multiple options, generally something with erasure codes (it is a game changer for storage) So far I have been using enterprise storage (that has some potential problems when mounted as nfs volumes), works for petabytes, already decouples storage from compute. More recently I was experimenting with MinIO. No conclusion so far. The problems are with Hadoop: - unfortunate design choices (namenode??) - extremely unfortunate implementation (I probably spent more time in the Hadoop codebase than any other, found many bugs, some I could fix, most I couldn’t) I think I have migrated away from Hadoop 10 PB worth of data infra in the last 5 years, mostly to AWS, some to Azure. Average cost saving is between 10-30% yoy. Some comments point out the network cost. The reality is most companies collect a giant amount of data (ingress) and publish dashboards (egress). It makes cloud pretty viable. S3 is beating the shit out of HDFS in reliability and cost, even though most Hadoop shops spread the fud that it is slow. Same way these companies used to spread the fud that snappy is best for data compression. As of 2021 even the latest adopters (banks and insurance companies) use cloud. Maybe extremely few dogmatic companies remain in the onprem crowd. Even those will eventually give up.

I don't understand what you're trying to say here. Is the methodology they're using outdated? Or perhaps their approach seems naive? Or are you saying something else?

_old_dude_ 5y ago
The whole section on how GCs work is far from accurate. It only describes how one GC, CMS, works or worked because it had been removed. The permanent generation does not exist anymore since Java 8, etc And i'm far from being an expert in GCs, but the basics are just wrong.
- viscanti 5y ago
  Maybe they have to use some kind of legacy JVM library that doesn't work on Java 8? It looks like they're a GoLang shop, so I'm not sure why we'd assume their Java stuff is representative of anything other than what they have to do because some useful JVM library for mapping or something needs to be supported.
  - erik_seaberg 5y ago
    Yikes. Fortunately Java is a supported language at Uber, though some teams are more serious about it than others. Scala is also accepted because of Spark.
    - viscanti 5y ago
      I guess I misunderstood things then. It's difficult to reconcile why they'd be using a legacy version of Java that required these specific GC tunings if they weren't forced to use it because of libraries that couldn't be replicated or replaced in GoLang. Sounds like it's a much bigger mess there than what I had assumed (I guess I was giving them too much of the benefit of the doubt).
  - _old_dude_ 5y ago
    The problem is that Java 7 is 10 years old, it means that you are missing 10 years of optimization. For GCs that's a lot. GCs before ~2010 have been optimized for throughput more than for latency. Since then, you can choose. Since 10 years, G1 let you set your maximum pause time and you get a huge warning if it miss that target, ZGC or Shenandoah are from the beginning latency first, throughput second.
    - lamontcg 5y ago
      I last tuned Java 6 in about ~2009 and I was a bit shocked at how familiar most of this article sounded... Like your handle, too.
    - acamillo 5y ago
      Any advise on good / recent sources of information about the various GC algorithm in the recent JVM ?
  - vsto 5y ago
    It is Hadoop that requires (mostly) Java 8 and Java 11 (runtime only). https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Ja...
    - tomnipotent 5y ago
      That's for Hadoop 3 - Uber is on CDH 5 last I read, which has min support for Java 7 and it sounds like that's what they're using if the CMS GC was being used (unless it just happened to work better for them in Java 8+).
- throwdbaaway 5y ago
  Well, at least they take the cake when it comes to running CMS with the largest heap size? It is also quite ironic that they paid for Zing C4, and then came up with "P90 average RPC queue time" as the metric, ignoring Gil Tene's wisdom that you should never average percentiles. Another problematic statement is this: > which in turn decreases request latency and increases throughput Gil Tene wants you to measure p99.99 or pmax, so that you will pay for Zing C4 to get lower tail latency, while sacrificing some throughput. If you end up getting both lower tail latency and more throughput, then something is already very wrong with the existing setup (i.e. the full GC).

StreamBright 5y ago

Using Hadoop like it was 2014 again too. I am surprised how much legacy inefficient crap is lingering around in companies like Uber.

pavitheran 5y ago
What should be used instead of Hadoop DFS in 2021?
- throwaway7783 5y ago
  Oh, these cloud kids... Edit: instant downvotes. Okay, S3/GCS/Azure is the typical answer (egress costs be damned)
  - StreamBright 5y ago
    I guess the same was said about those non-mainframe kids 30 years ago. Tech gets cheaper and better. You can rent CPU time instead of buying a server. It is just that simple.
    - ppf 5y ago
      You mean, like a mainframe? Tech goes round in circles, more like.
    - notyourday 5y ago
      And overpay in orders of magnitude on a 5 year depreciation schedule. And the amount of fun you are going to have getting charged for the S3 API calls and pulling that data out of S3 for processing would make anyone reasonable CFO head spin 360 degrees every minute.
- StreamBright 5y ago
  Depends on multiple factors: - S3 or compatible would be trivial choice for storage - if on-prem is a must there are multiple options, generally something with erasure codes (it is a game changer for storage) So far I have been using enterprise storage (that has some potential problems when mounted as nfs volumes), works for petabytes, already decouples storage from compute. More recently I was experimenting with MinIO. No conclusion so far. The problems are with Hadoop: - unfortunate design choices (namenode??) - extremely unfortunate implementation (I probably spent more time in the Hadoop codebase than any other, found many bugs, some I could fix, most I couldn’t) I think I have migrated away from Hadoop 10 PB worth of data infra in the last 5 years, mostly to AWS, some to Azure. Average cost saving is between 10-30% yoy. Some comments point out the network cost. The reality is most companies collect a giant amount of data (ingress) and publish dashboards (egress). It makes cloud pretty viable. S3 is beating the shit out of HDFS in reliability and cost, even though most Hadoop shops spread the fud that it is slow. Same way these companies used to spread the fud that snappy is best for data compression. As of 2021 even the latest adopters (banks and insurance companies) use cloud. Maybe extremely few dogmatic companies remain in the onprem crowd. Even those will eventually give up.