10 ms·

I'm starting to see why systems programming people (C/zig/rust) sneer at java/node.js/dot net. The jvm and v8 is just another layer of abstraction that gets in

I'm starting to see why systems programming people (C/zig/rust) sneer at java/node.js/dot net. The jvm and v8 is just another layer of abstraction that gets in the way when performances problems like this arise at scale. The tools used in this article require an understanding of what the hardware is actually doing.

delibes 4y ago
I've worked in a company that wrote Java code that needed to avoid false cacheing. It was fairly understandable and the tooling made it quite testable for performance regressions. However, the issue in the Netflix blogpost is in the JVM C++ code. I think it's entirely possible to encounter the problem in any language if you're writing performance-critical code.
- saagarjha 4y ago
  It’s often more common! Because the JVM will typically give you ok layouts “for free” and your hand-written native code might be too naive to do the same.
  - wahern 4y ago
    JVM will typically give you ok layouts presumably because almost everything is an object, and when accounting for various headers and instrumentation two distinct objects are more likely to be >= 64 bytes apart even when allocated in succession. What I find a little odd is why those variables were only on different cache lines 1/8th (12.5%) of the time. What linker behavior or feature would result in randomly shifting those objects while preserving their adjacency? ASLR is the first thing that comes to mind, randomizing the base address of their shared region. But heap allocators on 64-bit architectures usually use 16-byte alignment rather than the random 8-byte alignment that would account for this behavior. Similarly, mmap normally would be page-aligned, even when randomized; or certainly at least 16-byte aligned?
    - FeepingCreature 4y ago
      Aiui, it's one object that happens to sometimes be allocated at an offset where two fields in it lie across a cache line boundary. ASLR only affects static offsets, not heap memory. edit: Pointers will be 8-aligned. Random 16-byte allocation if one pointer is at 8-offset and the next is at 0-offset will sometimes give you a cache-line crossing. Admittedly it should be 25%, not 12%... Maybe Java's allocator is only 8-aligned?
    - MobiusHorizons 4y ago
      Why 16 byte aligned? That’s 128 bit, double the word size right? What is the significance of that alignment?
      - menaerus 4y ago
        long double on x86-64 Linux is 16 bytes big and therefore malloc implementation, or any other allocation routines for that matter, must return ptr aligned to the multiples of the largest primitive type. Interestingly, x86-64 Windows long double is 8 bytes so malloc on Windows returns ptrs aligned to the multiples of 8.
ithrow 4y ago
The tools used in this article require an understanding of what the hardware is actually doing. Which doesn't apply to 99% of business applications out there.
- m00x 4y ago
  Wait no, I want to spend months figuring out how to optimize user login for my 70 users to go from 20ms to 12ms.
  - Aeolun 4y ago
    I think user login is one of the few things in your application that is better if it takes longer (e.g. stronger password hashing).
  - TillE 4y ago
    Well, do you see programming as an obligatory tool to make stuff, or do you aspire to expertise and mastery of the craft? I think those are both probably legitimate approaches, but if it's the latter you really want to understand how the hardware works.
    - m00x 4y ago
      It's fine for a side project, but a waste of time for a business. I'm not going to charge my employer or client hundreds of extra hours just so I can learn a cool new thing.
  - saagarjha 4y ago
    I too have side projects :)
- FeepingCreature 4y ago
  It doesn't apply to 99% or maybe 99.9% of tasks, but given enough development time, it applies to most business applications eventually.
Thaxll 4y ago
This issue could happen in any language, as a matter of fact it was happening in native JVM code ( C++ ).
hayley-patton 4y ago
I wouldn't expect caching vs. cache ping-ponging to be an easy and clear-cut decision in any language, and any external dependency is thus possibly an issue there.
lmm 4y ago
How would C/zig/rust have helped? They don't expose CPU cache lines to you any more than Java does. And with C in particular, you lose more in memory safety bugs than you could ever hope to gain from increased performance, unless you're working in an industry where correctness doesn't actually matter.
- vlovich123 4y ago
  Not for cache line per se, but no runtime reflection means that any type information is static and won’t cause this kind of problem (+ no GC which is nice and makes leaks less likely). That’s a pretty significant RAM and performance saving and gets you within shouting distance of hand tuned assembly, especially since hot paths ARE frequently hand-tuned assembly. I think that’s more what OP meant. I’ll also note that while I don’t know what this service is doing, 300 RPS seems mighty low. That means each core is doing 30 RPS if you’re 12 wide. Maybe 60 since I don’t know what that 50% autoscaling piece means. Think about that. Each request is burning 16ms of CPU time. While I don’t know what this service does, this does seem like a lot when you consider just how fast a modern CPU is. That being said, it’s not impossible that even tweaking the Java code itself might unlock more CPU and that would be a more cost effective solution than using native code which is typically a bad fit for most web services (even rust).
  - lmm 4y ago
    > Not for cache line per se, but no runtime reflection means that any type information is static and won’t cause this kind of problem (+ no GC which is nice and makes leaks less likely). Possibly. Equally, maybe that runtime type information is what enables monomorphism that wouldn't be possible in C (Yes, C doesn't explicitly have virtual functions, but you end up needing to do the same thing, so you pass function pointers around and it's the same thing at runtime), and the result is better performance. > That’s a pretty significant RAM and performance saving and gets you within shouting distance of hand tuned assembly, especially since hot paths ARE frequently hand-tuned assembly. In my experience higher-level languages significantly outperform lower-level languages for realistic-sized business problems, because programmer effort is almost always the limiting factor. Most of the people using C/zig/rust because "muh fast" don't even know how to use a profiler, yet alone the level of multiple-tool analysis that we see in the article. And again, this is the kind of thing you'd need to be doing in C to address performance issues in the same kind of code. Sure, maybe you don't have reflective type information so it shows up as a mispredicted branch rather than a cache issue - but guess what, you also need this kind of low-level tool to get that information, C won't tell you anything about branch prediction either.
vitiral 4y ago
Fron the article: > We tend to think of modern JVMs as highly optimized runtime environments, in many cases rivaling more “performance-oriented” languages like C++. While it holds true for the majority of workloads What you say more aligns with my thoughts. I've never heard someone express the opinion in this article. Then again, I don't know anyone quite as knowledgeable about the jvm as this article is.
h0l0cube 4y ago
You can add erlang BEAM to that list. I was reading this article to see if it would inform me on how to better scale my elixir apps, and the answer is: no. It’s built to handle embarrassingly parallel computation by ensuring all lightweight processes enjoy isolation, while at the same time not having to think about threading at all when writing code.
ilyt 4y ago
well JS is just horrible programming language at every level. Frankly it's a technological wonder it can be run as fast as it is now.
P5fRxh5kUvp2th 4y ago
This has been understood for years and years. I remember there being a writeup about paint.net and what they had to do to keep performance. They were worrying a lot about memory and how to manipulate .net to manage it. If you want performance, you don't get to ignore memory no matter the language, but some languages/platforms make it easier and some get in the way.

delibes 4y ago

I've worked in a company that wrote Java code that needed to avoid false cacheing. It was fairly understandable and the tooling made it quite testable for performance regressions. However, the issue in the Netflix blogpost is in the JVM C++ code. I think it's entirely possible to encounter the problem in any language if you're writing performance-critical code.

saagarjha 4y ago
It’s often more common! Because the JVM will typically give you ok layouts “for free” and your hand-written native code might be too naive to do the same.
- wahern 4y ago
  JVM will typically give you ok layouts presumably because almost everything is an object, and when accounting for various headers and instrumentation two distinct objects are more likely to be >= 64 bytes apart even when allocated in succession. What I find a little odd is why those variables were only on different cache lines 1/8th (12.5%) of the time. What linker behavior or feature would result in randomly shifting those objects while preserving their adjacency? ASLR is the first thing that comes to mind, randomizing the base address of their shared region. But heap allocators on 64-bit architectures usually use 16-byte alignment rather than the random 8-byte alignment that would account for this behavior. Similarly, mmap normally would be page-aligned, even when randomized; or certainly at least 16-byte aligned?
  - FeepingCreature 4y ago
    Aiui, it's one object that happens to sometimes be allocated at an offset where two fields in it lie across a cache line boundary. ASLR only affects static offsets, not heap memory. edit: Pointers will be 8-aligned. Random 16-byte allocation if one pointer is at 8-offset and the next is at 0-offset will sometimes give you a cache-line crossing. Admittedly it should be 25%, not 12%... Maybe Java's allocator is only 8-aligned?
  - MobiusHorizons 4y ago
    Why 16 byte aligned? That’s 128 bit, double the word size right? What is the significance of that alignment?
    - menaerus 4y ago
      long double on x86-64 Linux is 16 bytes big and therefore malloc implementation, or any other allocation routines for that matter, must return ptr aligned to the multiples of the largest primitive type. Interestingly, x86-64 Windows long double is 8 bytes so malloc on Windows returns ptrs aligned to the multiples of 8.

ithrow 4y ago

The tools used in this article require an understanding of what the hardware is actually doing. Which doesn't apply to 99% of business applications out there.

m00x 4y ago
Wait no, I want to spend months figuring out how to optimize user login for my 70 users to go from 20ms to 12ms.
- Aeolun 4y ago
  I think user login is one of the few things in your application that is better if it takes longer (e.g. stronger password hashing).
- TillE 4y ago
  Well, do you see programming as an obligatory tool to make stuff, or do you aspire to expertise and mastery of the craft? I think those are both probably legitimate approaches, but if it's the latter you really want to understand how the hardware works.
  - m00x 4y ago
    It's fine for a side project, but a waste of time for a business. I'm not going to charge my employer or client hundreds of extra hours just so I can learn a cool new thing.
- saagarjha 4y ago
  I too have side projects :)
FeepingCreature 4y ago
It doesn't apply to 99% or maybe 99.9% of tasks, but given enough development time, it applies to most business applications eventually.

Thaxll 4y ago

This issue could happen in any language, as a matter of fact it was happening in native JVM code ( C++ ).

hayley-patton 4y ago

I wouldn't expect caching vs. cache ping-ponging to be an easy and clear-cut decision in any language, and any external dependency is thus possibly an issue there.

lmm 4y ago

How would C/zig/rust have helped? They don't expose CPU cache lines to you any more than Java does. And with C in particular, you lose more in memory safety bugs than you could ever hope to gain from increased performance, unless you're working in an industry where correctness doesn't actually matter.

vlovich123 4y ago
Not for cache line per se, but no runtime reflection means that any type information is static and won’t cause this kind of problem (+ no GC which is nice and makes leaks less likely). That’s a pretty significant RAM and performance saving and gets you within shouting distance of hand tuned assembly, especially since hot paths ARE frequently hand-tuned assembly. I think that’s more what OP meant. I’ll also note that while I don’t know what this service is doing, 300 RPS seems mighty low. That means each core is doing 30 RPS if you’re 12 wide. Maybe 60 since I don’t know what that 50% autoscaling piece means. Think about that. Each request is burning 16ms of CPU time. While I don’t know what this service does, this does seem like a lot when you consider just how fast a modern CPU is. That being said, it’s not impossible that even tweaking the Java code itself might unlock more CPU and that would be a more cost effective solution than using native code which is typically a bad fit for most web services (even rust).
- lmm 4y ago
  > Not for cache line per se, but no runtime reflection means that any type information is static and won’t cause this kind of problem (+ no GC which is nice and makes leaks less likely). Possibly. Equally, maybe that runtime type information is what enables monomorphism that wouldn't be possible in C (Yes, C doesn't explicitly have virtual functions, but you end up needing to do the same thing, so you pass function pointers around and it's the same thing at runtime), and the result is better performance. > That’s a pretty significant RAM and performance saving and gets you within shouting distance of hand tuned assembly, especially since hot paths ARE frequently hand-tuned assembly. In my experience higher-level languages significantly outperform lower-level languages for realistic-sized business problems, because programmer effort is almost always the limiting factor. Most of the people using C/zig/rust because "muh fast" don't even know how to use a profiler, yet alone the level of multiple-tool analysis that we see in the article. And again, this is the kind of thing you'd need to be doing in C to address performance issues in the same kind of code. Sure, maybe you don't have reflective type information so it shows up as a mispredicted branch rather than a cache issue - but guess what, you also need this kind of low-level tool to get that information, C won't tell you anything about branch prediction either.

vitiral 4y ago

Fron the article: > We tend to think of modern JVMs as highly optimized runtime environments, in many cases rivaling more “performance-oriented” languages like C++. While it holds true for the majority of workloads What you say more aligns with my thoughts. I've never heard someone express the opinion in this article. Then again, I don't know anyone quite as knowledgeable about the jvm as this article is.

h0l0cube 4y ago

You can add erlang BEAM to that list. I was reading this article to see if it would inform me on how to better scale my elixir apps, and the answer is: no. It’s built to handle embarrassingly parallel computation by ensuring all lightweight processes enjoy isolation, while at the same time not having to think about threading at all when writing code.

ilyt 4y ago

well JS is just horrible programming language at every level. Frankly it's a technological wonder it can be run as fast as it is now.

P5fRxh5kUvp2th 4y ago

This has been understood for years and years. I remember there being a writeup about paint.net and what they had to do to keep performance. They were worrying a lot about memory and how to manipulate .net to manage it. If you want performance, you don't get to ignore memory no matter the language, but some languages/platforms make it easier and some get in the way.