8 ms·
Seeing through hardware counters: A journey to 3x performance increase
- jareds 4y agoI consider myself a good Java programmer and I would never figure something like this out on my own. I'm ok with this fact.
- koyote 4y agoI would say that good chunks are things that people who've had to work in high performance computing will know about or would be able to figure out. Having said that, this line is probably the most difficult part of figuring the whole thing out: > Based on the fact that (5) “repne scan” is a rather rare operation in the JVM codebase, we were able to link this snippet to the routine for subclass checking I am not a Java programmer, but surely something like the above is not common knowledge?
- sponaugle 4y agoI too found that interesting. I would not have guessed that the use of 'repne scas' ( scan a string ) would be uncommon in a large codebase... perhaps there is some particular reason why that would be well known if you did a lot of Java profiling.
- Aeolun 4y agoIt strikes me as something that comes back from Google if you search ‘repne scan’?
- saagarjha 4y agoAmusingly this is something you can synthesize without knowing what the instruction is, precisely because you don’t know what the instruction is. Most code is simple arithmetic and memory operations that are relatively well known. The use of specialty instructions is pretty rare and often a good way to identify specific code sequences. For example, I don’t really hear much about “vgetmantsd” but the fact that I don’t is a good sign it does a specialized operation that would make it a good needle. (Ok, I can probably guess what it does, but ignore that.)
- hinoki 4y agoI have no idea what the super class cache is used for, but as soon as I read that there was good performance 12% = 1/8 of the time, after trying to scale up a server workload, I immediately thought of false sharing. Different people have different skills, but it’s still interesting to read about things you’ll never use. EDIT: and as sibling points out, I haven’t heard of the REPNE scan before, so that’s something interesting to look up.
- Thaxll 4y agoIt's ok, for good Java dev I assume they know at least how to troubleshoot performance issue with JFR, if you need to go deeper in native / perf events that's more the job of a very senior or perf people.
- stickfigure 4y agoThe issue was not in Java, but in the C++ implementation of the JVM. I think you can be forgiven.
- pyrolistical 4y agoI'm starting to see why systems programming people (C/zig/rust) sneer at java/node.js/dot net. The jvm and v8 is just another layer of abstraction that gets in the way when performances problems like this arise at scale. The tools used in this article require an understanding of what the hardware is actually doing.
- delibes 4y agoI've worked in a company that wrote Java code that needed to avoid false cacheing. It was fairly understandable and the tooling made it quite testable for performance regressions. However, the issue in the Netflix blogpost is in the JVM C++ code. I think it's entirely possible to encounter the problem in any language if you're writing performance-critical code.
- saagarjha 4y agoIt’s often more common! Because the JVM will typically give you ok layouts “for free” and your hand-written native code might be too naive to do the same.
- wahern 4y agoJVM will typically give you ok layouts presumably because almost everything is an object, and when accounting for various headers and instrumentation two distinct objects are more likely to be >= 64 bytes apart even when allocated in succession. What I find a little odd is why those variables were only on different cache lines 1/8th (12.5%) of the time. What linker behavior or feature would result in randomly shifting those objects while preserving their adjacency? ASLR is the first thing that comes to mind, randomizing the base address of their shared region. But heap allocators on 64-bit architectures usually use 16-byte alignment rather than the random 8-byte alignment that would account for this behavior. Similarly, mmap normally would be page-aligned, even when randomized; or certainly at least 16-byte aligned?
- FeepingCreature 4y ago
- sponaugle 4y agoIt is great to see a tie from a higher level application performance problem all the way down to a processor architecture issue (the L1 Cache). I was working at Intel when VTune first came into existence and it was such an awesome way to see how small code changes results in significant differences in actual execution. This was especially true when the Pentium Pro came out with the uOps architecture. One of the many good reasons to have a real 'full stack' appreciation of the platforms we work on.
- Aeolun 4y agoI really like these Netflix tech blog posts. They actually dive into areas that I’m not aware of and show me some technical details instead of high level waffling. It’s interesting because the longer I work, the better my own code becomes, and as I make fewer mistakes myself, I’m starting to see more unexpected issues that have their root cause deeper down the stack. In my case that’s mostly dependencies. At Netflix they go a step further and find issues with the JVM itself :)
- DanAtC 4y agoFor anyone interested in how the graphs were made: https://oss.oetiker.ch/rrdtool/ https://oss.oetiker.ch/rrdtool/ Happy to see people still using RRD after all these years.
- ilyt 4y agoYeah I had "I know that look" reaction when seeing it. The first actual TSDB in a way
- MobiusHorizons 4y agoThanks for linking this. I used it in one of my first programming jobs, and forgot the name. Had occasion to use something similar recently and couldn’t find it.
- tetha 4y agoI was totally happy to see these graphs after all these years, haha. Now I'm wondering if netflix has munin or cacti or something like that running in their infra.
- bbyers 4y ago
- fnordpiglet 4y agoThese posts both make me violently aroused as well as profoundly depressed basically every employer I’ve ever had in 30 years doesn’t value such analysis. This is what engineering looks like, and it’s something no product manager can ever comprehend.
- skellington 4y agoYup, very few workplaces would know how to reward people with this skill level.
- fnordpiglet 4y agoAlmost every engineer I’ve worked with have this skill level, but they’re not given the leeway, direction, or advice to get to the conclusion. Netflix must have an amazing culture in these areas.
- ilyt 4y agoReally that just need manager that presents the results to higher-ups as raw cash savings. "My team makes your team apps and make them run cheaper"
- bitcharmer 4y agoI've been a high performance systems engineer for a number of investment banks over the last 10+ years and people with deep knowledge like that are quite rare; that's why we get paid handsomely. Mind sharing what field you work in that gathers so much talent like that?
- fnordpiglet 4y agoKnowledge != skill. With the skill you can acquire the knowledge given the right conditions and motivation. I would say the vast majority of engineers I work with have never had the need or opportunity to invest the time or experiential opportunities to acquire the knowledge required - but they COULD. I actually find that quite sad, and hence my depression I noted. Nowadays this is extending all the way through school. Machine architecture and low level machine microcode etc isn’t universally taught in CS programs any more. I’ve worked in all the industries, including a long time in yours. We probably even overlapped or maybe worked together!
- dmak 4y agoThat was a lot lower level than I expected. I like how humbling it is to see such deep dives in areas that I would not know how to debug.
- jeffbee 4y agoI am a little confused because I do not get this level of PMU access on AWS instances short of "metal". Usually if I want to analyze some performance problem I need to move the workload to "metal" instance types and usually the c6i.metal, since the Ice Lake PMU is so much more capable. If I try to count machine clears on a non-metal instance perf tells me the event doesn't exist, and if I give it the raw event encoding it runs but says the event isn't supported.
- emilburzo 4y agohttps://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2.html https://www.brendangregg.com/blog/2017-05-04/the-pmcs-of-ec2...
- xxpor 4y agoputs incredibly nitpicky code reviewer hat on Does the JVM allow using __attribute__ annotations? There's one for alignment, __attribute__((aligned(L1_DCACHE_LINE_SIZE))) (where L1_DCACHE_LINE_SIZE is defined by gcc -DL1_DCACHE_LINE_SIZE=`getconf LEVEL1_DCACHE_LINESIZE`) that could be used instead of the array, see: https://gcc.gnu.org/onlinedocs/gcc-12.2.0/gcc/Common-Variable-Attributes.html#Common-Variable-Attributes https://gcc.gnu.org/onlinedocs/gcc-12.2.0/gcc/Common-Variabl...
- bitcharmer 4y agoI think the array might be better because it works on all architectures and all platforms.
- deleted 4y ago[deleted]
- wahern 4y agoC11 added `_Alignas`, and it looks like C++11 has `alignas`, so you don't need to rely on vendor-specific extensions for alignment, unless the JVM restricts itself to an earlier standard. LEVEL1_DCACHE_LINESIZE looks to be specific to Linux and Solaris, however.
- raggi 4y agothey can run vtune on their aws nodes, what a privilege!
- emilburzo 4y agohttps://www.intel.com/content/www/us/en/develop/documentation/vtune-cookbook/top/configuration-recipes/profiling-applications-in-aws-ec2-instances.html https://www.intel.com/content/www/us/en/develop/documentatio... https://www.intel.com/content/www/us/en/developer/articles/technical/intel-vtune-amplifier-functionality-on-aws-instances.html https://www.intel.com/content/www/us/en/developer/articles/t... it doesn't seem netflix specific?
- raggi 4y agoIt’s not Netflix specific, but access to hardware counters is more rare among instances than you might realize if you only scan the table on Intels pages.
- jkrubin 4y agoI thought based on the pre : titled that this was about epoxy counter tops.
- thrwawy74 4y agoThis is a cool post. Initially I was discouraged by the list of things needed to properly diagnose it, but that doesn't seem unreasonable: Java Flight Recorder perf-map-agent PerfSpect (Performance Monitoring Counters (PMCs) Intel vTune As folks keep looking to Rust for safety and performance, I wonder what in-language facilities exist to say "this variable should exist in a separate cpu cache line". That would be neat level of control in contrast to recognizing this in debugging + adding arbitrary padding.
- menaerus 4y agoThat's not the something that can be applied through compile-time analysis nor it is something that the language can "solve". There is actually nothing to be solved unless you have an actual workload which justifies making a change like this. As a matter of fact, applying such pattern blindly would have a negative effect on an increased memory pressure. You're increasing your application runtime memory footprint by doing this.
- thrwawy74 4y agoI meant in the context that you knew ahead of time this would cause a cache coherency issue. I agree you're almost always finding this after the fact. :)
- brenns10 4y agoThis is an excellent description of a false sharing perfoance issue, and I wouldn't envy the work involved in tracking that code down through the JVM. These sorts of issues are common inside the OS kernel itself. Kernels are managing lots of shared resources, and as you add more CPUs, these sorts of issues emerge as bottlenecks. They're enough of a problem that the perf tool recently got a new sub-command, "perf c2c", which tries to help identify false sharing (and other cache ping-ponging) issues much quicker. If you're interested in learning more about this class of issues and the new tool, there was an excellent talk at LPC 2022 by Arnaldo Carvalho de Melo. And as luck would have it, the videos just came out, so I can link it here. In my opinion it was one of the best talks of the conference. https://youtu.be/A23UopXKq6E https://youtu.be/A23UopXKq6E
- ttsiodras 4y agoThanks for sharing! The slides are here: http://vger.kernel.org/~acme/prez/linux-plumbers-2022/ http://vger.kernel.org/~acme/prez/linux-plumbers-2022/
- nyanpasu64 4y agoWould an atomic mutable subclass cache (not sure what it's used for, downcasting?) be unnecessary in a language built around static rather than dynamic dispatch by default, like C/C++/Rust and perhaps Go? Or would it still speed up dynamically dispatched code, but is less practical or worthwhile so it isn't used in practice? (Though Rust's Arc also suffers from atomic contention similar to this blog post, when used across dozens of threads: https://pkolaczk.github.io/server-slower-than-a-laptop/ https://pkolaczk.github.io/server-slower-than-a-laptop/) Also it's somewhat ironic that the JVM source code (https://github.com/openjdk/jdk8u/blob/jdk8u352-b07/hotspot/src/cpu/x86/vm/macroAssembler_x86.cpp#L5093 https://github.com/openjdk/jdk8u/blob/jdk8u352-b07/hotspot/s...) says "This code is rarely used, so simplicity is a virtue here" at the site of a bottleneck.
- JonChesterfield 4y agoDisproportionately annoyed by inserting a whole cache line of padding between the variables instead of just enough to move them to different lines. That's probably just to make the blog post simpler though.
- tdiff 4y agoJust in case somebody missed it from the article, there is a 1-hour long youtube video about the same issue: https://www.youtube.com/watch?v=G40VfIsnCdo https://www.youtube.com/watch?v=G40VfIsnCdo
- darkwater 4y agoReading the post I can imagine the days/weeks it took for those bright individuals (I spot Brendan Gregg among them, and being Netflix I can safely infer they are all very smart indeed) to find the issue behind this - and all the years of work on the tooling to make it possible. Now, add the Kubernetes layer on top of this and I think you will add at very least one extra week of debugging...
- Const-me 4y agoNice article, but strange fix. What they did: Klass* field1; uint8_t padding[ 64 ]; Klass** field2; What I usually do: alignas(64) Klass* field1; alignas(64) Klass** field2;
- deleted 4y ago[deleted]
- iamchp 4y agoThe JDK issue ticket linked towards the bottom of the post has some interesting discussions. Apparently Developers at RedHat ran into a similar problem and decided to build an 'agent' to help developers identify code patterns triggering the issue. Slightly different approach to the Netflix post, but still an interesting line of effort to find problematic Java code! For anyone interested: https://bugs.openjdk.org/browse/JDK-8180450?focusedCommentId=14536886&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14536886 https://bugs.openjdk.org/browse/JDK-8180450?focusedCommentId...