4 ms·
My gut reaction is that this[1] looks like a NUMA issue. The flip-flop performance profile immediately reminds me of a very gnarly prod issue i lead the investi
by CraigJPerry 5y ago
My gut reaction is that this[1] looks like a NUMA issue. The flip-flop performance profile immediately reminds me of a very gnarly prod issue i lead the investigation of. I’d want to eliminate NUMA first from my investigation.
You can use “numactl --cpunodebind=0 --membind=0 <your original java benchmark command>” to pin the JVM to a particular NUMA node and immediately discount my notion as bunkum.
[1] all the reasons mentioned that make me think numa:
1. The machine affected has multiple NUMA domains
2. The M1 is a single NUMA domain chip and does not exhibit the behaviour
3. The real thing making me think this - the consistent performance flip flop is close to the ~40% performance overhead of accessing remote memory vs local that i’ve seen previously. You’re seeing higher overheads but your code is a much tighter loop of memory access than my previous experience so that could explain the difference i think.
- cospaia 5y agoThis seems to be the issue. Running it 10 times gives me quite consistent performance near the top. 0 - Passes: 5876, count: 78498, Valid: true 1 - Passes: 5870, count: 78498, Valid: true 2 - Passes: 5944, count: 78498, Valid: true 3 - Passes: 5903, count: 78498, Valid: true 4 - Passes: 6051, count: 78498, Valid: true 5 - Passes: 5874, count: 78498, Valid: true 6 - Passes: 5902, count: 78498, Valid: true 7 - Passes: 6041, count: 78498, Valid: true 8 - Passes: 5906, count: 78498, Valid: true 9 - Passes: 5889, count: 78498, Valid: true Will run it 100 times now to lower the risk that this is just a coincidence.
- CraigJPerry 5y agoAhh nice! The thing that blew my mind when i ran into this was that the jvm had actually been NUMA aware for many years at that point but it was all turned off by default. This was jdk8 days so things are probably better now. After i figured the root cause, I spent about a day logging results in excel as i tested various GC options, the only saving grace was i recall the Oracle documentation for GC options and their NUMA effects was pretty good. That optimisation time was really well spent though, i got meaningful improvements in both service latency and throughput.
- cospaia 5y agoI had this issue using JDK17 and 18 as well, so at least by default things are not better. =) I wonder if there is a way that does not involve using `numactl`. It needs the container to run in privileged mode and that is not how the containers are run in the drag-racing benchmark...
- CraigJPerry 5y agoCan you use the jvm’s -xx:+UseNUMA option instead?
- cospaia 5y agoThanks! It does and it does not. Here's 100 rounds where it does: https://github.com/PEZ/ghost-chase-condition/tree/master/test-runs/java-use-numa https://github.com/PEZ/ghost-chase-condition/tree/master/tes... But in other cases I have it seems to not help that much. Not sure how to minimize those, but will try figure it out.