7 ms·

This seems to be the issue. Running it 10 times gives me quite consistent performance near the top. 0 - Passes: 5876, count: 78498, Valid: true 1 - Pas

This seems to be the issue. Running it 10 times gives me quite consistent performance near the top. 0 - Passes: 5876, count: 78498, Valid: true 1 - Passes: 5870, count: 78498, Valid: true 2 - Passes: 5944, count: 78498, Valid: true 3 - Passes: 5903, count: 78498, Valid: true 4 - Passes: 6051, count: 78498, Valid: true 5 - Passes: 5874, count: 78498, Valid: true 6 - Passes: 5902, count: 78498, Valid: true 7 - Passes: 6041, count: 78498, Valid: true 8 - Passes: 5906, count: 78498, Valid: true 9 - Passes: 5889, count: 78498, Valid: true Will run it 100 times now to lower the risk that this is just a coincidence.

CraigJPerry 5y ago
Ahh nice! The thing that blew my mind when i ran into this was that the jvm had actually been NUMA aware for many years at that point but it was all turned off by default. This was jdk8 days so things are probably better now. After i figured the root cause, I spent about a day logging results in excel as i tested various GC options, the only saving grace was i recall the Oracle documentation for GC options and their NUMA effects was pretty good. That optimisation time was really well spent though, i got meaningful improvements in both service latency and throughput.
- cospaia 5y ago
  I had this issue using JDK17 and 18 as well, so at least by default things are not better. =) I wonder if there is a way that does not involve using `numactl`. It needs the container to run in privileged mode and that is not how the containers are run in the drag-racing benchmark...
  - CraigJPerry 5y ago
    Can you use the jvm’s -xx:+UseNUMA option instead?
    - cospaia 5y ago
      Thanks! It does and it does not. Here's 100 rounds where it does: https://github.com/PEZ/ghost-chase-condition/tree/master/test-runs/java-use-numa https://github.com/PEZ/ghost-chase-condition/tree/master/tes... But in other cases I have it seems to not help that much. Not sure how to minimize those, but will try figure it out.

CraigJPerry 5y ago

Ahh nice! The thing that blew my mind when i ran into this was that the jvm had actually been NUMA aware for many years at that point but it was all turned off by default. This was jdk8 days so things are probably better now. After i figured the root cause, I spent about a day logging results in excel as i tested various GC options, the only saving grace was i recall the Oracle documentation for GC options and their NUMA effects was pretty good. That optimisation time was really well spent though, i got meaningful improvements in both service latency and throughput.

cospaia 5y ago
I had this issue using JDK17 and 18 as well, so at least by default things are not better. =) I wonder if there is a way that does not involve using `numactl`. It needs the container to run in privileged mode and that is not how the containers are run in the drag-racing benchmark...
- CraigJPerry 5y ago
  Can you use the jvm’s -xx:+UseNUMA option instead?
  - cospaia 5y ago
    Thanks! It does and it does not. Here's 100 rounds where it does: https://github.com/PEZ/ghost-chase-condition/tree/master/test-runs/java-use-numa https://github.com/PEZ/ghost-chase-condition/tree/master/tes... But in other cases I have it seems to not help that much. Not sure how to minimize those, but will try figure it out.