5 ms·

Know Thy Java Object Memory Layout

66 points by javinpaul 13y ago

kasey_junk 13y ago
Thank you! You just saved me another day trying to track down why some "supposed" hot fields were causing cache misses. At some point it would be great if Java added some support for memory "hints" that would allow the JVM to interpret these hacks for what they are.
- fmstephe 13y ago
  I agree. Maybe not news to you, but I think that there are a few things on the cards for dealing with memory layout. I know there is a strong push from certain corners, including Doug Lea, for (object) headerless struct values to be included in Java 9. There is a hint of a an @Contended annotation to get the JVM to put a variable on its own cache line. But it seems to me that a very effective fix would simply be to ask all JVMs to stop fiddling the variable ordering.
  - kasey_junk 13y ago
    I'd much prefer a solution that allowed the code to be read easily by humans (ie, wouldn't need to know variable ordering rules), and the hints be there to handle the machine implementation. That said, I'd take standardizing the variable ordering.
    - fmstephe 13y ago
      I think that the proposed solutions, @Contended and perhaps some sort of @Respect-My-Variable-Order, is adding more knowledge overhead than a simple rule, the JVM will lay out your variables in the order you declare them and it won't drop any even if it knows you aren't using them. This feels like the simplest option to implement on both sides, the JVMs do less work and the programmers have simple rules to do in-object memory layout if we like that sort of thing. More than anything else it means we don't have to wait on a new JSR for each type of memory layout we want to achieve. (I suppose there might be some CPU arch portability that the JVM might be able to do if we go with the annotations - but I think that we may be chasing a fools dream down that road)
  - twistedpair 13y ago
    > there are a few things on the cards for dealing with memory layout LOL. Cards... ha. You must be referring to the G1 GC and the memory 'card' design? Ha.
    - deleted 13y ago
      [deleted]
  - lmm 13y ago
    >But it seems to me that a very effective fix would simply be to ask all JVMs to stop fiddling the variable ordering. That would make the common case much worse. 99% of Java programmers want the JVM to optimize their field ordering for them, to the best of its abilities, so that they can put them in the order that makes sense in the source code and simply not think about whether a different order would be better. It's the kind of low-level implementation detail that Java is all about ignoring; if you want C++, you know where to find it. I support an annotation-hinting approach; that would keep the common case the way it is today, while allowing people who really want to fiddle with these things to do so.
    - fmstephe 13y ago
      I would expect that any advantage the JVM gains from variable order fiddling (and variable dropping) is minimal. Whereas the problems we get on multicore systems with false sharing, and other cache-line maladies, are significant. This is, however, just what I expect to be the case. It may be that in practice the JVM is getting good results from fiddling with our variables. You may be right that annotations would suit Java better than allowing programmers to layout objects as they wish. The strong motivation for this is that the fix would be simple (and perhaps have no serious downside?) and mean that work can be done without waiting for a JSR to reach maturity. From what I read the state of the art around this is to use relatively unstable hacks which require very strong knowledge of JVM implementations - see the example code from Martin Thomson in the linked article. You could still happily write your variables for humans in most cases. I would love to know how large the downside is to fixed variable ordering.
  - a-priori 13y ago
    I think instead a @ReorderingBarrier annotation might be a good solution. Like a memory barrier which doesn't allow reordering of memory operations across that point, a reordering barrier wouldn't allow fields declared before that point to be placed in memory after the field it's declared on or any declared afterwards. It could also be used to control the alignment of the following field, to a cache line for example: public class TestClass { int a1; long a2; OtherClassA a3; @ReorderingBarrier(alignment=CACHE_LINE) OtherClassB b1; int b2; int b3; long b4; // ... } If I understand the reordering rules correctly, without the annotation it would be rearranged as: a2, b4, a1, b2, b3, a3, b1. With the barrier it would be rearranged as: a2, a1, a3, (cache line alignment break), b4, b2, b3, b1.
    - fmstephe 13y ago
      If they provide an alignment argument then the annotation based approach becomes much more attractive. You could even go all cross platform and have 'alignment=cacheline' That would be a lot of fun to play with.
      - a-priori 13y ago
        Hah, just as you were writing that I had the same thought and did a ninja edit where I switched from saying a specific alignment to CACHE_LINE. :)
  - AlisdairO 13y ago
    > I know there is a strong push from certain corners, including Doug Lea, for (object) headerless struct values to be included in Java 9. Thank christ. Being able to avoid the per-object overhead when necessary would be extremely welcome.
- dclusin 13y ago
  What kinds of problems are you working on where this change in behavior of the JVM has implications for your software stack? I myself work on a financial trading platform and these sorts of things matter to us. Curious what other kinds of problems/domains are affected by this issue.
  - kasey_junk 13y ago
    Financial trading as well, but lots of real time systems would have similar issues.
  - fmstephe 13y ago
    Afraid my interest in this comes from building a matching engine. There are definitely other domains where this stuff is important but I bet the financial sector is the biggest employer. NB: I write a matching engine in my spare time. For pay I don't touch anything as interesting this. https://github.com/fmstephe/matching_engine https://github.com/fmstephe/matching_engine
    - deleted 13y ago
      [deleted]
- SanderMak 13y ago
  The packed objects proposal is pretty interesting in this regard: http://duimovich.blogspot.nl/2012/11/packed-objects-in-java.html http://duimovich.blogspot.nl/2012/11/packed-objects-in-java....
RyanZAG 13y ago
Isn't there a decent amount of overhead in declaring and extending an abstract class? Does the cache grouping outweigh the overhead?
- kasey_junk 13y ago
  A) It depends on the JVM and B) on HotSpot at least, I believe that it is smart enough to figure out that all virtual methods are of the same type and optimize them as if they were non virtual.
  - fmstephe 13y ago
    I can confirm that HotSpot is very aggressive at collapsing type hierachies and de-virtualising method calls. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117.2420&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117... This is one of the places where Java can outperform C++. Where c++ is unable to statically devirtualise method calls and HotSpot will just do it.
benjamincburns 13y ago
Does this ordering happen by the JVM at runtime, or is it part of compilation? Is there a static analysis heuristic which could detect false sharing? Maybe his could be better solved in the compiler than with implementation-specific hacks like these?
- tsewlliw 13y ago
  It cannot be done as part of compilation. For example, the superclass you run against may not be the same as the superclass you compile against.

66 points by javinpaul 13y ago

Thank you! You just saved me another day trying to track down why some "supposed" hot fields were causing cache misses. At some point it would be great if Java added some support for memory "hints" that would allow the JVM to interpret these hacks for what they are.

fmstephe 13y ago
I agree. Maybe not news to you, but I think that there are a few things on the cards for dealing with memory layout. I know there is a strong push from certain corners, including Doug Lea, for (object) headerless struct values to be included in Java 9. There is a hint of a an @Contended annotation to get the JVM to put a variable on its own cache line. But it seems to me that a very effective fix would simply be to ask all JVMs to stop fiddling the variable ordering.
- kasey_junk 13y ago
  I'd much prefer a solution that allowed the code to be read easily by humans (ie, wouldn't need to know variable ordering rules), and the hints be there to handle the machine implementation. That said, I'd take standardizing the variable ordering.
  - fmstephe 13y ago
    I think that the proposed solutions, @Contended and perhaps some sort of @Respect-My-Variable-Order, is adding more knowledge overhead than a simple rule, the JVM will lay out your variables in the order you declare them and it won't drop any even if it knows you aren't using them. This feels like the simplest option to implement on both sides, the JVMs do less work and the programmers have simple rules to do in-object memory layout if we like that sort of thing. More than anything else it means we don't have to wait on a new JSR for each type of memory layout we want to achieve. (I suppose there might be some CPU arch portability that the JVM might be able to do if we go with the annotations - but I think that we may be chasing a fools dream down that road)
- twistedpair 13y ago
  > there are a few things on the cards for dealing with memory layout LOL. Cards... ha. You must be referring to the G1 GC and the memory 'card' design? Ha.
  - deleted 13y ago
    [deleted]
- lmm 13y ago
  >But it seems to me that a very effective fix would simply be to ask all JVMs to stop fiddling the variable ordering. That would make the common case much worse. 99% of Java programmers want the JVM to optimize their field ordering for them, to the best of its abilities, so that they can put them in the order that makes sense in the source code and simply not think about whether a different order would be better. It's the kind of low-level implementation detail that Java is all about ignoring; if you want C++, you know where to find it. I support an annotation-hinting approach; that would keep the common case the way it is today, while allowing people who really want to fiddle with these things to do so.
  - fmstephe 13y ago
    I would expect that any advantage the JVM gains from variable order fiddling (and variable dropping) is minimal. Whereas the problems we get on multicore systems with false sharing, and other cache-line maladies, are significant. This is, however, just what I expect to be the case. It may be that in practice the JVM is getting good results from fiddling with our variables. You may be right that annotations would suit Java better than allowing programmers to layout objects as they wish. The strong motivation for this is that the fix would be simple (and perhaps have no serious downside?) and mean that work can be done without waiting for a JSR to reach maturity. From what I read the state of the art around this is to use relatively unstable hacks which require very strong knowledge of JVM implementations - see the example code from Martin Thomson in the linked article. You could still happily write your variables for humans in most cases. I would love to know how large the downside is to fixed variable ordering.
- a-priori 13y ago
  I think instead a @ReorderingBarrier annotation might be a good solution. Like a memory barrier which doesn't allow reordering of memory operations across that point, a reordering barrier wouldn't allow fields declared before that point to be placed in memory after the field it's declared on or any declared afterwards. It could also be used to control the alignment of the following field, to a cache line for example: public class TestClass { int a1; long a2; OtherClassA a3; @ReorderingBarrier(alignment=CACHE_LINE) OtherClassB b1; int b2; int b3; long b4; // ... } If I understand the reordering rules correctly, without the annotation it would be rearranged as: a2, b4, a1, b2, b3, a3, b1. With the barrier it would be rearranged as: a2, a1, a3, (cache line alignment break), b4, b2, b3, b1.
  - fmstephe 13y ago
    If they provide an alignment argument then the annotation based approach becomes much more attractive. You could even go all cross platform and have 'alignment=cacheline' That would be a lot of fun to play with.
    - a-priori 13y ago
      Hah, just as you were writing that I had the same thought and did a ninja edit where I switched from saying a specific alignment to CACHE_LINE. :)
- AlisdairO 13y ago
  > I know there is a strong push from certain corners, including Doug Lea, for (object) headerless struct values to be included in Java 9. Thank christ. Being able to avoid the per-object overhead when necessary would be extremely welcome.
dclusin 13y ago
What kinds of problems are you working on where this change in behavior of the JVM has implications for your software stack? I myself work on a financial trading platform and these sorts of things matter to us. Curious what other kinds of problems/domains are affected by this issue.
- kasey_junk 13y ago
  Financial trading as well, but lots of real time systems would have similar issues.
- fmstephe 13y ago
  Afraid my interest in this comes from building a matching engine. There are definitely other domains where this stuff is important but I bet the financial sector is the biggest employer. NB: I write a matching engine in my spare time. For pay I don't touch anything as interesting this. https://github.com/fmstephe/matching_engine https://github.com/fmstephe/matching_engine
  - deleted 13y ago
    [deleted]
SanderMak 13y ago
The packed objects proposal is pretty interesting in this regard: http://duimovich.blogspot.nl/2012/11/packed-objects-in-java.html http://duimovich.blogspot.nl/2012/11/packed-objects-in-java....

RyanZAG 13y ago

Isn't there a decent amount of overhead in declaring and extending an abstract class? Does the cache grouping outweigh the overhead?

kasey_junk 13y ago
A) It depends on the JVM and B) on HotSpot at least, I believe that it is smart enough to figure out that all virtual methods are of the same type and optimize them as if they were non virtual.
- fmstephe 13y ago
  I can confirm that HotSpot is very aggressive at collapsing type hierachies and de-virtualising method calls. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117.2420&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117... This is one of the places where Java can outperform C++. Where c++ is unable to statically devirtualise method calls and HotSpot will just do it.

benjamincburns 13y ago

Does this ordering happen by the JVM at runtime, or is it part of compilation? Is there a static analysis heuristic which could detect false sharing? Maybe his could be better solved in the compiler than with implementation-specific hacks like these?

tsewlliw 13y ago
It cannot be done as part of compilation. For example, the superclass you run against may not be the same as the superclass you compile against.