6 ms·

JVM Anatomy Quarks: Compressed References

141 points by luu 8y ago

kristianp 8y ago
"The name underlines the fact that the single post is not enough to form reasonable matter" What does that bit of legalese mean?
- userbinator 8y ago
  That is an exceedingly awkward sentence --- either due to non-native influence or an attempt to sound fancy. It's really saying something like "the articles refer to each other so you might not understand everything from reading just one."
- monocasa 8y ago
  I think they're just making a joke, alluding to the fact that single articles here won't tell you a lot about the JVM.
- eschutte2 8y ago
  Reference to physical quarks
fulafel 8y ago
Data layout optimizations are a really neglected part of compiler engineering. Performance bottlenecks are now in caching and moving data in the memory hierarchy, instead of ALU throughput. So compact data representation lets you fit more items in the same amount of cache and move more items in the same amoutn of banwidth. Compressed references is one of the most basic optimizations that is routinely done by hand when programmers do performance work. Others include storing data in a columnar form (aka AoS->SoA transformation), altering array layout to be cache-friendly (tiling / swizzling), moving rarely accessed members out of line, and field shrinking based on dynamic range analysis. The history of "compressed references", afaict, is reactive on the JVM: people were shocked at memory usage increases when moving from 32-bit to 64-bit machines. So it came about semi-accidentally and is not really indicative of systematic work on data layout optimizations in the JVM... There is also a term of art in CS that refers to economical use of memory: https://en.wikipedia.org/wiki/Succinct_data_structure https://en.wikipedia.org/wiki/Succinct_data_structure and a second one that is for shrinking data structures based on content entropy: https://en.wikipedia.org/wiki/Compressed_data_structure https://en.wikipedia.org/wiki/Compressed_data_structure - this literature is dealing with more advanced encoding tricks that are rather far from what a compiler could realistically do on its own, and often make big tradeoffs such as making the structure read-only.
- denormalfloat 8y ago
  > Data layout optimizations are a really neglected part of compiler engineering. From what I understand this is mostly impossible in other popular languages because the compiler just puts fields of your structs in the order you wrote it in. (You wanted this, right? Here ya go!) JIT'd languages don't have this problem because they can rewrite the layouts however they want and based on real time profiling. AoT languages either have to run first and compile with the profiling information to have hope of doing this, or else the author has to break out perf and VTune and do it manually. Both are difficult and time consuming, which is why it doesn't happen much in practice. A minor example of this is Protobuf, where the IDL translates to the same field layout in C++. Could the fields be packed into a more cache friendly layout? Sure, but only on a case-by-case, application-by-application basis. It doesn't work in general, even though proto authors don't care about the precise layout.
  - vardump 8y ago
    > JIT'd languages don't have this problem because they can rewrite the layouts however they want and based on real time profiling. Any examples of this? > AoT languages either have to run first and compile with the profiling information to have hope of doing this... What prevents one from JITting AoT compiled binaries?
    - pjmlp 8y ago
      > Any examples of this? Yes, Java and .NET. Some JVM implementations, e.g. IBM J9, can save profile information between runs and use it for speeding up code generation, based on the complete history of the executable. Google also started doing this when they rebooted ART into a mix of interpreter written in Assembly/JIT/AOT runtime. Starting with Android P they are even uploading profile information back to the store, so that similar devices can reach optimized AOT compiled code faster. .NET Framework allows some control to save PGO data, and variants like that WinRT/UWP cloud compiler take it further. > What prevents one from JITting AoT compiled binaries? It relatively complex, versus just starting from bytecodes. Hence why mainframes always took the approach of either JITing at installation time, or using microcoded CPUs. Like on Burroughs, Xerox PARC and IBM cases.
zamalek 8y ago
> Once the heap size gets larger than the threshold under which compressed references are working, a surprising thing happens: references suddenly become uncompressed Am I correct in assuming that this is made possible because Hotspot can re-JIT? So the GC/Allocator just warns the JITter, "hey, all of your Oops [flags] code is no longer valid," resulting in a re-JIT of everything? Or a prologue in every method that checks the Oops flags (and triggers a re-JIT)?
- deadc0de 8y ago
  No, compressed oops mode doesn't change at runtime. That's pretty hard to pull off. You'd need to change object layout dynamically and unless you want to reformat the whole heap you'd need tags or regions with different pointer rules. Any of these techniques would have a very substantial runtime overhead.
- xxs 8y ago
  nope, CompressedOps is an old technique that depends on the max heap. less than 32GB, 64bit -> compressedOps
colechristensen 8y ago
The short version is that you are often better off running multiple instances of a JVM application with a heap <32GB when you have the choice between scaling up and scaling out.
- speedplane 8y ago
  > The short version is that you are often better off running multiple instances of a JVM application with a heap <32GB when you have the choice between scaling up and scaling out. We're in an awkward time right now, where many servers have more than 32GB of RAM, but not that much more. Once it becomes routine for servers to have 1TB of memory, this will no longer be much of an issue.
- cbsmith 8y ago
  Depends on the use case. Sometimes compressed oops is counter productive, and it really isn't a significant piece of overhead if you are say... doing matrix multiplication with very large arrays.
- vbezhenar 8y ago
  I wonder if using 16 bytes as an alignment would help in those corner cases. Some objects would use up to 33% more space (24 bytes -> 32 bytes), but I think that for most objects the increase will not be very noticeable and one extra bit would allow to use up to 64 GB of RAM with compressed pointers.
- callmeal 8y ago
  That depends on your memory usage. There is an awkward range (32GB - 48GB) where the increase in heap size does not give you a proportional increase in available memory because of the doubling of references. That is where you're better off running multiple instances. But for heaps bigger than 48GB running a single jvm is more efficient.

141 points by luu 8y ago

"The name underlines the fact that the single post is not enough to form reasonable matter" What does that bit of legalese mean?

userbinator 8y ago
That is an exceedingly awkward sentence --- either due to non-native influence or an attempt to sound fancy. It's really saying something like "the articles refer to each other so you might not understand everything from reading just one."
monocasa 8y ago
I think they're just making a joke, alluding to the fact that single articles here won't tell you a lot about the JVM.
eschutte2 8y ago
Reference to physical quarks

fulafel 8y ago

Data layout optimizations are a really neglected part of compiler engineering. Performance bottlenecks are now in caching and moving data in the memory hierarchy, instead of ALU throughput. So compact data representation lets you fit more items in the same amount of cache and move more items in the same amoutn of banwidth. Compressed references is one of the most basic optimizations that is routinely done by hand when programmers do performance work. Others include storing data in a columnar form (aka AoS->SoA transformation), altering array layout to be cache-friendly (tiling / swizzling), moving rarely accessed members out of line, and field shrinking based on dynamic range analysis. The history of "compressed references", afaict, is reactive on the JVM: people were shocked at memory usage increases when moving from 32-bit to 64-bit machines. So it came about semi-accidentally and is not really indicative of systematic work on data layout optimizations in the JVM... There is also a term of art in CS that refers to economical use of memory: https://en.wikipedia.org/wiki/Succinct_data_structure https://en.wikipedia.org/wiki/Succinct_data_structure and a second one that is for shrinking data structures based on content entropy: https://en.wikipedia.org/wiki/Compressed_data_structure https://en.wikipedia.org/wiki/Compressed_data_structure - this literature is dealing with more advanced encoding tricks that are rather far from what a compiler could realistically do on its own, and often make big tradeoffs such as making the structure read-only.

denormalfloat 8y ago
> Data layout optimizations are a really neglected part of compiler engineering. From what I understand this is mostly impossible in other popular languages because the compiler just puts fields of your structs in the order you wrote it in. (You wanted this, right? Here ya go!) JIT'd languages don't have this problem because they can rewrite the layouts however they want and based on real time profiling. AoT languages either have to run first and compile with the profiling information to have hope of doing this, or else the author has to break out perf and VTune and do it manually. Both are difficult and time consuming, which is why it doesn't happen much in practice. A minor example of this is Protobuf, where the IDL translates to the same field layout in C++. Could the fields be packed into a more cache friendly layout? Sure, but only on a case-by-case, application-by-application basis. It doesn't work in general, even though proto authors don't care about the precise layout.
- vardump 8y ago
  > JIT'd languages don't have this problem because they can rewrite the layouts however they want and based on real time profiling. Any examples of this? > AoT languages either have to run first and compile with the profiling information to have hope of doing this... What prevents one from JITting AoT compiled binaries?
  - pjmlp 8y ago
    > Any examples of this? Yes, Java and .NET. Some JVM implementations, e.g. IBM J9, can save profile information between runs and use it for speeding up code generation, based on the complete history of the executable. Google also started doing this when they rebooted ART into a mix of interpreter written in Assembly/JIT/AOT runtime. Starting with Android P they are even uploading profile information back to the store, so that similar devices can reach optimized AOT compiled code faster. .NET Framework allows some control to save PGO data, and variants like that WinRT/UWP cloud compiler take it further. > What prevents one from JITting AoT compiled binaries? It relatively complex, versus just starting from bytecodes. Hence why mainframes always took the approach of either JITing at installation time, or using microcoded CPUs. Like on Burroughs, Xerox PARC and IBM cases.

zamalek 8y ago

> Once the heap size gets larger than the threshold under which compressed references are working, a surprising thing happens: references suddenly become uncompressed Am I correct in assuming that this is made possible because Hotspot can re-JIT? So the GC/Allocator just warns the JITter, "hey, all of your Oops [flags] code is no longer valid," resulting in a re-JIT of everything? Or a prologue in every method that checks the Oops flags (and triggers a re-JIT)?

deadc0de 8y ago
No, compressed oops mode doesn't change at runtime. That's pretty hard to pull off. You'd need to change object layout dynamically and unless you want to reformat the whole heap you'd need tags or regions with different pointer rules. Any of these techniques would have a very substantial runtime overhead.
xxs 8y ago
nope, CompressedOps is an old technique that depends on the max heap. less than 32GB, 64bit -> compressedOps

colechristensen 8y ago

The short version is that you are often better off running multiple instances of a JVM application with a heap <32GB when you have the choice between scaling up and scaling out.

speedplane 8y ago
> The short version is that you are often better off running multiple instances of a JVM application with a heap <32GB when you have the choice between scaling up and scaling out. We're in an awkward time right now, where many servers have more than 32GB of RAM, but not that much more. Once it becomes routine for servers to have 1TB of memory, this will no longer be much of an issue.
cbsmith 8y ago
Depends on the use case. Sometimes compressed oops is counter productive, and it really isn't a significant piece of overhead if you are say... doing matrix multiplication with very large arrays.
vbezhenar 8y ago
I wonder if using 16 bytes as an alignment would help in those corner cases. Some objects would use up to 33% more space (24 bytes -> 32 bytes), but I think that for most objects the increase will not be very noticeable and one extra bit would allow to use up to 64 GB of RAM with compressed pointers.
callmeal 8y ago
That depends on your memory usage. There is an awkward range (32GB - 48GB) where the increase in heap size does not give you a proportional increase in available memory because of the doubling of references. That is where you're better off running multiple instances. But for heaps bigger than 48GB running a single jvm is more efficient.