7 ms·

There's still a memory tradeoff, some due to GC, some due to Java (lots of runtime reflection...). Guessing 2-4x.

masklinn 5y ago
The GC memory overhead affects all languages with a GC more advanced than refcounting. It certainly does affect Go as well.
kaba0 5y ago
Also, throughput. But latency and throughput are almost universally opposite ends of the same axis — that’s why it’s great that Java allows for choosing a GC implementation.
- jatone 5y ago
  you can fix throughput problems by adding compute resources, you can't fix latency issues. I'll always pick a GC that optimizes latency over throughput. its easier to maintain the software.
marginalia_nu 5y ago
This is a bit of a tangent, but you can get into situations where Java's memory-overhead becomes pretty untenable. I was in a situation of having to keep track of ~1 billion short strings of a median length of maybe 7 characters. In terms of just data, that should clock in at about 10 Gb; in practice it was closer to 24 Gb. I tried going with just byte[]-instances instead, which didn't help a lot. Using long byte[]-instances and indexing those doesn't help as much as you'd think because they get sliced up into small objects behind the scenes. I ended up memory mapping blocks of memory and basically implementing my own append-only allocator.
- hashmash 5y ago
  The Lilliput project aims to address this: https://wiki.openjdk.java.net/display/lilliput https://wiki.openjdk.java.net/display/lilliput
- jeeeb 5y ago
  FWIW. This would probably present a challenge in most (all?) languages. For example in libc++ due to SSO an std::string has a minimum size of 24 bytes. For a billion strings less than 15 chars (+ the null byte) that gets you to 24GB, and that’s optimistically assuming each string is allocated in place. I doubt heap allocated char* would do much better either. Just having a billion 8 byte pointers eats a lot of memory. You’d really need some sort of string packing scheme similar to what you did in Java.
  - marginalia_nu 5y ago
    It's a lot easier to build custom allocators in C++ though. For one, Java has a maximum mmap-size of 2 Gb, and as a cherry on top of that turd, you have no control over their lifecycle. The language is very clearly not designed for this type of work, and if you try to make it do it anyway, it fights you every step of the way.
    - kasperni 5y ago
      The foreign memory API which is currently incubating should help with most of these limitations: https://openjdk.java.net/jeps/419 https://openjdk.java.net/jeps/419
      - native_samples 5y ago
        Right - specifically they invented a way to make closing an mmapped segment safe and fast. The reason you can't officially (without private APIs) unmap something in current Java is because if you did then other threads would segfault, and "no segfaults" is kind a defining characteristic of Java. The new API fixes this using more VM magic, so closing a mapping from one thread will cause other threads to throw exceptions, but this is done without sacrificing performance (it doesn't require checking a status every time you read from memory).
    - _old_dude_ 5y ago
      It has nothing to do with the language but with the API / VM capabilities. Both those restrictions are fixed, granted that this is not yet in the offical API, it's an incubation module (the equivalent of from __future__ of Python). https://docs.oracle.com/en/java/javase/17/docs/api/jdk.incubator.foreign/jdk/incubator/foreign/MemorySegment.html#mapFile(java.nio.file.Path,long,long,java.nio.channels.FileChannel.MapMode,jdk.incubator.foreign.ResourceScope) https://docs.oracle.com/en/java/javase/17/docs/api/jdk.incub...
- CyberDildonics 5y ago
  I would never write something like this in java, but to be fair, a program shouldn't be written like this in the first place. If you "need" a billion strings in memory and you didn't design for that with something that would scale better, you messed up a long time ago.
  - marginalia_nu 5y ago
    Huh, I didn't really have a problem solving the problem as it came up. Like many scaling problems, it wasn't a problem until it was. Then I fixed it. Now I have a solution that can deal with ten times as many strings as before. If I grow out of that one, I'll come up with a better design. I could have gotten 10 times as much hardware instead, but that would be an incredible waste of money compared to just spending a few days writing more hardware-efficient code.
    - sitkack 5y ago
      I have addressed similar problems using a typed array of contiguous memory and another array of lengths.
      - marginalia_nu 5y ago
        I did try exactly that, but the GC overhead was still prohibitively high for my use case, made because big arrays in practice are composed from shorter non-contiguous arrays to make garbage collection even remotely possible.
        sitkack 5y ago
        Hmmm, if the typed array only contains primitives, the GC isn't looking inside the array. If the thing pointed to, can never contain a pointer, it won't need to be scanned.
- lmm 5y ago
  A factor of 2.4 is extremely unlikely to be what makes the difference between a program being viable and not. You have to be in an extremely fine-tuned situation for that to make the difference, and unless your usage level is unusually static it's probably only going to make the difference for a few weeks or months.
  - vardump 5y ago
    Playing a devil's advocate here: a use case might already be hitting the maximum per server RAM. A factor of 2.4 might mean 2.4 times more hardware. You might need 240 servers instead of 100.
    - lmm 5y ago
      Sure. And? Everything that you need to do to manage 240 servers, you need to do to manage 100 servers. Like I said, if you're in the scaling stage then you'll be getting to 240 in a few months either way, so you need to be ready for it. 2.4x higher operating expenses matter eventually, when you're a mature organisation, but if the software is under active development you almost certainly have higher priorities.
  - marginalia_nu 5y ago
    So for some context, I'm running a search engine on consumer hardware. Coaxing power out of limited hardware is sort of my bread and butter. How many of these strings I can keep in memory limits how many documents I can index. Doubling that number is quite significant. This number is orthogonal to how many searches I can serve, that is already a solved problem (through similar craftiness). Multiple searches per second sustained load is fine. The search engine has held up to the hacker news front page, which is something many supposedly non-interactive sites struggle with. This is a non-profit operation, so unless someone just decides to pay me a months salary or two, I'm not able to meaningfully increase the hardware. I'm stuck with what I've got and will make the most of it.
    - lmm 5y ago
      The time of someone who can do that kind of crafty tuning is almost certainly worth more than the hardware. If nothing else, I'd think having you do a week or two of consultancy and buy more hardware with the proceeds would be more efficient in terms of getting things done. (I appreciate that nonprofits have a lot of particular constraints that mean that that's probably not a practical way forward).

The GC memory overhead affects all languages with a GC more advanced than refcounting. It certainly does affect Go as well.

kaba0 5y ago

Also, throughput. But latency and throughput are almost universally opposite ends of the same axis — that’s why it’s great that Java allows for choosing a GC implementation.

jatone 5y ago
you can fix throughput problems by adding compute resources, you can't fix latency issues. I'll always pick a GC that optimizes latency over throughput. its easier to maintain the software.

marginalia_nu 5y ago

This is a bit of a tangent, but you can get into situations where Java's memory-overhead becomes pretty untenable. I was in a situation of having to keep track of ~1 billion short strings of a median length of maybe 7 characters. In terms of just data, that should clock in at about 10 Gb; in practice it was closer to 24 Gb. I tried going with just byte[]-instances instead, which didn't help a lot. Using long byte[]-instances and indexing those doesn't help as much as you'd think because they get sliced up into small objects behind the scenes. I ended up memory mapping blocks of memory and basically implementing my own append-only allocator.

hashmash 5y ago
The Lilliput project aims to address this: https://wiki.openjdk.java.net/display/lilliput https://wiki.openjdk.java.net/display/lilliput
jeeeb 5y ago
FWIW. This would probably present a challenge in most (all?) languages. For example in libc++ due to SSO an std::string has a minimum size of 24 bytes. For a billion strings less than 15 chars (+ the null byte) that gets you to 24GB, and that’s optimistically assuming each string is allocated in place. I doubt heap allocated char* would do much better either. Just having a billion 8 byte pointers eats a lot of memory. You’d really need some sort of string packing scheme similar to what you did in Java.
- marginalia_nu 5y ago
  It's a lot easier to build custom allocators in C++ though. For one, Java has a maximum mmap-size of 2 Gb, and as a cherry on top of that turd, you have no control over their lifecycle. The language is very clearly not designed for this type of work, and if you try to make it do it anyway, it fights you every step of the way.
  - kasperni 5y ago
    The foreign memory API which is currently incubating should help with most of these limitations: https://openjdk.java.net/jeps/419 https://openjdk.java.net/jeps/419
    - native_samples 5y ago
      Right - specifically they invented a way to make closing an mmapped segment safe and fast. The reason you can't officially (without private APIs) unmap something in current Java is because if you did then other threads would segfault, and "no segfaults" is kind a defining characteristic of Java. The new API fixes this using more VM magic, so closing a mapping from one thread will cause other threads to throw exceptions, but this is done without sacrificing performance (it doesn't require checking a status every time you read from memory).
  - _old_dude_ 5y ago
    It has nothing to do with the language but with the API / VM capabilities. Both those restrictions are fixed, granted that this is not yet in the offical API, it's an incubation module (the equivalent of from __future__ of Python). https://docs.oracle.com/en/java/javase/17/docs/api/jdk.incubator.foreign/jdk/incubator/foreign/MemorySegment.html#mapFile(java.nio.file.Path,long,long,java.nio.channels.FileChannel.MapMode,jdk.incubator.foreign.ResourceScope) https://docs.oracle.com/en/java/javase/17/docs/api/jdk.incub...
CyberDildonics 5y ago
I would never write something like this in java, but to be fair, a program shouldn't be written like this in the first place. If you "need" a billion strings in memory and you didn't design for that with something that would scale better, you messed up a long time ago.
- marginalia_nu 5y ago
  Huh, I didn't really have a problem solving the problem as it came up. Like many scaling problems, it wasn't a problem until it was. Then I fixed it. Now I have a solution that can deal with ten times as many strings as before. If I grow out of that one, I'll come up with a better design. I could have gotten 10 times as much hardware instead, but that would be an incredible waste of money compared to just spending a few days writing more hardware-efficient code.
  - sitkack 5y ago
    I have addressed similar problems using a typed array of contiguous memory and another array of lengths.
    - marginalia_nu 5y ago
      I did try exactly that, but the GC overhead was still prohibitively high for my use case, made because big arrays in practice are composed from shorter non-contiguous arrays to make garbage collection even remotely possible.
      - sitkack 5y ago
        Hmmm, if the typed array only contains primitives, the GC isn't looking inside the array. If the thing pointed to, can never contain a pointer, it won't need to be scanned.
lmm 5y ago
A factor of 2.4 is extremely unlikely to be what makes the difference between a program being viable and not. You have to be in an extremely fine-tuned situation for that to make the difference, and unless your usage level is unusually static it's probably only going to make the difference for a few weeks or months.
- vardump 5y ago
  Playing a devil's advocate here: a use case might already be hitting the maximum per server RAM. A factor of 2.4 might mean 2.4 times more hardware. You might need 240 servers instead of 100.
  - lmm 5y ago
    Sure. And? Everything that you need to do to manage 240 servers, you need to do to manage 100 servers. Like I said, if you're in the scaling stage then you'll be getting to 240 in a few months either way, so you need to be ready for it. 2.4x higher operating expenses matter eventually, when you're a mature organisation, but if the software is under active development you almost certainly have higher priorities.
- marginalia_nu 5y ago
  So for some context, I'm running a search engine on consumer hardware. Coaxing power out of limited hardware is sort of my bread and butter. How many of these strings I can keep in memory limits how many documents I can index. Doubling that number is quite significant. This number is orthogonal to how many searches I can serve, that is already a solved problem (through similar craftiness). Multiple searches per second sustained load is fine. The search engine has held up to the hacker news front page, which is something many supposedly non-interactive sites struggle with. This is a non-profit operation, so unless someone just decides to pay me a months salary or two, I'm not able to meaningfully increase the hardware. I'm stuck with what I've got and will make the most of it.
  - lmm 5y ago
    The time of someone who can do that kind of crafty tuning is almost certainly worth more than the hardware. If nothing else, I'd think having you do a week or two of consultancy and buy more hardware with the proceeds would be more efficient in terms of getting things done. (I appreciate that nonprofits have a lot of particular constraints that mean that that's probably not a practical way forward).