4 ms·
This can't come soon enough as far as I'm concerned. Being constrained to only memory map 2 Gb chunks of memory, with no control over their lifecycle, is extrem
by marginalia_nu 5y ago
This can't come soon enough as far as I'm concerned. Being constrained to only memory map 2 Gb chunks of memory, with no control over their lifecycle, is extremely cumbersome and arguably one of the biggest problem with Java right now.
- lichtenberger 5y agoI'll have to check how to call `madvise` to set MADV_RANDOM as index accesses in my use case are rather random, though. But the 2Gb chunks at most have been a really big issue. Luckily, the Foreign Memory API is getting to it's final version I think. The latest changes aren't that big IMHO :-)
- marginalia_nu 5y agoYeah, tell me about it. I've built a search engine index including an on-disk hash table off MappedByteBuffers. Like chopping down a tree with a pasta sieve.
- zylepe 5y agoI recently found this approach: https://github.com/onthegomap/planetiler/blob/main/planetiler-core/src/main/java/com/onthegomap/planetiler/util/Madvise.java https://github.com/onthegomap/planetiler/blob/main/planetile... but I’m hoping this api will eventually let you do that without an external dependency or writing/compiling C connectors.
- exabrial 5y agoIs that the limitation of arrays being addressed by an int instead of a long in java?
- marginalia_nu 5y agoThat is a part of it, but large arrays are also sliced up behind the scenes rather than kept as one contiguous segment of memory. This causes a fair bit of memory overhead, as there needs to be GC book-keeping for all the slices.
- samus 5y agoThis is super interesting to know! This means that future implementations of Frozen Arrays could optimize copy-on-write operations by sharing such chunks!