5 ms·

You jest, but think about how unbelievably painful it'd be to write a program that uses >1TB of RAM in C++ .... any bug that causes a segfault, div by zero, or

You jest, but think about how unbelievably painful it'd be to write a program that uses >1TB of RAM in C++ .... any bug that causes a segfault, div by zero, or really any kind of crash at all would mean you'd have to reload the entire dataset into RAM from scratch. That's gonna take a while no matter what. You could work around it by using shared memory regions and the like but then you're doing a lot of extra work. With a managed language and a bit of care around exception handling, you can write code that's pretty much invincible without much effort because you can't corrupt things arbitrarily. Also, depending on the dataset in question you might find that things shrink. The latest HotSpots can deduplicate strings in memory as they garbage collect. If your dataset has a lot of repeated strings then you effectively get an interning scheme for free. I don't know if G1 can really work well with over 1TB of heap, though. I've only ever heard of it going up to a few hundred gigabytes.

Kristine1975 10y ago
>With a managed language and a bit of care around exception handling, you can write code that's pretty much invincible without much effort because you can't corrupt things arbitrarily. The JVM has crashed on me in the past (as in hard crash, not a Java exception). Less often than the C++ programs I write do? Yes, but I of course I wouldn't test a program on a 1TB dataset before ironing out all the kinks. >The latest HotSpots can deduplicate strings in memory as they garbage collect Obviously when working with huge datasets I would implement some kind of string deduplication myself. Most likely even a special string class an memory allocation scheme optimized for write-once, read-many access and cache friendliness. Or I would use memory mapping for the input file and let the OS's virtual memory management sort it out.
tosseraccount 10y ago
Use shared memory.
0xfaded 10y ago
mmap is not "a lot of extra work".
- icedchai 10y ago
  Hah. You'd be surprised how few developers actually know that shared memory exists.
- sievebrain 10y ago
  You're assuming the dataset is a file you can just mmap there. If it's more complex (built up from a network feed or something) then yes, there is more work involved than just an mmap call, coordinating shm segments is not zero complexity. But the real impact is if you want to be mutating that data set. The default behaviour of "tear down the entire process on error" can of course be worked around even if you ignore data corruption errors, but not having to do things by hand is the point of managed runtimes in the first place.

Kristine1975 10y ago

>With a managed language and a bit of care around exception handling, you can write code that's pretty much invincible without much effort because you can't corrupt things arbitrarily. The JVM has crashed on me in the past (as in hard crash, not a Java exception). Less often than the C++ programs I write do? Yes, but I of course I wouldn't test a program on a 1TB dataset before ironing out all the kinks. >The latest HotSpots can deduplicate strings in memory as they garbage collect Obviously when working with huge datasets I would implement some kind of string deduplication myself. Most likely even a special string class an memory allocation scheme optimized for write-once, read-many access and cache friendliness. Or I would use memory mapping for the input file and let the OS's virtual memory management sort it out.

tosseraccount 10y ago

Use shared memory.

0xfaded 10y ago

mmap is not "a lot of extra work".

icedchai 10y ago
Hah. You'd be surprised how few developers actually know that shared memory exists.
sievebrain 10y ago
You're assuming the dataset is a file you can just mmap there. If it's more complex (built up from a network feed or something) then yes, there is more work involved than just an mmap call, coordinating shm segments is not zero complexity. But the real impact is if you want to be mutating that data set. The default behaviour of "tear down the entire process on error" can of course be worked around even if you ignore data corruption errors, but not having to do things by hand is the point of managed runtimes in the first place.