3 ms·
> * Try implementing an on-disk hash table in Python and you're stuck manually packing ints and longs in an out of arrays. In C++ it could a simple template use
by noctune 6y ago
> * Try implementing an on-disk hash table in Python and you're stuck manually packing ints and longs in an out of arrays. In C++ it could a simple template used with a memory-mapped file.
This is only simpler if you don't need to handle read failures. If you want to handle read failures, memory mapping can quickly become way more complicated. You have to install signal handlers and do a complicated dance to correctly handle not having data in some range.
- eps 6y agoIndeed. Memory mapped files are the bubble sort 2.0. They look simple and approachable on the surface, lead to an elegant code and seem to solve the problem in basic test cases, but in practice they have nasty scalability problems. In particular, trying to work with very large files leads to cache thrashing that affects the entire system, which makes mmaped files particularly unfit for implementing off-memory data structures. Moreover, the code has zero control over this behavior, because we ultimately over-allocate memory and outsource swapping details to the OS.