3 ms·
you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it.
by danudey 6y ago
you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it.
You get a pointer to the start of the range, and you know how large it is, so from there it's just pointer arithmetic and assignment. Easy peasy. Now you just have to make sure that you zero data when you're done with it, track how much of that memory is available or used (and which sections), and (optionally) implement your own defragmentation and garbage collection.
Unless you desperately need to specifically manage the memory layout of your data (e.g. for alignment purposes for massive data processing), none of this is worth the time or trouble to do for most projects, not only because it's a huge amount of work for little gain, but because you'll almost definitely screw it up and potentially corrupt your entire data set.
Note that, at least for the on-disk hashmap set, you absolutely can do this in python, so if you're interested in experimenting with this sort of data management, practice it there first.
A good place to start: https://github.com/luispedro/diskhash https://github.com/luispedro/diskhash
- barumi 6y ago> you can malloc() to get a memory range, or mmap() to get a disk-backed memory range with kernel-managed paging, and then you can do whatever you want with it. No need for that, as C++ provides placement new(). https://en.cppreference.com/w/cpp/language/new https://en.cppreference.com/w/cpp/language/new
- saagarjha 6y agoThis doesn’t really solve the same problem, even though the two often go hand-in-hand.
- gpderetta 6y agoIt is really not that much work and there are plenty of libs that can do it for you if you do not want to.