3 ms·
Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram. It’ll be slower, perhaps by a lot, but most
by ninkendo 2y ago
Right, which is why you can mmap way more data than you have ram, and treat it as though you do have that much ram.
It’ll be slower, perhaps by a lot, but most “big data” stuff is already so god damned slow that mmap probably still beats it, while being immeasurably simpler and cheaper.
- marginalia_nu 2y agoReally depends on the shape of the data. mmap can be suboptimal in many cases. For CSV it flat out doesn't matter what you do since the format is so inefficient and needs to be read start to finish, but something like parquet probably benefits from explicit read syscalls, since it's block based and highly structured, where you can predict the read patterns much better than the kernel can.