2 ms·
> You access object #n by using an index And in a C (or K) mmapped file, you access the n's object by accessing a memory location. No index lookup, whether O(l
by beagle3 13y ago
> You access object #n by using an index
And in a C (or K) mmapped file, you access the n's object by accessing a memory location. No index lookup, whether O(log n) or a hash table.
I'm indeed not familiar with Parquet - Hadoop failed me so badly I do not feel like I need to keep up to date with every new thing. I might take another look when when it's been around for a while.
> Right now, the fastest way to copy a file is apparently by doing lots of ~8kB read/write pairs
That again discusses reading the whole file. When mmap delivers the huge gains is when you don't actually need the whole file - rather, when you need parts of it that you discover while reading. Which in many workloads is the case (or would be the case if you did not have to conform with some framework's constraints about the order and kind of I/O you can do).
mmap gives you hardware virtual memory and assistance for caching; Every other solution is a software virtual memory and implementation for caching. While it has its own constraints (e.g., you have to store data in a directly usable layout, rather than "disk frozen" layout), if you do that, things do work exceptionally well when you do.