3 ms·
Some of k's magical powers come from mmap > That's something which should be marginally faster when it's faster, No. If used properly, it is crazy faster. Le
by beagle3 13y ago
Some of k's magical powers come from mmap
> That's something which should be marginally faster when it's faster,
No. If used properly, it is crazy faster.
Let's say what you need from the data is element no. 1821942 in the stream.
Hadoop standard: read & deserialize & discard 1821941 items (just so you can get to the 1821942nd item). read & deserialize and use one item.
K standard: (equivalent to): seek directly to element 1821942, read it and use it.
except .. K does it slightly faster than that, using mmap: It just accesses the location for this element in memory. If it is not in memory, then the operating system will arrange for it to appear in memory in a way that's usually more efficient than read. If it is already in memory, then it is exactly one memory read.
Now, if Hadoop & friends stored their data in such a way that you could do a random access read (and used a read syscall) then mmap would _still_ be about 1000 times faster than a read() call for access if the datum is already in the o/s buffers.
The magic is not that K can read stuff faster -- e.g., it can't read a hadoop stream faster than hadoop can. The magic is that K makes it easiest to just store your data in a random-access mmapable way, so that mapping the data back in takes ~10ms, and from then on, you only pay for the data you actually use without having to explicitly read it -- and usually, you pay less than you would have paid if you explicitly read it.
- justin66 13y ago> No. If used properly, it is crazy faster. I've been impressed with, for example, Symas's lightning mdb: http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/ However, it's not as if someone replaced my Toyota with a Formula 1 car. Properly used mmap is very nice but some people are getting carried away with the superlatives. I hope this doesn't come off the wrong way but I really wonder if the "crazy faster" amazingness you're referring to isn't a function of comparing K to Hadoop. Which, you know, might just be setting the bar way too low.
- beagle3 13y ago> Properly used mmap is very nice but some people are getting carried away with the superlatives. Well, it all depends on your access patterns. The bottom line is: in a modern (later than 1985 or so) Unix system, mmap is hardware based virtual memory; read() and write() can give you software based virtual memory. Who do you think is going to have better performance? Compare and contrast: a) mmap an 8GB file into 64-bit address space, start using immediately. only stuff you actually use gets read from disk. b) using only read() access, read what you need, manage caching to avoid syscalls, etc. It is unlikely b) can be faster at all, and it will only be competitive if the granularity of reading + the cost of cache management is low. In cases where either granularity or cache management is expensive (which is common), mmap is likely to win big time.
- cmccabe 13y agoNow, if Hadoop & friends stored their data in such a way that you could do a random access read (and used a read syscall) then mmap would _still_ be about 1000 times faster than a read() call for access if the datum is already in the o/s buffers. Yeah, it's too bad this method doesn't exist: http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#read(long http://hadoop.apache.org/docs/current/api/org/apache/hadoop/..., byte[], int, int) It's too bad that this doesn't exist: http://www.parquet.io http://www.parquet.io It's too bad that this doesn't exist either: https://issues.apache.org/jira/browse/HDFS-4953 https://issues.apache.org/jira/browse/HDFS-4953 You are amazingly, astoundingly, misinformed. It's no surprise that you are wrong about mmap as well. It's only a performance advantage when the PTE (page table entries) are already populated. The PTEs may not be populated, even if the file region in question is resident in memory. And even then it's only about 2x, not "1000 times faster". There was no "serialization and deserialization" involved in our benchmarks, either.
- beagle3 13y ago> Yeah, it's too bad this method doesn't exist: http://hadoop.apache.org/docs/current/api/org/apache/hadoop/... http://hadoop.apache.org/docs/current/api/org/apache/hadoop/..., byte[], int, int) And how, if you may help me, do you access object #n in the stream (for non-trivial n)? Not byte n in the stream, which this appears to give you - but object record #n ? > It's too bad that this doesn't exist either: https://issues.apache.org/jira/browse/HDFS-4953 https://issues.apache.org/jira/browse/HDFS-4953 Your patch is two weeks old, and you expect anyone to be aware of it? > You are amazingly, astoundingly, misinformed. No, I have inherited a hadoop project from people who did it "by the book". And then rewrote it in C with mmap to gain a 20-1000 fold improvement in speed depending on workflow. That had been in 2011, but I'd be surprised if things changed so significantly since then. > It's no surprise that you are wrong about mmap as well. It's only a performance advantage when the PTE (page table entries) are already populated. The PTEs may not be populated, even if the file region in question is resident in memory. And even then it's only about 2x, not "1000 times faster". There was no "serialization and deserialization" involved in our benchmarks, either. Are you using hadoop map reduce, or your own HDFS code? Because every single non-trivial map-reduce jobs that I've met serializes and deserializes objects, spends 90% of its time on I/O, serialization, network, compression and stuff - and that's actually the recommended way to do so. Which is why, incidentally, spark manages to be so much faster (100 times claimed - I've heard of >70) than hadoop - it keeps stuff in memory. It's no surprise that you are wrong about mmap as well - your description is correct if and only if you are going to read the entire file. Which is standard for hadoop map reduce, but actually not that common in a well designed program, and certainly not common in the K world. And you certainly did not understand the 2x vs 1000 times faster comment, which stems apparently from your (faulty) understanding that everything has to be read. In an mmaped file, if you need one byte from the middle, you pay for reading one block from the middle exactly once. You do not need to pre-read everything, and you don't need to manage caching yourself. That's where the 1000x times come from. If you do read everything, it's not 1000x times. But you've possibly wasted 1000x times memory -- and 1000x times I/O to get to that byte in the middle that you need.