5 ms·
I have used Hadoop; it is miserably slow (as in 10 to 1000 times slower AND a lot harder to work with) then the comparable K program. If mmap doesn't bring you
by beagle3 13y ago
I have used Hadoop; it is miserably slow (as in 10 to 1000 times slower AND a lot harder to work with) then the comparable K program.
If mmap doesn't bring you significant gains, you are doing stuff wrong. E.g. You serialize objects. Don't.
Hadoop is horrible. I've heard Spark manages reasonable performance (subject to the shackles that Hadoop compatibility entails), but haven't used it.
If you think SQL is "more powerful" than APL... Qualify that. Because e.g. APL is Turing complete, whereas SQL isn't.
- technolem 13y agoAs a sidenote, SQL is: http://stackoverflow.com/questions/900055/is-sql-or-even-tsql-turing-complete http://stackoverflow.com/questions/900055/is-sql-or-even-tsq.... Not that you'd even want to use it as such.
- beagle3 13y ago:) a corollary of Greenspun's tenth rule: every language spec will be revised at least until it is possible to implement a slow, incomplete version of Common Lisp inside it.
- justin66 13y agoWhy are people attributing k's magical powers to mmap? That's something which should be marginally faster when it's faster, if every other product that uses one or the other can be used as an example.
- beagle3 13y agoSome of k's magical powers come from mmap > That's something which should be marginally faster when it's faster, No. If used properly, it is crazy faster. Let's say what you need from the data is element no. 1821942 in the stream. Hadoop standard: read & deserialize & discard 1821941 items (just so you can get to the 1821942nd item). read & deserialize and use one item. K standard: (equivalent to): seek directly to element 1821942, read it and use it. except .. K does it slightly faster than that, using mmap: It just accesses the location for this element in memory. If it is not in memory, then the operating system will arrange for it to appear in memory in a way that's usually more efficient than read. If it is already in memory, then it is exactly one memory read. Now, if Hadoop & friends stored their data in such a way that you could do a random access read (and used a read syscall) then mmap would _still_ be about 1000 times faster than a read() call for access if the datum is already in the o/s buffers. The magic is not that K can read stuff faster -- e.g., it can't read a hadoop stream faster than hadoop can. The magic is that K makes it easiest to just store your data in a random-access mmapable way, so that mapping the data back in takes ~10ms, and from then on, you only pay for the data you actually use without having to explicitly read it -- and usually, you pay less than you would have paid if you explicitly read it.
- justin66 13y ago> No. If used properly, it is crazy faster. I've been impressed with, for example, Symas's lightning mdb: http://symas.com/mdb/microbench/ http://symas.com/mdb/microbench/ However, it's not as if someone replaced my Toyota with a Formula 1 car. Properly used mmap is very nice but some people are getting carried away with the superlatives. I hope this doesn't come off the wrong way but I really wonder if the "crazy faster" amazingness you're referring to isn't a function of comparing K to Hadoop. Which, you know, might just be setting the bar way too low.
- beagle3 13y ago> Properly used mmap is very nice but some people are getting carried away with the superlatives. Well, it all depends on your access patterns. The bottom line is: in a modern (later than 1985 or so) Unix system, mmap is hardware based virtual memory; read() and write() can give you software based virtual memory. Who do you think is going to have better performance? Compare and contrast: a) mmap an 8GB file into 64-bit address space, start using immediately. only stuff you actually use gets read from disk. b) using only read() access, read what you need, manage caching to avoid syscalls, etc. It is unlikely b) can be faster at all, and it will only be competitive if the granularity of reading + the cost of cache management is low. In cases where either granularity or cache management is expensive (which is common), mmap is likely to win big time.
- cmccabe 13y agoNow, if Hadoop & friends stored their data in such a way that you could do a random access read (and used a read syscall) then mmap would _still_ be about 1000 times faster than a read() call for access if the datum is already in the o/s buffers. Yeah, it's too bad this method doesn't exist: http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html#read(long http://hadoop.apache.org/docs/current/api/org/apache/hadoop/..., byte[], int, int) It's too bad that this doesn't exist: http://www.parquet.io http://www.parquet.io It's too bad that this doesn't exist either: https://issues.apache.org/jira/browse/HDFS-4953 https://issues.apache.org/jira/browse/HDFS-4953 You are amazingly, astoundingly, misinformed. It's no surprise that you are wrong about mmap as well. It's only a performance advantage when the PTE (page table entries) are already populated. The PTEs may not be populated, even if the file region in question is resident in memory. And even then it's only about 2x, not "1000 times faster". There was no "serialization and deserialization" involved in our benchmarks, either.