22 ms·
What sort of map-reduce space needs fast random I/O? The only things which come to mind are disguised table joins, where the right solution is to use sorting i
by cperciva 14y ago
What sort of map-reduce space needs fast random I/O? The only things which come to mind are disguised table joins, where the right solution is to use sorting instead.
(Not trying to be argumentative, I'm just trying to figure out what you have in mind.)
- jedberg 14y agoIt's not just random I/O that's fast, but sequential is a lot faster too. But what I'm thinking of is the tokenization of a large corpus of text into bigrams and trigrams for example.
- bravura 14y agoAdmittedly, it would be fun to pull in the Common Crawl and process it. Note that there's a code contest for Common Crawl that recently was announced: http://commoncrawl.org/first-ever-code-contest/ http://commoncrawl.org/first-ever-code-contest/
- cperciva 14y agoGood point, with all the focus on random I/O in the announcement I was forgetting about the sequential I/O performance benefit. Hmm, that reminds me, I have an interesting toy problem which is sequential I/O limited...