4 ms·

Thank you for the answer. Did you ever do or would it be possible for you to do a benchmark on relatively large corpus of how fast can you count, say 10 millio

Thank you for the answer. Did you ever do or would it be possible for you to do a benchmark on relatively large corpus of how fast can you count, say 10 million words? Or 100 million? I'm really curious about how well does this method work relatively to usual map-reduce on clusters. For comparison: single home PC does about 2.5 million word counts per minute using local files and mergesort instead of network. This is why I asked - 300 words per minute using using 6.5 million randomly available nodes vs 2.5 million words per minute using one home node - that looks like a huge waste. Would it be possible to do more fair comparison?

jhuckestein 16y ago
Yes, we could benchmark that. Most likely our framework would be orders of magnitude slower. There's a couple of reasons for this: - Data transfer is slow. Streaming the data in is not possible on all browsers. - There is a very high sunk cost for each single computation, that is multiple times bigger than the time it takes for the average map/reduce function to run. This constant overhead per step is hard to minimize. - A single map or reduce computation is further chunked down into multiple iterations of the same function. This is to avoid the "slow script" warning that browser show for long-running tasks. This technique is necessary and further adds overhead to the computation. As a result our framework might not be well suited for many traditional MapReduce scenarios like word count. More suited tasks are ones that have a lot of overhead even if I ran them on my own cluster. This could be any tasks that involves pulling in data from a third party that is not local to the computation. For example counting word frequencies in all wikipedia pages will be vastly faster on our cluster than it could ever be on a single machine, even if we have only few workers. We still need to figure many things out. The opportunities here are the availability of idle processing power and the possibility to sandbox computations in browsers. We need to find out exactly what the best way is to exploit those opportunities.

jhuckestein 16y ago

Yes, we could benchmark that. Most likely our framework would be orders of magnitude slower. There's a couple of reasons for this: - Data transfer is slow. Streaming the data in is not possible on all browsers. - There is a very high sunk cost for each single computation, that is multiple times bigger than the time it takes for the average map/reduce function to run. This constant overhead per step is hard to minimize. - A single map or reduce computation is further chunked down into multiple iterations of the same function. This is to avoid the "slow script" warning that browser show for long-running tasks. This technique is necessary and further adds overhead to the computation. As a result our framework might not be well suited for many traditional MapReduce scenarios like word count. More suited tasks are ones that have a lot of overhead even if I ran them on my own cluster. This could be any tasks that involves pulling in data from a third party that is not local to the computation. For example counting word frequencies in all wikipedia pages will be vastly faster on our cluster than it could ever be on a single machine, even if we have only few workers. We still need to figure many things out. The opportunities here are the availability of idle processing power and the possibility to sandbox computations in browsers. We need to find out exactly what the best way is to exploit those opportunities.