5 ms·
Testing Mincemeat.py MapReduce Scalability using GridCentric
- michaelfairley 16y agoWithout details on the task he's running and the setup of his cluster, it's hard to know how meaningful these results are. Using a variation of the example, I've seen roughly 80% improvement with each doubling of the cluster size (testing up to 100 workers).
- dscannell 16y agoHow big was the dataset that you saw the 80% improvement? Remember this test was done on a relatively small dataset, 28 MB, so it may have reached a diminishing point a lot faster than with a larger dataset.
- michaelfairley 16y agoAh yes, I missed the part about the size of the dataset. I'm running with a 3GB corpus of text.