4 ms·

I spun this our of some work I was doing for an NLP related thesis. Large scale text processing where there's already a shared file system is a perfect fit for

I spun this our of some work I was doing for an NLP related thesis. Large scale text processing where there's already a shared file system is a perfect fit for mincemeat.py. I basically wanted something that was much, much easier to develop for, setup, and run than Hadoop. I've heard many academic researchers complain that although they had a algorithm that would fit neatly into MapReduce, they didn't want to bother setting up Hadoop and importing all of their data for a process that would only get run a few times (and that was already coded in Python).

adamtj 16y ago
That makes sense. I hadn't considered academia. I was thinking that if there is a business need for mapreduce, then the solution is probably going to become long-term infrastructure, in which case the cost of setting up hadoop can be amortized. But it makes sense to me that researchers would have a higher proportion of one-off problems, and in that situation, a simpler python solution can be a big win. Thanks.
StavrosK 16y ago
The only downside for me is that not only do you have to find a massively parallel problem to work on, but the computation function also needs to be much slower than network latency. With network latency being in the ms range, the algorithm solving problem needs to be really slow to benefit.
- leif 16y ago
  Not necessarily. It can also just be a vast amount of data, in which case bandwidth (which is generally pretty good), not latency, is your limiting factor with mapreduce. Also, you only need one part of your infrastructure to require mapreduce's parallelism in order to argue for using it across the board. If you have simpler problems to solve, you may as well solve them with mapreduce since you'll be thinking in that computation model anyway, and you can easier use the results later in a computation that may require mapreduce.
  - StavrosK 16y ago
    Your problem still has to have the property that loading and processing the dataset needs to be slower than sending it over the network and getting the result back, though...
- deleted 16y ago
  [deleted]

adamtj 16y ago

That makes sense. I hadn't considered academia. I was thinking that if there is a business need for mapreduce, then the solution is probably going to become long-term infrastructure, in which case the cost of setting up hadoop can be amortized. But it makes sense to me that researchers would have a higher proportion of one-off problems, and in that situation, a simpler python solution can be a big win. Thanks.

StavrosK 16y ago

The only downside for me is that not only do you have to find a massively parallel problem to work on, but the computation function also needs to be much slower than network latency. With network latency being in the ms range, the algorithm solving problem needs to be really slow to benefit.

leif 16y ago
Not necessarily. It can also just be a vast amount of data, in which case bandwidth (which is generally pretty good), not latency, is your limiting factor with mapreduce. Also, you only need one part of your infrastructure to require mapreduce's parallelism in order to argue for using it across the board. If you have simpler problems to solve, you may as well solve them with mapreduce since you'll be thinking in that computation model anyway, and you can easier use the results later in a computation that may require mapreduce.
- StavrosK 16y ago
  Your problem still has to have the property that loading and processing the dataset needs to be slower than sending it over the network and getting the result back, though...
deleted 16y ago
[deleted]