4 ms·

I think the real question is "Is MapReduce Our Only Hope?" MapReduce has a lot of limitations. It doesn't have a query language, instead, you need to figure o

I think the real question is "Is MapReduce Our Only Hope?" MapReduce has a lot of limitations. It doesn't have a query language, instead, you need to figure out the sequence of map and reduce steps and implement those in your favourite low level language yourself. And it can't do efficient joins. That means you need to visit each and every row for each and every map-reduce stage. There's no b-tree or other "lookup structure". And it's a batch based framework, which means if you add 1% more data, you have to re-analyze the entire data set, rather than update previous results with the new 1%. Disclaimer: I work at Endeca, which is about to launch Latitude, an Enterprise platform for big data analysis. But (a) I work in Engineering, not sales or marketing, so I spend my time thinking about the advantages and disadvantages of various technologies, rather than how to sell them, and (b) I'm an actual human being who has independent thoughts.

cdavid 15y ago
I don't understand the claim that adding new data force you to recompute everything. What requires re-computation will depend on the algorithm, but for most simples cases I can think of, at least the map part will not require computation if you record its result. I believe that's how couchdb view work, for example
- yummyfajitas 15y ago
  Hadoop is a fairly low level framework. So it requires the programmer to write logic to incrementally calculate the map (i.e., input_data_may -> map_results_may, repeat for june). Similarly, it's up to the programmer to write a reducer which allows incremental additions of data. And even then, you still need to make a pass over all the data.
  - cdavid 15y ago
    That's a limitation of hadoop, though, not the MR idea by itself.
    - yummyfajitas 15y ago
      The limitation on the map end is only a limitation for hadoop. But the limit on the reducer is fundamental. Some reduce functions are not associative, and some don't even have type [T x U] -> T x U. In those cases, there is nothing to be done but redo the reduce.
      - cdavid 15y ago
        Indeed, reduce is the difficult part. OTOH, I think this limitation is seen in many algorithms at a fairly fundamental level, and not just an artefact of MR. The only alternative framework I can think of for dealing with really large datasets in a distributed manner is sampling-based methods, with one-pass algorithms (or mostly one pass algorithm).

cdavid 15y ago

I don't understand the claim that adding new data force you to recompute everything. What requires re-computation will depend on the algorithm, but for most simples cases I can think of, at least the map part will not require computation if you record its result. I believe that's how couchdb view work, for example

yummyfajitas 15y ago
Hadoop is a fairly low level framework. So it requires the programmer to write logic to incrementally calculate the map (i.e., input_data_may -> map_results_may, repeat for june). Similarly, it's up to the programmer to write a reducer which allows incremental additions of data. And even then, you still need to make a pass over all the data.
- cdavid 15y ago
  That's a limitation of hadoop, though, not the MR idea by itself.
  - yummyfajitas 15y ago
    The limitation on the map end is only a limitation for hadoop. But the limit on the reducer is fundamental. Some reduce functions are not associative, and some don't even have type [T x U] -> T x U. In those cases, there is nothing to be done but redo the reduce.
    - cdavid 15y ago
      Indeed, reduce is the difficult part. OTOH, I think this limitation is seen in many algorithms at a fairly fundamental level, and not just an artefact of MR. The only alternative framework I can think of for dealing with really large datasets in a distributed manner is sampling-based methods, with one-pass algorithms (or mostly one pass algorithm).