5 ms·

A dfs isn't a requirement for map/reduce.

seanmcdirmid 12y ago
From http://en.wikipedia.org/wiki/MapReduce http://en.wikipedia.org/wiki/MapReduce: > MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. ... > The "MapReduce System" (also called "infrastructure" or "framework") orchestrates by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance. ... > The name MapReduce originally referred to the proprietary Google technology but has since been genericized. So it would be quite impossible to have a MapReduce system without distributed computing infrastructure; even if you were doing mapping and reducing, it wouldn't be MapReduce.
- supermatt 12y ago
  I see no mention of a distributed file system there. Local storage is not a requirement of distributed processing.
  - 1stop 12y ago
    How do you do distributed processing without a distributed filesystem? Do you mean you'd load the filesystem into memory and send it to the "processors"?
    - supermatt 12y ago
      The data could be stored on a network device, such as a file server or database, for example. It could indeed be local, but it needn't be distributed. In the example GP gave, the data could possibly have been stored in a database queried using segmentation via consistent hashing (a basic way to distribute jobs across a known number of workers).
      - srean 12y ago
        ...defeating the entire purpose: of large scale parallelism on commodity machines. OTOH if you have a way of achieving order 500X parallelism with a centralized commodity server or database, I would love to hear. EDIT @supermatt Ah I see, we differ in the definition then, to me it isnt bigdata/largescale unless it churns through big amounts of stored data. Bitcoin mining is no where in the ball park of this, its an append only log of solutions computed in parallel.
        supermatt 12y ago
        How on earth do you think bitcoin mining pools work (as an extremely trivial example). They coordinate ranges between a number of workers. The stored size of those ranges is miniscule in comparison to the data of the hashes between those ranges calculated on each 'miner'. These 'coordinators' absolutely work as a centralised 'commodity' storage server (or database) resource for 500x+ parallelism. 'Big Data' means 'Big Data', not 'Big Storage'. They are completely different things.
        seanmcdirmid 12y ago
        Big data doesn't mean big computation, it actually means big data on lots of disks across many nodes. They are completely different things. You might be into HPC, but that's not what Sanjay and Jeff did. HPC and big data loads are quite different.
        supermatt 12y ago
        The bitcoin example may be a bit oversimplified, and may indeed lean more towards HPC. The example was intended to illustrate data locality (as per the parent question), not the actual computation. Big Data may incorporate data from various 3rd party, remote, local, or even random sources. For example, testing whether URLs in a search engines index are currently available. This may be a map/reduce job, it may utilize a local source of urls, but it will also incorporate a remote check of the url. As I said a few links up: DFS is not a requirement for map/reduce.
    - walshemj 12y ago
      I our case the first stage synced up all the required file systems and applied all the required updates before kicking off the mapper stage.
    - walshemj 12y ago
      effectively yes each worker machine had an identical copy of the required ISAM files which where kept in sync by our system. We had to build a lot of the functionality that comes out of the box in more modern system like hadoop

From http://en.wikipedia.org/wiki/MapReduce http://en.wikipedia.org/wiki/MapReduce: > MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. ... > The "MapReduce System" (also called "infrastructure" or "framework") orchestrates by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance. ... > The name MapReduce originally referred to the proprietary Google technology but has since been genericized. So it would be quite impossible to have a MapReduce system without distributed computing infrastructure; even if you were doing mapping and reducing, it wouldn't be MapReduce.

supermatt 12y ago
I see no mention of a distributed file system there. Local storage is not a requirement of distributed processing.
- 1stop 12y ago
  How do you do distributed processing without a distributed filesystem? Do you mean you'd load the filesystem into memory and send it to the "processors"?
  - supermatt 12y ago
    The data could be stored on a network device, such as a file server or database, for example. It could indeed be local, but it needn't be distributed. In the example GP gave, the data could possibly have been stored in a database queried using segmentation via consistent hashing (a basic way to distribute jobs across a known number of workers).
    - srean 12y ago
      ...defeating the entire purpose: of large scale parallelism on commodity machines. OTOH if you have a way of achieving order 500X parallelism with a centralized commodity server or database, I would love to hear. EDIT @supermatt Ah I see, we differ in the definition then, to me it isnt bigdata/largescale unless it churns through big amounts of stored data. Bitcoin mining is no where in the ball park of this, its an append only log of solutions computed in parallel.
      - supermatt 12y ago
        How on earth do you think bitcoin mining pools work (as an extremely trivial example). They coordinate ranges between a number of workers. The stored size of those ranges is miniscule in comparison to the data of the hashes between those ranges calculated on each 'miner'. These 'coordinators' absolutely work as a centralised 'commodity' storage server (or database) resource for 500x+ parallelism. 'Big Data' means 'Big Data', not 'Big Storage'. They are completely different things.
        seanmcdirmid 12y ago
        Big data doesn't mean big computation, it actually means big data on lots of disks across many nodes. They are completely different things. You might be into HPC, but that's not what Sanjay and Jeff did. HPC and big data loads are quite different.
        supermatt 12y ago
        The bitcoin example may be a bit oversimplified, and may indeed lean more towards HPC. The example was intended to illustrate data locality (as per the parent question), not the actual computation. Big Data may incorporate data from various 3rd party, remote, local, or even random sources. For example, testing whether URLs in a search engines index are currently available. This may be a map/reduce job, it may utilize a local source of urls, but it will also incorporate a remote check of the url. As I said a few links up: DFS is not a requirement for map/reduce.
  - walshemj 12y ago
    I our case the first stage synced up all the required file systems and applied all the required updates before kicking off the mapper stage.
  - walshemj 12y ago
    effectively yes each worker machine had an identical copy of the required ISAM files which where kept in sync by our system. We had to build a lot of the functionality that comes out of the box in more modern system like hadoop