4 ms·
British Telecom used map reduce in billing systems for the dialcom (telecom gold) platform in the 80's - that was on the largest (non black) prime minicomputer
by walshemj 12y ago
British Telecom used map reduce in billing systems for the dialcom (telecom gold) platform in the 80's - that was on the largest (non black) prime minicomputer site in the UK.
Back then 17x 750's would be roughly the same as one the 5k plus clusters that yahoo etal use.
We even sold the system to NZ telecom
- seanmcdirmid 12y agoInteresting. What kind of distributed file system were they using?
- supermatt 12y agoA dfs isn't a requirement for map/reduce.
- seanmcdirmid 12y agoFrom http://en.wikipedia.org/wiki/MapReduce http://en.wikipedia.org/wiki/MapReduce: > MapReduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. ... > The "MapReduce System" (also called "infrastructure" or "framework") orchestrates by marshalling the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for redundancy and fault tolerance. ... > The name MapReduce originally referred to the proprietary Google technology but has since been genericized. So it would be quite impossible to have a MapReduce system without distributed computing infrastructure; even if you were doing mapping and reducing, it wouldn't be MapReduce.
- supermatt 12y agoI see no mention of a distributed file system there. Local storage is not a requirement of distributed processing.
- 1stop 12y agoHow do you do distributed processing without a distributed filesystem? Do you mean you'd load the filesystem into memory and send it to the "processors"?
- supermatt 12y agoThe data could be stored on a network device, such as a file server or database, for example. It could indeed be local, but it needn't be distributed. In the example GP gave, the data could possibly have been stored in a database queried using segmentation via consistent hashing (a basic way to distribute jobs across a known number of workers).
- srean 12y ago...defeating the entire purpose: of large scale parallelism on commodity machines. OTOH if you have a way of achieving order 500X parallelism with a centralized commodity server or database, I would love to hear. EDIT @supermatt Ah I see, we differ in the definition then, to me it isnt bigdata/largescale unless it churns through big amounts of stored data. Bitcoin mining is no where in the ball park of this, its an append only log of solutions computed in parallel.
- supermatt 12y agoHow on earth do you think bitcoin mining pools work (as an extremely trivial example). They coordinate ranges between a number of workers. The stored size of those ranges is miniscule in comparison to the data of the hashes between those ranges calculated on each 'miner'. These 'coordinators' absolutely work as a centralised 'commodity' storage server (or database) resource for 500x+ parallelism. 'Big Data' means 'Big Data', not 'Big Storage'. They are completely different things.
- seanmcdirmid 12y agoBig data doesn't mean big computation, it actually means big data on lots of disks across many nodes. They are completely different things. You might be into HPC, but that's not what Sanjay and Jeff did. HPC and big data loads are quite different.
- walshemj 12y agowe used the normal file system (primes probably descended from ITS) and had a load of JCL written in CPL (prime JCL) language to sync every thing up over our Cambridge ring to two sites. (we had oxford street dug up for our 10MBs link)