3 ms·

Thanks for your feedback guys. This was our first shot. Next we need to figure out - What kind of problems lend themselves well to our framework? Most MapReduc

Thanks for your feedback guys. This was our first shot. Next we need to figure out - What kind of problems lend themselves well to our framework? Most MapReduce algorithms crunch huge loads of data. Our framework is more useful for long-running computations on freely available data - How does the technical side look? I'm thinking node as stateless master servers, hypertable for storage, and RabbitMQ for flexible persistent job queues. The client runner has some bigger problems, like circumventing the same-origin policy (for pulling data e.g. from graph.facebook.com or twitter) and streaming in large data. - What to do about confidentiality. All computations will by virtue of the framework be visible to the workers. That may not be ideal in some cases. - How to handle worker failure and incorrect results. In our system, workers are unreliable, not trustworthy and slow. That is very different from other implementations ;) That said, if anyone here runs a lot of MapReduce tasks or is generally interested in the subject, drop us a line at team@maprejuice.com or leave us your email address at http://maprejuice.com http://maprejuice.com. Next we would like to take a number of real-world tasks and try to calculate the correct results on our "cluster" :)

endtime 16y ago
Here's something I tried running (index generation): map = function (docID, text) { var results = []; var words = text.split(' '); for(var index in words) { results.push({key: words[index], value: docID}); } this.done = true; this.results = results; } reduce = function (word, docIDs) { self.postMessage("starting reduce for distributed relevance-sorted index generation"); var counts = {}; var results = []; for(var index in docIDs) { var docID = docIDs[index]; if(docID in counts) { counts[docID]++; } else { counts[docID] = 1; results.push(docID); } } results.sort(function(a, b) {return counts[b] - counts[a];}) this.done = true; this.results = results; } data = {1: "the quick brown fox jumps over the lazy dog while the slow yellow fox stumbles around the yard", 2: "on the ning nang nong where the cows go bong and the monkeys all say boo", 3: "lions and tigers and bears oh my , i suppose next we'll see monkeys and maybe a fox"} Didn't seem to work, but maybe I did something wrong.
- Klonoar 16y ago
  Assuming you entered it in the format we take it in right now (separated out, etc), it's just very possible that we're hanging on the backend. We aren't able to touch the thing for a week while it's being judged, so we've had to periodically kill off things to keep it alive. It's not our desire either, but give us time to deal with it and we'll show you some rockin' stuff. ;)
  - endtime 16y ago
    The first time it timed out. After I posted that I tried again and it let me create the job and then start it...but then the results were the empty object. This made me think I screwed up my JS, but it's pretty simple and I did test each function individually, so I think it's correct. Anyway, looking forward to you guys getting the kinks ironed out. Very cool idea.
    - Klonoar 16y ago
      Yeah, your problem just hasn't run yet, it would seem. We didn't have time to implement running multiple problems at once - currently they enter a queue and wait in line for their turn to get distributed. :( Glad to know you at least got it in there, though!

endtime 16y ago

Here's something I tried running (index generation): map = function (docID, text) { var results = []; var words = text.split(' '); for(var index in words) { results.push({key: words[index], value: docID}); } this.done = true; this.results = results; } reduce = function (word, docIDs) { self.postMessage("starting reduce for distributed relevance-sorted index generation"); var counts = {}; var results = []; for(var index in docIDs) { var docID = docIDs[index]; if(docID in counts) { counts[docID]++; } else { counts[docID] = 1; results.push(docID); } } results.sort(function(a, b) {return counts[b] - counts[a];}) this.done = true; this.results = results; } data = {1: "the quick brown fox jumps over the lazy dog while the slow yellow fox stumbles around the yard", 2: "on the ning nang nong where the cows go bong and the monkeys all say boo", 3: "lions and tigers and bears oh my , i suppose next we'll see monkeys and maybe a fox"} Didn't seem to work, but maybe I did something wrong.

Klonoar 16y ago
Assuming you entered it in the format we take it in right now (separated out, etc), it's just very possible that we're hanging on the backend. We aren't able to touch the thing for a week while it's being judged, so we've had to periodically kill off things to keep it alive. It's not our desire either, but give us time to deal with it and we'll show you some rockin' stuff. ;)
- endtime 16y ago
  The first time it timed out. After I posted that I tried again and it let me create the job and then start it...but then the results were the empty object. This made me think I screwed up my JS, but it's pretty simple and I did test each function individually, so I think it's correct. Anyway, looking forward to you guys getting the kinks ironed out. Very cool idea.
  - Klonoar 16y ago
    Yeah, your problem just hasn't run yet, it would seem. We didn't have time to implement running multiple problems at once - currently they enter a queue and wait in line for their turn to get distributed. :( Glad to know you at least got it in there, though!