3 ms·

What makes you think the data is on disk? No, seriously. Picture if all of your data is in RAM, somewhere. If you can move your program to that machine quick

What makes you think the data is on disk? No, seriously. Picture if all of your data is in RAM, somewhere. If you can move your program to that machine quickly, then that's a huge win.

thrownaway2424 11y ago
It just doesn't seem very likely to me. It seems a lot more likely that the data would be spread around on thousands of individual hosts, one megabyte at a time. The only processes by which I imagine the entire contents of one shard of a mapreduce ending up in memory of a single host are 1) a miracle, 2) the previous stage of the mapreduce wrote it there, in which case it is effectively remote access before the fact, and 3) the data lives there all the time for some other reason, in which case you don't have a mapreduce, you have a coprocessor.
- VikingCoder 11y ago
  I agree about the one megabyte at a time. I disagree that that implies it's likely evicted from RAM. For one, if I'm running a MapReduce, it's either likely on the same data someone else is investigating... Or I'm likely to have to re-run my MapReduce a few times, calculating exactly what I'm looking for. So, I may be cheating by saying "Only the first time is the data cold. Every other time, the data is likely to be hot in RAM." Because you'd be right to say, "Well, duh. I'm talking about the FIRST time. And since I'm talking about the first time, moving the data across the network wouldn't be that bad. After that, yes, of course it will still likely be hot, just as you describe." My only real response is, I think it depends on how much data, versus how much code you're talking about. And can you maybe even broadcast the code to the right nodes, to even more effectively use the network. "in which case you don't have a mapreduce, you have a coprocessor." ...a coprocessor which uses the MapReduce API. Yes, I think that is exactly what you have. And it works whether the data is local or remote, cold or hot. One unified API for all of those things, and it's probably optimized to work local and hot, because I'm betting more than half of all MapReduces actually work out to be local and hot.

What makes you think the data is on disk? No, seriously. Picture if all of your data is in RAM, somewhere. If you can move your program to that machine quickly, then that's a huge win.

thrownaway2424 11y ago

It just doesn't seem very likely to me. It seems a lot more likely that the data would be spread around on thousands of individual hosts, one megabyte at a time. The only processes by which I imagine the entire contents of one shard of a mapreduce ending up in memory of a single host are 1) a miracle, 2) the previous stage of the mapreduce wrote it there, in which case it is effectively remote access before the fact, and 3) the data lives there all the time for some other reason, in which case you don't have a mapreduce, you have a coprocessor.

VikingCoder 11y ago
I agree about the one megabyte at a time. I disagree that that implies it's likely evicted from RAM. For one, if I'm running a MapReduce, it's either likely on the same data someone else is investigating... Or I'm likely to have to re-run my MapReduce a few times, calculating exactly what I'm looking for. So, I may be cheating by saying "Only the first time is the data cold. Every other time, the data is likely to be hot in RAM." Because you'd be right to say, "Well, duh. I'm talking about the FIRST time. And since I'm talking about the first time, moving the data across the network wouldn't be that bad. After that, yes, of course it will still likely be hot, just as you describe." My only real response is, I think it depends on how much data, versus how much code you're talking about. And can you maybe even broadcast the code to the right nodes, to even more effectively use the network. "in which case you don't have a mapreduce, you have a coprocessor." ...a coprocessor which uses the MapReduce API. Yes, I think that is exactly what you have. And it works whether the data is local or remote, cold or hot. One unified API for all of those things, and it's probably optimized to work local and hot, because I'm betting more than half of all MapReduces actually work out to be local and hot.