5 ms·
Right. There are some EC2 specific issues though. Most significant is the network speed. Small instances share a machine with other small instances so they can
by notaddicted 18y ago
Right. There are some EC2 specific issues though.
Most significant is the network speed. Small instances share a machine with other small instances so they can get more than their "fair share" of network capacity. So because of this the environment maybe effectively heterogeneous if some of the nodes are relatively starved for I/O because they are on a busy machine. Storing all the data on S3 makes this particularly relevant.
Since one of the defining features the the MapReduce model (as detailed by the 2004 Google paper: http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html) is that "moving computation is cheaper than moving data" the implementation of S3 becomes salient as well, where exactly is the data?
In a normal implementation of Hadoop one would run HDFS, which stores the data on the same machines that run the job. The scheduler attempts to minimize transfer AND transfer is still often the bottleneck when the data is shuffled for the reduce job. So interesting to see how this service compares.
EDIT: I haven't tried the new service though, but I will for sure after I get some sleep.
- rjurney 18y agoThey still run HDFS, they just do it from S3. Presumably they are using EBS internally. This is exactly how you would do it if you set it up yourself... they're just doing it for you. Small instances suck at Hadoop, you probably won't use them.
- monological 18y agoIn other words... MapReduce implementations have to take into account locality, i.e., the dataNodes have to be rack-aware for optimal I/O speed. Therefore, if their dataNodes are actually just cpu cores reading/writing blocks to S3 (other racks) then they are effectively getting kicked in the face.
- rjurney 18y agoHow many people have dedicated hadoop clusters available to them in the first place? The point of the pattern, framework and product is to scale linearly and on demand, not to do the fastest computation per-machine possible. As to IO - HDFS is going to divvy up the workload from S3 when the cluster bootstraps. After that the data will be local to each compute node, so they won't be getting kicked in the face. There is no reason to suppose that S3 replaces HDFS - its just used to load your data in. Also, one would hope that their Hadoop setup IS rack aware - why shouldn't it be?
- pskomoroch 18y agoThis is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines. Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/input/", "-output", "hdfs:///home/hadoop/output2/", Step 2: "-input", "hdfs:///home/hadoop/output2/", "-output", "hdfs:///home/hadoop/output3/", etc.. There is no explicit support for mounting data from EBS yet, but in theory I think you can hack something together in the initial mapper to mount an EBS volume to be used in the second map task...