4 ms·
In other words... MapReduce implementations have to take into account locality, i.e., the dataNodes have to be rack-aware for optimal I/O speed. Therefore, if
by monological 18y ago
In other words...
MapReduce implementations have to take into account locality, i.e., the dataNodes have to be rack-aware for optimal I/O speed. Therefore, if their dataNodes are actually just cpu cores reading/writing blocks to S3 (other racks) then they are effectively getting kicked in the face.
- rjurney 18y agoHow many people have dedicated hadoop clusters available to them in the first place? The point of the pattern, framework and product is to scale linearly and on demand, not to do the fastest computation per-machine possible. As to IO - HDFS is going to divvy up the workload from S3 when the cluster bootstraps. After that the data will be local to each compute node, so they won't be getting kicked in the face. There is no reason to suppose that S3 replaces HDFS - its just used to load your data in. Also, one would hope that their Hadoop setup IS rack aware - why shouldn't it be?
- pskomoroch 18y agoThis is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines. Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/input/", "-output", "hdfs:///home/hadoop/output2/", Step 2: "-input", "hdfs:///home/hadoop/output2/", "-output", "hdfs:///home/hadoop/output3/", etc.. There is no explicit support for mounting data from EBS yet, but in theory I think you can hack something together in the initial mapper to mount an EBS volume to be used in the second map task...