3 ms·

For a quick demonstration of why Amazon would do this, take a look at the old way: http://wiki.apache.org/hadoop/AmazonEC2 http://wiki.apache.org/hadoop/AmazonE

For a quick demonstration of why Amazon would do this, take a look at the old way: http://wiki.apache.org/hadoop/AmazonEC2 http://wiki.apache.org/hadoop/AmazonEC2 Also presumably they have the cluster well tuned, which is not simple, here is a relevant research paper about the scheduling algorithm which also goes into some of the tradeoffs and particularities of Hadoop on EC2: http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99.html http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99... This can be pretty useful for data analysis and science. I don't see Amazon making money from this, but they must think the market is larger than I do. I may have to adjust my opinion.

rjurney 18y ago
Actually, the way it works they aren't dealing with a heterogeneous environment and a hadoop scheduler - it runs on an EC2 compute cluster just for you that they spawn on your behalf. Your job is the only job running on your compute cluster. One job per cluster. Sharing jobs would be better, and sharing a cluster with other users even better, but I don't see that it can do that yet. I think that Amazon believes that making Hadoop easy to use and popular is the best way to sell more EC2 and S3 by giving people an easy way to burn cycles and fill gigabytes. I think they're right.
- nessence 18y ago
  Moreover, Hadoop usage can be intense and it's sensible for Amazon to segregate it as a service to ensure quality and configure hypervisor's accordingly.
  - wmf 18y ago
    I don't see any evidence that Amazon has done this. It sounds like they're just using normal EC2 instances.
- notaddicted 18y ago
  Right. There are some EC2 specific issues though. Most significant is the network speed. Small instances share a machine with other small instances so they can get more than their "fair share" of network capacity. So because of this the environment maybe effectively heterogeneous if some of the nodes are relatively starved for I/O because they are on a busy machine. Storing all the data on S3 makes this particularly relevant. Since one of the defining features the the MapReduce model (as detailed by the 2004 Google paper: http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html) is that "moving computation is cheaper than moving data" the implementation of S3 becomes salient as well, where exactly is the data? In a normal implementation of Hadoop one would run HDFS, which stores the data on the same machines that run the job. The scheduler attempts to minimize transfer AND transfer is still often the bottleneck when the data is shuffled for the reduce job. So interesting to see how this service compares. EDIT: I haven't tried the new service though, but I will for sure after I get some sleep.
  - rjurney 18y ago
    They still run HDFS, they just do it from S3. Presumably they are using EBS internally. This is exactly how you would do it if you set it up yourself... they're just doing it for you. Small instances suck at Hadoop, you probably won't use them.
  - monological 18y ago
    In other words... MapReduce implementations have to take into account locality, i.e., the dataNodes have to be rack-aware for optimal I/O speed. Therefore, if their dataNodes are actually just cpu cores reading/writing blocks to S3 (other racks) then they are effectively getting kicked in the face.
    - rjurney 18y ago
      How many people have dedicated hadoop clusters available to them in the first place? The point of the pattern, framework and product is to scale linearly and on demand, not to do the fastest computation per-machine possible. As to IO - HDFS is going to divvy up the workload from S3 when the cluster bootstraps. After that the data will be local to each compute node, so they won't be getting kicked in the face. There is no reason to suppose that S3 replaces HDFS - its just used to load your data in. Also, one would hope that their Hadoop setup IS rack aware - why shouldn't it be?
      - pskomoroch 18y ago
        This is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines. Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/input/", "-output", "hdfs:///home/hadoop/output2/", Step 2: "-input", "hdfs:///home/hadoop/output2/", "-output", "hdfs:///home/hadoop/output3/", etc.. There is no explicit support for mounting data from EBS yet, but in theory I think you can hack something together in the initial mapper to mount an EBS volume to be used in the second map task...

rjurney 18y ago

Actually, the way it works they aren't dealing with a heterogeneous environment and a hadoop scheduler - it runs on an EC2 compute cluster just for you that they spawn on your behalf. Your job is the only job running on your compute cluster. One job per cluster. Sharing jobs would be better, and sharing a cluster with other users even better, but I don't see that it can do that yet. I think that Amazon believes that making Hadoop easy to use and popular is the best way to sell more EC2 and S3 by giving people an easy way to burn cycles and fill gigabytes. I think they're right.

nessence 18y ago
Moreover, Hadoop usage can be intense and it's sensible for Amazon to segregate it as a service to ensure quality and configure hypervisor's accordingly.
- wmf 18y ago
  I don't see any evidence that Amazon has done this. It sounds like they're just using normal EC2 instances.
notaddicted 18y ago
Right. There are some EC2 specific issues though. Most significant is the network speed. Small instances share a machine with other small instances so they can get more than their "fair share" of network capacity. So because of this the environment maybe effectively heterogeneous if some of the nodes are relatively starved for I/O because they are on a busy machine. Storing all the data on S3 makes this particularly relevant. Since one of the defining features the the MapReduce model (as detailed by the 2004 Google paper: http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html) is that "moving computation is cheaper than moving data" the implementation of S3 becomes salient as well, where exactly is the data? In a normal implementation of Hadoop one would run HDFS, which stores the data on the same machines that run the job. The scheduler attempts to minimize transfer AND transfer is still often the bottleneck when the data is shuffled for the reduce job. So interesting to see how this service compares. EDIT: I haven't tried the new service though, but I will for sure after I get some sleep.
- rjurney 18y ago
  They still run HDFS, they just do it from S3. Presumably they are using EBS internally. This is exactly how you would do it if you set it up yourself... they're just doing it for you. Small instances suck at Hadoop, you probably won't use them.
- monological 18y ago
  In other words... MapReduce implementations have to take into account locality, i.e., the dataNodes have to be rack-aware for optimal I/O speed. Therefore, if their dataNodes are actually just cpu cores reading/writing blocks to S3 (other racks) then they are effectively getting kicked in the face.
  - rjurney 18y ago
    How many people have dedicated hadoop clusters available to them in the first place? The point of the pattern, framework and product is to scale linearly and on demand, not to do the fastest computation per-machine possible. As to IO - HDFS is going to divvy up the workload from S3 when the cluster bootstraps. After that the data will be local to each compute node, so they won't be getting kicked in the face. There is no reason to suppose that S3 replaces HDFS - its just used to load your data in. Also, one would hope that their Hadoop setup IS rack aware - why shouldn't it be?
    - pskomoroch 18y ago
      This is correct, in the job configuration only the initial data is loaded from s3, afterwards you can chain mapreduce steps that run off hdfs on the local machines. Step 1: "-input", "s3n://elasticmapreduce/samples/similarity/lastfm/input/", "-output", "hdfs:///home/hadoop/output2/", Step 2: "-input", "hdfs:///home/hadoop/output2/", "-output", "hdfs:///home/hadoop/output3/", etc.. There is no explicit support for mounting data from EBS yet, but in theory I think you can hack something together in the initial mapper to mount an EBS volume to be used in the second map task...