7 ms·

Amazon announces Elastic MapReduce

148 points by thamer 18y ago

andr 18y ago
Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.
- deleted 18y ago
  [deleted]
  - derefr 18y ago
    Well, it does say "elastic"; perhaps you sometimes have huge batches to run, worthy of map/reducing, and sometimes have batches that could be run on a single machine instead, but since you're applying what's fundamentally the same transformation to both data sets, you only have one codebase, and so it would be nice if you could do it all with a single kind of infrastructure as well.
    - rjurney 18y ago
      Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs. Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before. I just launched the SAME JOB in 15 seconds. You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity) I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.
      - tlrobinson 18y ago
        There's no way to automatically kill the cluster after the job is done? There definitely should be.
        bd 18y ago
        I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).
        rjurney 18y ago
        Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown. Elastic Reduce does that for you. Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.
        rjurney 18y ago
        There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done. http://groups.google.com/group/ec2ubuntu/browse_thread/thread/62fbb31456bbd6f0 http://groups.google.com/group/ec2ubuntu/browse_thread/threa...
      - ggruschow 18y ago
        Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.
- deleted 18y ago
  [deleted]
- dave_au 18y ago
  I remember one of their chief expenses a while back was under-utilization - they need enough capacity for the christmas rush etc, but it does unused during the slow times. It looks like they're letting other people pay for their under-utilized computing power and also deal with the utilization problem themselves beneath the 1 hour mark. Every now and then I wonder if there's a market for someone to do something like app engine on EC2 in order to cater to the 10 minute job market. With something like that you could just leave the EC2 instances up and just swap jobs in and out, track the time and charge some small markup for the convenience. As long as the demand was there you'd just have to bring more instances up in peak times then let them die off one by one when things slow down. Put a nice client side API on it and use Amazons billing API so the client can handle all of their computing payments at the one point and you're ready to go.
  - david927 18y ago
    That's a great idea.
    - dave_au 18y ago
      It's alright as an idea, might not be too great as a business depending on the demand. You'd need decent customer service when things went wrong or if people had questions about jobs / billing, so I imagine you'd need pretty high and reasonably consistent demand to cover that, especially since the markup would probably be tiny. If anyone once to give it a try I'd like to see how it turns out for them.
- njharman 18y ago
  If you have 1 10 minute job that runs on 1 machine, freaking run it on your laptop or desktop. And if you spread your job across N machines so you get results in 10 min then be willing to pay for your impatience.
rjurney 18y ago
Here's a really interesting part of the offering: Cloudburst RMAP DNA searches. A supercomputer by the hour for every biologist that wants one: that is revolutionary. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2272&categoryID=263 http://developer.amazonwebservices.com/connect/entry.jspa?ex... The cloudblast project is here: http://apps.sourceforge.net/mediawiki/cloudburst-bio/index.php?title=CloudBurst http://apps.sourceforge.net/mediawiki/cloudburst-bio/index.p... The paper on BlastReduce is here: http://www.cs.umd.edu/Grad/scholarlypapers/papers/MichaelSchatz.pdf http://www.cs.umd.edu/Grad/scholarlypapers/papers/MichaelSch... Now someone needs to build better open visualization tools to handle the output of this system.
dood 18y ago
It is intersting to watch Amazon's strategy as they improve existing cloud computing services and add new services. The way they are going, it looks like Amazon will totally dominate the market. I expect this will have huge importance to the web, and computing more generally, in the next 5 years or so. I wouldn't be surprised if this becomes as important, or even more important than their traditional business.
- ashot 18y ago
  I don't agree. Though I think its great and technologically the next step forward this will never be a high margin business and will quickly turn into a race to the bottom as the switching costs are zero.
  - rjurney 18y ago
    If you look at what they're doing, they've gone from offering raw storage and computers by the hour to offering DNA analysis as a service. They will keep building more sophisticated offerings. They're not racing to the bottom, they're racing to the top.
    - ashot 18y ago
      dna analysis is a niche, this is a service which by its nature is very easy to commoditize. To be useful it has to be invisible, and to be invisible it must have zero switch cost
      - rjurney 18y ago
        DNA analysis is a niche, and they chose it because it is a commonly executed algorithm with a Hadoop implementation lying around. The point is that by including it they are showing that they will be offering more and more such algorithms for execution as implementations become available. First it was just machines, now they're running specific software and specific jobs on that software for you. The stack just keeps getting taller and nobody else's stack comes close. The more advanced the service you run, the more they are charging. Amazon is winning by building more and more sophisticated services on top of their infrastructure to sell more and more use of the basic services. I hate to sound repetitive, but everything about it is a race to the top, not the bottom.
notaddicted 18y ago
For a quick demonstration of why Amazon would do this, take a look at the old way: http://wiki.apache.org/hadoop/AmazonEC2 http://wiki.apache.org/hadoop/AmazonEC2 Also presumably they have the cluster well tuned, which is not simple, here is a relevant research paper about the scheduling algorithm which also goes into some of the tradeoffs and particularities of Hadoop on EC2: http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99.html http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99... This can be pretty useful for data analysis and science. I don't see Amazon making money from this, but they must think the market is larger than I do. I may have to adjust my opinion.
- rjurney 18y ago
  Actually, the way it works they aren't dealing with a heterogeneous environment and a hadoop scheduler - it runs on an EC2 compute cluster just for you that they spawn on your behalf. Your job is the only job running on your compute cluster. One job per cluster. Sharing jobs would be better, and sharing a cluster with other users even better, but I don't see that it can do that yet. I think that Amazon believes that making Hadoop easy to use and popular is the best way to sell more EC2 and S3 by giving people an easy way to burn cycles and fill gigabytes. I think they're right.
  - nessence 18y ago
    Moreover, Hadoop usage can be intense and it's sensible for Amazon to segregate it as a service to ensure quality and configure hypervisor's accordingly.
    - wmf 18y ago
      I don't see any evidence that Amazon has done this. It sounds like they're just using normal EC2 instances.
  - notaddicted 18y ago
    Right. There are some EC2 specific issues though. Most significant is the network speed. Small instances share a machine with other small instances so they can get more than their "fair share" of network capacity. So because of this the environment maybe effectively heterogeneous if some of the nodes are relatively starved for I/O because they are on a busy machine. Storing all the data on S3 makes this particularly relevant. Since one of the defining features the the MapReduce model (as detailed by the 2004 Google paper: http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html) is that "moving computation is cheaper than moving data" the implementation of S3 becomes salient as well, where exactly is the data? In a normal implementation of Hadoop one would run HDFS, which stores the data on the same machines that run the job. The scheduler attempts to minimize transfer AND transfer is still often the bottleneck when the data is shuffled for the reduce job. So interesting to see how this service compares. EDIT: I haven't tried the new service though, but I will for sure after I get some sleep.
jacktang 18y ago
Doung Cutting should be hired by Amazon!
vidar 18y ago
My first thought was that http://www.cloudera.com/ http://www.cloudera.com/ was in big trouble but I actually think that this is the best thing that could have happened for them.
- jimbokun 18y ago
  This is in reference to a previous thread, but interesting that 3 of the cloudera team have a degree in Mathematics. http://www.cloudera.com/about http://www.cloudera.com/about Maybe the question should be, does a Mathematician need to know programming, or can he just learn it on an as needed basis?
- inerte 18y ago
  Yeah, pretty much like a Starbucks opening right next to your coffe shop. Suddenly every neighbor wants a cup of coffe, and you'll see increased attendance. Cloudera offers are more like consulting. People need help setting up Hadoop jobs, understand MapReduce, or don't want to send data to S3.
gfunk911 18y ago
Does anyone see any obvious documentation on defining your own reduce function? All the example use the built in "aggregate," which looks like it refers to the Hadoop aggregate sub-package. The lines output by the map functions start with the name of a Hadoop class within the aggregate package, which must tell it how to aggregate.
- gfunk911 18y ago
  The PDF here seems to be a good start. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2297&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex...
- rjurney 18y ago
  Map/Reduce is really hard. As you can tell, this thing takes the operational pain out of Hadoop... not the thinkin part :) Beyond something like this: http://hadoop.apache.org/core/docs/current/mapred_tutorial.html http://hadoop.apache.org/core/docs/current/mapred_tutorial.h... there isn't too much in the way of docs on how to think in map/reduce that I'm aware of, but I found the classes in the Pig tutorial to be quite helpful. edit: oh, guy above me has great link!
  - pskomoroch 18y ago
    Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic http://www.cloudera.com/hadoop-training-basic
- pskomoroch 18y ago
  The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.py http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming. The article is here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2294&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex... Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.
tdonia 18y ago
Does information beget information? If so, their model will probably prove quite lucrative.
jackowayed 18y ago
Why is it >6x cheaper than the same EC2 instance? Is it because they know that no one will just have it constantly run it as a true server like some do with EC2, so they'll get more fractional hours? Even though that's true to some extent, it doesn't seem like enough. Any insight? EDIT: Twitter ftw! Apparently, the prices are additive, not replacitive. http://twitter.com/mndoci/statuses/1442516630 http://twitter.com/mndoci/statuses/1442516630

148 points by thamer 18y ago

andr 18y ago

Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.

deleted 18y ago
[deleted]
- derefr 18y ago
  Well, it does say "elastic"; perhaps you sometimes have huge batches to run, worthy of map/reducing, and sometimes have batches that could be run on a single machine instead, but since you're applying what's fundamentally the same transformation to both data sets, you only have one codebase, and so it would be nice if you could do it all with a single kind of infrastructure as well.
  - rjurney 18y ago
    Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs. Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before. I just launched the SAME JOB in 15 seconds. You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity) I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.
    - tlrobinson 18y ago
      There's no way to automatically kill the cluster after the job is done? There definitely should be.
      - bd 18y ago
        I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).
        rjurney 18y ago
        Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown. Elastic Reduce does that for you. Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.
      - rjurney 18y ago
        There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done. http://groups.google.com/group/ec2ubuntu/browse_thread/thread/62fbb31456bbd6f0 http://groups.google.com/group/ec2ubuntu/browse_thread/threa...
    - ggruschow 18y ago
      Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.
deleted 18y ago
[deleted]
dave_au 18y ago
I remember one of their chief expenses a while back was under-utilization - they need enough capacity for the christmas rush etc, but it does unused during the slow times. It looks like they're letting other people pay for their under-utilized computing power and also deal with the utilization problem themselves beneath the 1 hour mark. Every now and then I wonder if there's a market for someone to do something like app engine on EC2 in order to cater to the 10 minute job market. With something like that you could just leave the EC2 instances up and just swap jobs in and out, track the time and charge some small markup for the convenience. As long as the demand was there you'd just have to bring more instances up in peak times then let them die off one by one when things slow down. Put a nice client side API on it and use Amazons billing API so the client can handle all of their computing payments at the one point and you're ready to go.
- david927 18y ago
  That's a great idea.
  - dave_au 18y ago
    It's alright as an idea, might not be too great as a business depending on the demand. You'd need decent customer service when things went wrong or if people had questions about jobs / billing, so I imagine you'd need pretty high and reasonably consistent demand to cover that, especially since the markup would probably be tiny. If anyone once to give it a try I'd like to see how it turns out for them.
njharman 18y ago
If you have 1 10 minute job that runs on 1 machine, freaking run it on your laptop or desktop. And if you spread your job across N machines so you get results in 10 min then be willing to pay for your impatience.

rjurney 18y ago

Here's a really interesting part of the offering: Cloudburst RMAP DNA searches. A supercomputer by the hour for every biologist that wants one: that is revolutionary. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2272&categoryID=263 http://developer.amazonwebservices.com/connect/entry.jspa?ex... The cloudblast project is here: http://apps.sourceforge.net/mediawiki/cloudburst-bio/index.php?title=CloudBurst http://apps.sourceforge.net/mediawiki/cloudburst-bio/index.p... The paper on BlastReduce is here: http://www.cs.umd.edu/Grad/scholarlypapers/papers/MichaelSchatz.pdf http://www.cs.umd.edu/Grad/scholarlypapers/papers/MichaelSch... Now someone needs to build better open visualization tools to handle the output of this system.

dood 18y ago

It is intersting to watch Amazon's strategy as they improve existing cloud computing services and add new services. The way they are going, it looks like Amazon will totally dominate the market. I expect this will have huge importance to the web, and computing more generally, in the next 5 years or so. I wouldn't be surprised if this becomes as important, or even more important than their traditional business.

ashot 18y ago
I don't agree. Though I think its great and technologically the next step forward this will never be a high margin business and will quickly turn into a race to the bottom as the switching costs are zero.
- rjurney 18y ago
  If you look at what they're doing, they've gone from offering raw storage and computers by the hour to offering DNA analysis as a service. They will keep building more sophisticated offerings. They're not racing to the bottom, they're racing to the top.
  - ashot 18y ago
    dna analysis is a niche, this is a service which by its nature is very easy to commoditize. To be useful it has to be invisible, and to be invisible it must have zero switch cost
    - rjurney 18y ago
      DNA analysis is a niche, and they chose it because it is a commonly executed algorithm with a Hadoop implementation lying around. The point is that by including it they are showing that they will be offering more and more such algorithms for execution as implementations become available. First it was just machines, now they're running specific software and specific jobs on that software for you. The stack just keeps getting taller and nobody else's stack comes close. The more advanced the service you run, the more they are charging. Amazon is winning by building more and more sophisticated services on top of their infrastructure to sell more and more use of the basic services. I hate to sound repetitive, but everything about it is a race to the top, not the bottom.

notaddicted 18y ago

For a quick demonstration of why Amazon would do this, take a look at the old way: http://wiki.apache.org/hadoop/AmazonEC2 http://wiki.apache.org/hadoop/AmazonEC2 Also presumably they have the cluster well tuned, which is not simple, here is a relevant research paper about the scheduling algorithm which also goes into some of the tradeoffs and particularities of Hadoop on EC2: http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99.html http://www.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-99... This can be pretty useful for data analysis and science. I don't see Amazon making money from this, but they must think the market is larger than I do. I may have to adjust my opinion.

rjurney 18y ago
Actually, the way it works they aren't dealing with a heterogeneous environment and a hadoop scheduler - it runs on an EC2 compute cluster just for you that they spawn on your behalf. Your job is the only job running on your compute cluster. One job per cluster. Sharing jobs would be better, and sharing a cluster with other users even better, but I don't see that it can do that yet. I think that Amazon believes that making Hadoop easy to use and popular is the best way to sell more EC2 and S3 by giving people an easy way to burn cycles and fill gigabytes. I think they're right.
- nessence 18y ago
  Moreover, Hadoop usage can be intense and it's sensible for Amazon to segregate it as a service to ensure quality and configure hypervisor's accordingly.
  - wmf 18y ago
    I don't see any evidence that Amazon has done this. It sounds like they're just using normal EC2 instances.
- notaddicted 18y ago
  Right. There are some EC2 specific issues though. Most significant is the network speed. Small instances share a machine with other small instances so they can get more than their "fair share" of network capacity. So because of this the environment maybe effectively heterogeneous if some of the nodes are relatively starved for I/O because they are on a busy machine. Storing all the data on S3 makes this particularly relevant. Since one of the defining features the the MapReduce model (as detailed by the 2004 Google paper: http://labs.google.com/papers/mapreduce.html http://labs.google.com/papers/mapreduce.html) is that "moving computation is cheaper than moving data" the implementation of S3 becomes salient as well, where exactly is the data? In a normal implementation of Hadoop one would run HDFS, which stores the data on the same machines that run the job. The scheduler attempts to minimize transfer AND transfer is still often the bottleneck when the data is shuffled for the reduce job. So interesting to see how this service compares. EDIT: I haven't tried the new service though, but I will for sure after I get some sleep.

jacktang 18y ago

Doung Cutting should be hired by Amazon!

vidar 18y ago

My first thought was that http://www.cloudera.com/ http://www.cloudera.com/ was in big trouble but I actually think that this is the best thing that could have happened for them.

jimbokun 18y ago
This is in reference to a previous thread, but interesting that 3 of the cloudera team have a degree in Mathematics. http://www.cloudera.com/about http://www.cloudera.com/about Maybe the question should be, does a Mathematician need to know programming, or can he just learn it on an as needed basis?
inerte 18y ago
Yeah, pretty much like a Starbucks opening right next to your coffe shop. Suddenly every neighbor wants a cup of coffe, and you'll see increased attendance. Cloudera offers are more like consulting. People need help setting up Hadoop jobs, understand MapReduce, or don't want to send data to S3.

gfunk911 18y ago

Does anyone see any obvious documentation on defining your own reduce function? All the example use the built in "aggregate," which looks like it refers to the Hadoop aggregate sub-package. The lines output by the map functions start with the name of a Hadoop class within the aggregate package, which must tell it how to aggregate.

gfunk911 18y ago
The PDF here seems to be a good start. http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2297&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex...
rjurney 18y ago
Map/Reduce is really hard. As you can tell, this thing takes the operational pain out of Hadoop... not the thinkin part :) Beyond something like this: http://hadoop.apache.org/core/docs/current/mapred_tutorial.html http://hadoop.apache.org/core/docs/current/mapred_tutorial.h... there isn't too much in the way of docs on how to think in map/reduce that I'm aware of, but I found the classes in the Pig tutorial to be quite helpful. edit: oh, guy above me has great link!
- pskomoroch 18y ago
  Here is another great resource for how to "think" in terms of mapreduce (joins, keyswaps etc): http://www.cloudera.com/hadoop-training-basic http://www.cloudera.com/hadoop-training-basic
pskomoroch 18y ago
The item similarity code example that uses the api has 4 python streaming mapreduce steps with custom reducers: http://elasticmapreduce.s3.amazonaws.com/samples/similarity/similarity.py http://elasticmapreduce.s3.amazonaws.com/samples/similarity/... AWS just put a pointer to the code bucket on s3, I'll add a link to a zip file with the full source including a script for converting the raw netflix data to something more friendly to Hadoop streaming. The article is here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2294&categoryID=265 http://developer.amazonwebservices.com/connect/entry.jspa?ex... Check out the resources link at the end, the Cloudera Hadoop Hacks site has a few good intros to custom reduers in java, ruby, and python.

tdonia 18y ago

Does information beget information? If so, their model will probably prove quite lucrative.

jackowayed 18y ago

Why is it >6x cheaper than the same EC2 instance? Is it because they know that no one will just have it constantly run it as a true server like some do with EC2, so they'll get more fractional hours? Even though that's true to some extent, it doesn't seem like enough. Any insight? EDIT: Twitter ftw! Apparently, the prices are additive, not replacitive. http://twitter.com/mndoci/statuses/1442516630 http://twitter.com/mndoci/statuses/1442516630