4 ms·
What about using elastic map reduce with spot instances instead of a custom job queue. Hadoop seems to do this for us and supports the arc format as an inputfo
by amalag 14y ago
What about using elastic map reduce with spot instances instead of a custom job queue. Hadoop seems to do this for us and supports the arc format as an inputformat.
- jeffbarr 14y agoYou can do that with ease. Here's my blog post: http://aws.typepad.com/aws/2011/08/run-amazon-elastic-mapreduce-on-ec2-spot-instances.html http://aws.typepad.com/aws/2011/08/run-amazon-elastic-mapred...
- amalag 14y agoYes I was wondering why the author implemented his own queuing mechanism instead of just using hadoop via EMR.