3 ms·
Anyone have any experience/thoughts on using spot instances with Hadoop? Specifically, regular instances with Hadoop installed (not via Elastic Map Reduce). Th
by cpenner461 14y ago
Anyone have any experience/thoughts on using spot instances with Hadoop? Specifically, regular instances with Hadoop installed (not via Elastic Map Reduce). The cost savings are potentially huge, but I'd hate to lose my instances 80-90% of the way through a set of long running (12-48h) jobs. I guess if I had EBS backed instances I could relaunch and resume, but I'm not sure how well that'd work in practice.
- Evbn 14y agoWhy are your jobs 12-48hr? Allocate more worker hosts during job runs.
- xtrahotsauce 14y agoWe bring up spot instances all the time with EMR as additional compute capacity, and we're okay with these instances possibly going away because we use HDFS very little. Instead, we store almost everything on S3.