4 ms·
Hah, we were doing this with EMR 6 years ago, I guess we were a little early :) https://www.youtube.com/watch?v=NF6zwHlbh_I https://www.youtube.com/watch?v=NF6
by gfodor 9y ago
Hah, we were doing this with EMR 6 years ago, I guess we were a little early :)
https://www.youtube.com/watch?v=NF6zwHlbh_I https://www.youtube.com/watch?v=NF6zwHlbh_I
We built a coordinator that would spin up specific categories of machines for each stage (some stages were MR jobs, some were hadoop streaming jobs) -- for example when doing in-memory work it was useful to have fewer nodes with more RAM, etc.
- matt_wulfeck 9y agoDidn't see the video, but what platform did you spin up the cluster on? On any public cloud I have a hard time seeing this make financial sense other than Google, which bills by the minute.
- mk89 9y agoEMR = Aws
- chimerasaurus 9y agoWe're also looking at making this a bit easier with some API changes (additions, really) and work with other OSS projects, like Apache Airflow. Streaming is a very interesting use case; a focus for us as well. Keeping clusters running for months/years presents some interesting challenges. Disclaimer - I'm the Cloud Dataproc PM. :)
- mk89 9y agoCool! EMR takes minutes, though, to spin up a cluster, and the billing seems to be different and less convenient than GCP.
- brango 9y agoSame here. Built a pooling algorithm so we can have a maximum number of pools for each job type and distribute more jobs onto them when the maximum is reached. Clusters shut down when idle. It's served us well and saved loads of time and money.