3 ms·

Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 t

by rjurney 18y ago

Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs. Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before. I just launched the SAME JOB in 15 seconds. You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity) I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.

tlrobinson 18y ago
There's no way to automatically kill the cluster after the job is done? There definitely should be.
- bd 18y ago
  I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).
  - rjurney 18y ago
    Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown. Elastic Reduce does that for you. Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.
- rjurney 18y ago
  There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done. http://groups.google.com/group/ec2ubuntu/browse_thread/thread/62fbb31456bbd6f0 http://groups.google.com/group/ec2ubuntu/browse_thread/threa...
ggruschow 18y ago
Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.

by rjurney 18y ago

tlrobinson 18y ago

There's no way to automatically kill the cluster after the job is done? There definitely should be.

bd 18y ago
I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).
- rjurney 18y ago
  Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown. Elastic Reduce does that for you. Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.
rjurney 18y ago
There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done. http://groups.google.com/group/ec2ubuntu/browse_thread/thread/62fbb31456bbd6f0 http://groups.google.com/group/ec2ubuntu/browse_thread/threa...

ggruschow 18y ago

Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.