3 ms·

Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.

by andr 18y ago

deleted 18y ago
[deleted]
- derefr 18y ago
  Well, it does say "elastic"; perhaps you sometimes have huge batches to run, worthy of map/reducing, and sometimes have batches that could be run on a single machine instead, but since you're applying what's fundamentally the same transformation to both data sets, you only have one codebase, and so it would be nice if you could do it all with a single kind of infrastructure as well.
  - rjurney 18y ago
    Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs. Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before. I just launched the SAME JOB in 15 seconds. You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity) I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.
    - tlrobinson 18y ago
      There's no way to automatically kill the cluster after the job is done? There definitely should be.
      - bd 18y ago
        I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).
        rjurney 18y ago
        Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown. Elastic Reduce does that for you. Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.
      - rjurney 18y ago
        There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done. http://groups.google.com/group/ec2ubuntu/browse_thread/thread/62fbb31456bbd6f0 http://groups.google.com/group/ec2ubuntu/browse_thread/threa...
    - ggruschow 18y ago
      Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.
deleted 18y ago
[deleted]
dave_au 18y ago
I remember one of their chief expenses a while back was under-utilization - they need enough capacity for the christmas rush etc, but it does unused during the slow times. It looks like they're letting other people pay for their under-utilized computing power and also deal with the utilization problem themselves beneath the 1 hour mark. Every now and then I wonder if there's a market for someone to do something like app engine on EC2 in order to cater to the 10 minute job market. With something like that you could just leave the EC2 instances up and just swap jobs in and out, track the time and charge some small markup for the convenience. As long as the demand was there you'd just have to bring more instances up in peak times then let them die off one by one when things slow down. Put a nice client side API on it and use Amazons billing API so the client can handle all of their computing payments at the one point and you're ready to go.
- david927 18y ago
  That's a great idea.
  - dave_au 18y ago
    It's alright as an idea, might not be too great as a business depending on the demand. You'd need decent customer service when things went wrong or if people had questions about jobs / billing, so I imagine you'd need pretty high and reasonably consistent demand to cover that, especially since the markup would probably be tiny. If anyone once to give it a try I'd like to see how it turns out for them.
njharman 18y ago
If you have 1 10 minute job that runs on 1 machine, freaking run it on your laptop or desktop. And if you spread your job across N machines so you get results in 10 min then be willing to pay for your impatience.

Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.

by andr 18y ago

Great, except that it's no good for small jobs. Looks like if you want to run a 10-minute job you are charged for a full hour for each instance.

deleted 18y ago

[deleted]

derefr 18y ago
Well, it does say "elastic"; perhaps you sometimes have huge batches to run, worthy of map/reducing, and sometimes have batches that could be run on a single machine instead, but since you're applying what's fundamentally the same transformation to both data sets, you only have one codebase, and so it would be nice if you could do it all with a single kind of infrastructure as well.
- rjurney 18y ago
  Actually, you can. And if you have small data, run the same job on one compute instance. On once instance, getting charged pennies per job from leftover EC2 time is not a big deal. If you do that kind of thing a lot instead of bigger jobs, map/reduce is the wrong way to solve your problem. Its for big jobs, not small jobs. Bottom line of what this means: it took me hours to get scripts going to launch a cloudburst hadoop compute cluster on demand on Monday. And I've written such scripts before. I just launched the SAME JOB in 15 seconds. You know what else? I got distracted and left my own custom compute cluster on overnight and it cost me $50. Elastic Reduce shuts it down for me. (edited for clarity) I was looking into building this. Amazon did it first. Glad I didn't put too much time into it.
  - tlrobinson 18y ago
    There's no way to automatically kill the cluster after the job is done? There definitely should be.
    - bd 18y ago
      I believe rjurney ment his hand-made cluster from before didn't automatically shut down (as opposed to this new Elastic MapReduce feature which presumably would).
      - rjurney 18y ago
        Right. The scripts mounted the data on EBS, configured a security group, etc. etc. but I didn't get around to automatic shutdown. Elastic Reduce does that for you. Although you can't do anything interesting like run Pig or Hive. 2.0 will be much better I'm sure.
    - rjurney 18y ago
      There are ways, I just didn't get there yet since I just got it up and running and expected to wait until it was done. http://groups.google.com/group/ec2ubuntu/browse_thread/thread/62fbb31456bbd6f0 http://groups.google.com/group/ec2ubuntu/browse_thread/threa...
  - ggruschow 18y ago
    Great point. Auto-shutoff makes up for the 15% pricing overhead. I've spent lots on forgetting to shutdown too. It should go in their marketing literature if they could figure out how to make it not sound like a negative for the normal service.

deleted 18y ago

[deleted]

dave_au 18y ago

I remember one of their chief expenses a while back was under-utilization - they need enough capacity for the christmas rush etc, but it does unused during the slow times. It looks like they're letting other people pay for their under-utilized computing power and also deal with the utilization problem themselves beneath the 1 hour mark. Every now and then I wonder if there's a market for someone to do something like app engine on EC2 in order to cater to the 10 minute job market. With something like that you could just leave the EC2 instances up and just swap jobs in and out, track the time and charge some small markup for the convenience. As long as the demand was there you'd just have to bring more instances up in peak times then let them die off one by one when things slow down. Put a nice client side API on it and use Amazons billing API so the client can handle all of their computing payments at the one point and you're ready to go.

david927 18y ago
That's a great idea.
- dave_au 18y ago
  It's alright as an idea, might not be too great as a business depending on the demand. You'd need decent customer service when things went wrong or if people had questions about jobs / billing, so I imagine you'd need pretty high and reasonably consistent demand to cover that, especially since the markup would probably be tiny. If anyone once to give it a try I'd like to see how it turns out for them.

njharman 18y ago

If you have 1 10 minute job that runs on 1 machine, freaking run it on your laptop or desktop. And if you spread your job across N machines so you get results in 10 min then be willing to pay for your impatience.