3 ms·

This is nice. I work at a University and I spoke to the guy who runs our High Performance Computing system about Amazon EC2 the other day. He made a good point

This is nice. I work at a University and I spoke to the guy who runs our High Performance Computing system about Amazon EC2 the other day. He made a good point which is worth noting: You can bring up a lot of EC2 instances to run large jobs in parallel and get a lot of CPU horsepower, but the main drawback of EC2 is the network. Most of the jobs that run on our local "super computer" involve processing terrabytes of data. Transferring terrabytes of data to the cloud is painful, and will continue to be painful for a long time to come. I suspect even transferring terrabytes of data between EC2 instances wont be smooth either. Still, EC2 is great and I love how fast they're bringing out new features.

liuliu 16y ago
I guess that is the reason why you really don't want to spin off a lot EC2 instances for large jobs. Instead, you may use the MapReduce framework s.t. you can exploit the data locality. And of course, MapReduce only solves a small portion of parallel problems.
swannodette 16y ago
Huh, isn't that the whole point of EC2 Cluster Instance. You can specify that you want your instances to be in the same physical location.
- mike-cardwell 16y ago
  I'm not sure I understand how that would help the issue of transferring terrabytes of data over the Internet? I've not come across the "Cluster Instance" though so maybe I'm missing something?
- SpikeGronim 16y ago
  The two main points of the cluster compute instance were 10 GBit/s full bisection bandwidth networking and high end hardware. Now you can add GPUs as the third reason to use cluster compute instances.
njharman 16y ago
They have (or had) a service to ship them media (stack of harddrives) for data transference. Good if you'll be using the same data for a while. AWS doesn't solve every computing infrastructure problem, it's still awesome.
- mike-cardwell 16y ago
  Cool. I didn't know that.
  - axomhacker 16y ago
    In case you have not stumbled on to it yet: http://aws.amazon.com/importexport http://aws.amazon.com/importexport
  - bbgm 16y ago
    And we have peering arrangements with several R&E networks and more coming.
    - aristus 16y ago
      That is pretty awesome to hear. Any plans to peer with other cloud providers? a la http://cloud-peering.com http://cloud-peering.com

liuliu 16y ago

I guess that is the reason why you really don't want to spin off a lot EC2 instances for large jobs. Instead, you may use the MapReduce framework s.t. you can exploit the data locality. And of course, MapReduce only solves a small portion of parallel problems.

swannodette 16y ago

Huh, isn't that the whole point of EC2 Cluster Instance. You can specify that you want your instances to be in the same physical location.

mike-cardwell 16y ago
I'm not sure I understand how that would help the issue of transferring terrabytes of data over the Internet? I've not come across the "Cluster Instance" though so maybe I'm missing something?
SpikeGronim 16y ago
The two main points of the cluster compute instance were 10 GBit/s full bisection bandwidth networking and high end hardware. Now you can add GPUs as the third reason to use cluster compute instances.

njharman 16y ago

They have (or had) a service to ship them media (stack of harddrives) for data transference. Good if you'll be using the same data for a while. AWS doesn't solve every computing infrastructure problem, it's still awesome.

mike-cardwell 16y ago
Cool. I didn't know that.
- axomhacker 16y ago
  In case you have not stumbled on to it yet: http://aws.amazon.com/importexport http://aws.amazon.com/importexport
- bbgm 16y ago
  And we have peering arrangements with several R&E networks and more coming.
  - aristus 16y ago
    That is pretty awesome to hear. Any plans to peer with other cloud providers? a la http://cloud-peering.com http://cloud-peering.com