5 ms·

I wish people would stop using the term 'supercomputer' for large scale batch processing on clusters. This seems like just the sort of problem that could be run

I wish people would stop using the term 'supercomputer' for large scale batch processing on clusters. This seems like just the sort of problem that could be run more cheaply using EC2 spot instances.

jacquesm 11y ago
That's a perfectly good use of the term these days.
michaelhoffman 11y ago
They used Stampede for this project, most recently #7 on the Top500 list of supercomputers. http://www.top500.org/lists/2014/11/ http://www.top500.org/lists/2014/11/
Wilya 11y ago
Unless you have all the details and you have compared the price of both options, you don't know that it's cheaper. That seems unlikely to me, for an entity of the size of UT Austin. They probably have enough jobs to run to keep their clusters busy.
- michaelhoffman 11y ago
  Stampede actually serves the entire U.S. research community. Any researcher at a U.S. institution can apply for a resource allocation.
  - StillBored 11y ago
    Yes, and IIRC (from the presentation I went to, and for those that don't know) the resource allocation doesn't have to be the whole cluster, they can split it up and run smaller less scalable jobs across just a portion of the hardware. In fact I think they said the vast majority of the jobs are unable to consume the entire cluster. AKA its AWS for HPC. Also AFAIK, that has been the model at TACC for at least a decade, maybe longer.
mitchty 11y ago
I work for a supercomputer company. There is almost no way you'll get anywhere near the performance you need on EC2. Supercomputers today are built to transfer data between nodes using locality of the node as well as its physical location. Ok EC2 could do the same thing or similar sure, but, it won't have fibre optic links that allow direct DMA of memory from one node to another. The interconnects and the overall use of a supercomputer is drastically different than what you have on EC2. Additionally, your process is just that, another process running on a bare metal system. No hypervisor to get in the way. Keep in mind that some compiled code can care about if an avx instruction is used or not. A tiny detail like that means things like a 5% reduction in cpu cost. When your job/process overall takes months to run, this is a significant reduction. Supercomputers exist because they're more economical than regular hardware. No really, it is true. These things can generate terabytes of data a second and shunt it around efficiently. Until you can demonstrate the same on EC2 you might want to read up on the architectures of modern supercomputers. As a fun note, today cpu power is mostly a gimme, now most simulations care more about memory. For the memory you need you effectively get the cpu for free.
- lmeyerov 11y ago
  You can roughly mimic the Oakridge Titan cluster (#2 in the world) by using AWS: 4 GPUs / node, 10 GE across nodes, and you can spin up a lot of them. Best of all, you don't need $60 mil. This is what we're pushing towards at graphistry ;-) Edit: link: https://en.wikipedia.org/wiki/Titan_(supercomputer) https://en.wikipedia.org/wiki/Titan_(supercomputer)
  - mitchty 11y ago
    Mimic sure, but it won't be quite the same. I won't deny you can't do a one off job possibly cheaper on aws. The thing is, these simulations run for weeks or months on end and these clusters are run and setup to run at 100% at all times. They really are efficient power wise, which is the true test for things like this. I'm hitting towards nda things but needless to say keep an eye out for gpu technology in supercomputers. As well as what that top 500 chinese supercomputer has. Note, its at the top mostly due to its accelerator cards. Most simulations don't need those. I know 60 mill seems like a lot but when it runs for 5 years it really isn't that expensive. For a one off job sure go with aws, it'll take longer, but i'm not kidding when you get down to the detail of the cpu architecture and keep track of things like how cache misses affect a program. It really is night and day difference in how things run. The avx comment is not a joke, reducing even cpu cost by 5% over months is a huge deal. These are just a few of the differences in how supercomputers operate. It is far less to do with the cpu and memory and more how they communicate amongst themselves. One node may compute a value, but another adjacent node then takes on that output to calculate more. This is the key differentiator, and if you aren't compiling fortran (not joking), its likely its not a very great simulation. If you'd like to know more, i'd recommend getting a job at a supercomputer company. They've solved a lot of the overall cluster problems aws has many years ago. And it is really fun stuff where details matter. Something I can't say about most aws deployments. PS: I'm working on something that will make Titan look like a kid. :) Every 2-3 years this stuff gets pushed aside its fun.

I wish people would stop using the term 'supercomputer' for large scale batch processing on clusters. This seems like just the sort of problem that could be run more cheaply using EC2 spot instances.

jacquesm 11y ago

That's a perfectly good use of the term these days.

michaelhoffman 11y ago

They used Stampede for this project, most recently #7 on the Top500 list of supercomputers. http://www.top500.org/lists/2014/11/ http://www.top500.org/lists/2014/11/

Wilya 11y ago

Unless you have all the details and you have compared the price of both options, you don't know that it's cheaper. That seems unlikely to me, for an entity of the size of UT Austin. They probably have enough jobs to run to keep their clusters busy.

michaelhoffman 11y ago
Stampede actually serves the entire U.S. research community. Any researcher at a U.S. institution can apply for a resource allocation.
- StillBored 11y ago
  Yes, and IIRC (from the presentation I went to, and for those that don't know) the resource allocation doesn't have to be the whole cluster, they can split it up and run smaller less scalable jobs across just a portion of the hardware. In fact I think they said the vast majority of the jobs are unable to consume the entire cluster. AKA its AWS for HPC. Also AFAIK, that has been the model at TACC for at least a decade, maybe longer.

mitchty 11y ago

I work for a supercomputer company. There is almost no way you'll get anywhere near the performance you need on EC2. Supercomputers today are built to transfer data between nodes using locality of the node as well as its physical location. Ok EC2 could do the same thing or similar sure, but, it won't have fibre optic links that allow direct DMA of memory from one node to another. The interconnects and the overall use of a supercomputer is drastically different than what you have on EC2. Additionally, your process is just that, another process running on a bare metal system. No hypervisor to get in the way. Keep in mind that some compiled code can care about if an avx instruction is used or not. A tiny detail like that means things like a 5% reduction in cpu cost. When your job/process overall takes months to run, this is a significant reduction. Supercomputers exist because they're more economical than regular hardware. No really, it is true. These things can generate terabytes of data a second and shunt it around efficiently. Until you can demonstrate the same on EC2 you might want to read up on the architectures of modern supercomputers. As a fun note, today cpu power is mostly a gimme, now most simulations care more about memory. For the memory you need you effectively get the cpu for free.

lmeyerov 11y ago
You can roughly mimic the Oakridge Titan cluster (#2 in the world) by using AWS: 4 GPUs / node, 10 GE across nodes, and you can spin up a lot of them. Best of all, you don't need $60 mil. This is what we're pushing towards at graphistry ;-) Edit: link: https://en.wikipedia.org/wiki/Titan_(supercomputer) https://en.wikipedia.org/wiki/Titan_(supercomputer)
- mitchty 11y ago
  Mimic sure, but it won't be quite the same. I won't deny you can't do a one off job possibly cheaper on aws. The thing is, these simulations run for weeks or months on end and these clusters are run and setup to run at 100% at all times. They really are efficient power wise, which is the true test for things like this. I'm hitting towards nda things but needless to say keep an eye out for gpu technology in supercomputers. As well as what that top 500 chinese supercomputer has. Note, its at the top mostly due to its accelerator cards. Most simulations don't need those. I know 60 mill seems like a lot but when it runs for 5 years it really isn't that expensive. For a one off job sure go with aws, it'll take longer, but i'm not kidding when you get down to the detail of the cpu architecture and keep track of things like how cache misses affect a program. It really is night and day difference in how things run. The avx comment is not a joke, reducing even cpu cost by 5% over months is a huge deal. These are just a few of the differences in how supercomputers operate. It is far less to do with the cpu and memory and more how they communicate amongst themselves. One node may compute a value, but another adjacent node then takes on that output to calculate more. This is the key differentiator, and if you aren't compiling fortran (not joking), its likely its not a very great simulation. If you'd like to know more, i'd recommend getting a job at a supercomputer company. They've solved a lot of the overall cluster problems aws has many years ago. And it is really fun stuff where details matter. Something I can't say about most aws deployments. PS: I'm working on something that will make Titan look like a kid. :) Every 2-3 years this stuff gets pushed aside its fun.