4 ms·

Cray CS-Storm Delivers High-Performance Computing In Million-Dollar Package

32 points by marklittlewood 12y ago

laxatives 12y ago
In what cases would this be better than AWS/other IaaS? Seems like a steep investment. Do labs use stuff like this?
- jff 12y ago
  Real scientific computing needs high-bandwidth low-latency networks.
  - kolencherry 12y ago
    Microsoft Azure actually offers InfiniBand for connectivity, which is interesting. I'm not sure how many people are using their HPC-oriented offering.
  - collyw 12y ago
    Go on then, let me know what "real scientific computing" is. I work with bioinformaticians, which is an area with lots of research going on. I don't see much that is complex there. Split up your jobs and send them to a cluster. I take it that's not "real scientific computing" though?
    - sp332 12y ago
      From the article: planet-wide weather forecasting, seismic measuring for oil exploration, financial analysis or crash simulations. You can't split up a physical simulation.
      - jff 12y ago
        You actually can split up a physical simulation. Imagine you're trying to model heat diffusion through a piece of metal. You split the chunk of metal up into many small pieces and model temperature in each one. Each node in your cluster is simulating the behavior of some portion of the model; at each timestep, nodes simulating chunks that border each other must swap the state of their "edges", sharing the outer edges of their own state with their neighbors. Edit: unless you meant that you can't split up a physical simulation on loosely-coupled nodes and still get good performance. Then, well, yeah.
        sp332 12y ago
        Well I guess the simulation would have to be broken up to run on multiple GPUs at once anyway. But even if you had a good interconnect, the GPUs in this box (K40) have almost 2x as many processors and 3x as much RAM. (Edit: compared to AWS GPU instances.) If you're doing a 3D sim, each node needs to communicate with 26 others, so just having 1/3 as many nodes is going to help a lot.
    - jff 12y ago
      Split up your jobs how? Is the job embarrassingly parallel? Because if it isn't, and most things aren't, it's going to require significant programmer overhead to simply "split up your jobs". And then once you've split up the job, the different portions are going to need to exchange their in-progress results, because most real scientific computing programs involve iterating until either a certain number of timesteps have passed or a certain residual is reached. I mean, I only write HPC programming frameworks for scientific computing as my job, what do I know. But if you're doing anything more complex than map-reduce, a high-speed interconnect is essential, and even in mapreduce it's pretty handy.
- sp332 12y ago
  Approximating numbers from https://arstechnica.com/information-technology/2013/11/18-hours-33k-and-156314-cores-amazon-cloud-hpc-hits-a-petaflop/ https://arstechnica.com/information-technology/2013/11/18-ho... a petaflop on AWS would cost $20k per day. So considering hardware alone, this would be cheaper in 50 days. Once you add in power and cooling for this device, and the "reserved instance" discount from Amazon, it would take longer to break even. But probably not more than a few months. Edit: it answers this in the article.
- m_mueller 12y ago
  On the opposite, I've never heard of a lab that does use cloud computing for HPC. Only for highly variable workloads with low networking and privacy requirements something like AWS would make sense.
- jostmey 12y ago
  Most labs could not afford this. But I could see universities buying lots of these for shared, general purpose calculations. I myself would not mind having access to a rack of them!
m_mueller 12y ago
I'm a bit confused, is the described 1PFlop system really only one million dollar, including cooling and power? That would be amazingly good value. The single digit PFlop systems I know of have cost two to four orders of magnitude more than this, even when powered by GPUs.
- leephillips 12y ago
  The computers cost 10 billion dollars? Who buys them? That would be above the noise floor of even the NSA's budget.
  - lutusp 12y ago
    That might be true, but to establish it, we would have to know the NSA'a budget (which is classified).
  - m_mueller 12y ago
    Sorry, I meant one to three orders of magnitude. The most expensive computer I know of is the Japanese K computer at ~ 1.25 billion and 10 something Petaflops. This was mostly politically motivated (Japanese chips) and too costly for what it does IMO. A typical new supercomputer in the Top 500 today is in the order of 100 million.
- MrVitaliy 12y ago
  from the article "250 Teraflops per rack" and "a single NVIDIA Tesla K40 chip costs around $4,000. ... If you have four fully loaded racks, that’s over 700 chips. We’re talking some serious power for serious cash." so that's about 3m for 1PFlop
  - m_mueller 12y ago
    4000 is only for the GPU though, not the whole node.
- collyw 12y ago
  I was reading this, thinking the numbers don't mean much (to me) without context. What does your average cluster cost and how many petaflops would you get for you buck there?
Aardwolf 12y ago
But does it come in a beautiful form factor with bench seats?
ajdecon 12y ago
The Cray brochure for this system has more information, including detailed specs: http://www.cray.com/Assets/PDF/products/cs/CrayCS-Storm.pdf http://www.cray.com/Assets/PDF/products/cs/CrayCS-Storm.pdf A few fun points from the brochure: - The internal network ("interconnect") is Infiniband in either a torus or fat tree topology - Up to 512 GB RAM per node - Power usage is up to 63 kW per rack - You have the option of including liquid-cooled rear door heat exchangers, if your datacenter air cooling isn't up to the task ;-) - Compute nodes can have disks installed, but the system supports diskless booting - Each compute node weighs 88 lbs One thing to remember about these machines is that the internal high-bandwidth, low-latency network is a big part of why they get such good performance. Most applications that run on these machines are long-running computations consisting of many processes spread across the entire machine, with lots of inter-process communication throughout the lifetime of the job. Think weather prediction, simulations of fluid dynamics, or complex mechanical systems. (I work on similar systems to these every day. Huge amounts of fun, but quite different needs than your typical web service, or even most Hadoop-ish data analysis problems.)
marklittlewood 12y ago
So going to get one as my new laptop. Might have to invest in a wheelbarrow though.

In what cases would this be better than AWS/other IaaS? Seems like a steep investment. Do labs use stuff like this?

jff 12y ago
Real scientific computing needs high-bandwidth low-latency networks.
- kolencherry 12y ago
  Microsoft Azure actually offers InfiniBand for connectivity, which is interesting. I'm not sure how many people are using their HPC-oriented offering.
- collyw 12y ago
  Go on then, let me know what "real scientific computing" is. I work with bioinformaticians, which is an area with lots of research going on. I don't see much that is complex there. Split up your jobs and send them to a cluster. I take it that's not "real scientific computing" though?
  - sp332 12y ago
    From the article: planet-wide weather forecasting, seismic measuring for oil exploration, financial analysis or crash simulations. You can't split up a physical simulation.
    - jff 12y ago
      You actually can split up a physical simulation. Imagine you're trying to model heat diffusion through a piece of metal. You split the chunk of metal up into many small pieces and model temperature in each one. Each node in your cluster is simulating the behavior of some portion of the model; at each timestep, nodes simulating chunks that border each other must swap the state of their "edges", sharing the outer edges of their own state with their neighbors. Edit: unless you meant that you can't split up a physical simulation on loosely-coupled nodes and still get good performance. Then, well, yeah.
      - sp332 12y ago
        Well I guess the simulation would have to be broken up to run on multiple GPUs at once anyway. But even if you had a good interconnect, the GPUs in this box (K40) have almost 2x as many processors and 3x as much RAM. (Edit: compared to AWS GPU instances.) If you're doing a 3D sim, each node needs to communicate with 26 others, so just having 1/3 as many nodes is going to help a lot.
  - jff 12y ago
    Split up your jobs how? Is the job embarrassingly parallel? Because if it isn't, and most things aren't, it's going to require significant programmer overhead to simply "split up your jobs". And then once you've split up the job, the different portions are going to need to exchange their in-progress results, because most real scientific computing programs involve iterating until either a certain number of timesteps have passed or a certain residual is reached. I mean, I only write HPC programming frameworks for scientific computing as my job, what do I know. But if you're doing anything more complex than map-reduce, a high-speed interconnect is essential, and even in mapreduce it's pretty handy.
sp332 12y ago
Approximating numbers from https://arstechnica.com/information-technology/2013/11/18-hours-33k-and-156314-cores-amazon-cloud-hpc-hits-a-petaflop/ https://arstechnica.com/information-technology/2013/11/18-ho... a petaflop on AWS would cost $20k per day. So considering hardware alone, this would be cheaper in 50 days. Once you add in power and cooling for this device, and the "reserved instance" discount from Amazon, it would take longer to break even. But probably not more than a few months. Edit: it answers this in the article.
m_mueller 12y ago
On the opposite, I've never heard of a lab that does use cloud computing for HPC. Only for highly variable workloads with low networking and privacy requirements something like AWS would make sense.
jostmey 12y ago
Most labs could not afford this. But I could see universities buying lots of these for shared, general purpose calculations. I myself would not mind having access to a rack of them!

I'm a bit confused, is the described 1PFlop system really only one million dollar, including cooling and power? That would be amazingly good value. The single digit PFlop systems I know of have cost two to four orders of magnitude more than this, even when powered by GPUs.

leephillips 12y ago
The computers cost 10 billion dollars? Who buys them? That would be above the noise floor of even the NSA's budget.
- lutusp 12y ago
  That might be true, but to establish it, we would have to know the NSA'a budget (which is classified).
- m_mueller 12y ago
  Sorry, I meant one to three orders of magnitude. The most expensive computer I know of is the Japanese K computer at ~ 1.25 billion and 10 something Petaflops. This was mostly politically motivated (Japanese chips) and too costly for what it does IMO. A typical new supercomputer in the Top 500 today is in the order of 100 million.
MrVitaliy 12y ago
from the article "250 Teraflops per rack" and "a single NVIDIA Tesla K40 chip costs around $4,000. ... If you have four fully loaded racks, that’s over 700 chips. We’re talking some serious power for serious cash." so that's about 3m for 1PFlop
- m_mueller 12y ago
  4000 is only for the GPU though, not the whole node.
collyw 12y ago
I was reading this, thinking the numbers don't mean much (to me) without context. What does your average cluster cost and how many petaflops would you get for you buck there?

Aardwolf 12y ago

But does it come in a beautiful form factor with bench seats?

ajdecon 12y ago

The Cray brochure for this system has more information, including detailed specs: http://www.cray.com/Assets/PDF/products/cs/CrayCS-Storm.pdf http://www.cray.com/Assets/PDF/products/cs/CrayCS-Storm.pdf A few fun points from the brochure: - The internal network ("interconnect") is Infiniband in either a torus or fat tree topology - Up to 512 GB RAM per node - Power usage is up to 63 kW per rack - You have the option of including liquid-cooled rear door heat exchangers, if your datacenter air cooling isn't up to the task ;-) - Compute nodes can have disks installed, but the system supports diskless booting - Each compute node weighs 88 lbs One thing to remember about these machines is that the internal high-bandwidth, low-latency network is a big part of why they get such good performance. Most applications that run on these machines are long-running computations consisting of many processes spread across the entire machine, with lots of inter-process communication throughout the lifetime of the job. Think weather prediction, simulations of fluid dynamics, or complex mechanical systems. (I work on similar systems to these every day. Huge amounts of fun, but quite different needs than your typical web service, or even most Hadoop-ish data analysis problems.)

marklittlewood 12y ago

So going to get one as my new laptop. Might have to invest in a wheelbarrow though.