6 ms·

Serious if likely naive question: what's the use case for supercomputers these days, instead of just spinning up a cluster in your favorite cloud and running yo

Serious if likely naive question: what's the use case for supercomputers these days, instead of just spinning up a cluster in your favorite cloud and running your favorite MapReduce derivative on it? These guys pulled off a petaflop on AWS with 156k cores for $33k, and that was in 2013. [1] http://arstechnica.com/information-technology/2013/11/18-hours-33k-and-156314-cores-amazon-cloud-hpc-hits-a-petaflop/ http://arstechnica.com/information-technology/2013/11/18-hou...

deleted 10y ago
[deleted]
azag0 10y ago
Mostly heavily parallel calculations I guess, where communication matters. Diagonalizing dense matrices, solving partial differential equations. Mostly scientific or engineering purposes.
- azernik 10y ago
  It's actually the opposite - heavily parallelizable calculations can easily be split across physically separated machines. Supercomputers are necessary for computations that are hard to parallelize, ie those where there's a lot of data dependency between individual subproblems and data transfer between computing nodes is a bottleneck.
  - azag0 10y ago
    That's what I wrote. I didn't use the word "heavily" correctly.
metafunctor 10y ago
Moving data between processors is much faster in a specially built supercomputer.
fnord123 10y ago
There's plenty of reasons. Here's a bullet list with some thoughts. * Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day. * Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR. * Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality. * MR isn't an ideal paradigm for every calculation. MR comes from search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you. * Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat. * A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine. Also, the page says it's only a theoretical petaflop: "While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."
- conjectures 10y ago
  The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage in industry. There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
CmdrSprinkles 10y ago
Different kinds of applications. Think of it like public transportation versus owning a car. For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle. Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving. And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations. Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.
- dekhn 10y ago
  Note that many of us who run science codes think the interconnect is being overbuilt. I always got pushback because my code of interest, AMBER, didn't scale above 128 nodes on typical supercomputers due to the communication requirements. I switched to running N independent simulations and pooled the stats, getting far more high quality results. I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches. Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.
reacweb 10y ago
https://en.wikipedia.org/wiki/Psychohistory_(fictional) https://en.wikipedia.org/wiki/Psychohistory_(fictional) (keep it a secret, otherwise it stop working).
davidhyde 10y ago
Great question. I have a friend who used to work on the human brain project and I asked him a similar question. What surprised me is that the software that gets run on super computers is usually quite simple BUT the computational complexity is high or the volume of data is large. Fairly non-technical people (i.e. not necessarily computer scientists but maybe mathematicians) would typically submit small programs written in Python that would be automatically compiled into something that used the hardware to its maximum potential (instead of compiling it to something more portable). I believe that software running on super computers is closer to the wire than clustered computers. The operating systems are often very stripped down and custom too.
deleted 10y ago
[deleted]
rgbrenner 10y ago
pulled off a petaflop on AWS with 156k cores for $33k They ran it for 18 hours. So that's $44k/day for 1 petaflop. This supercomputer is 130 petaflops. So let's pretend it scales lineally.. that's $5.7m/day. In just 30 days on AWS, you could buy the $173m supercomputer in the article. This isn't a serious alternative.
- aries1980 10y ago
  That 130 petaflop is for specific applications, not in general computation.
Veratyr 10y ago
Others have mentioned practical issues like performance but there's also cost. When you're actually using a lot of capacity and doing so pretty consistently, Amazon is extremely expensive compared to dedicated hardware.
visarga 10y ago
> what's the use case for supercomputers these days Anime rendering? pretty rad action scenes!
davidf18 10y ago
HPC at centers like NCSA have a much lower latency interconnect than what is used by Amazon, Google, etc. and there are some computational problems that require the lower latency interconnect. Newer Intel chips will be integrating low-latency interconnects so in the future, Amazon, Google might be adopting low-latency interconnect computing.
sedachv 10y ago
For one, getting data into AWS is a problem for scientific computing. For example, the Large Hadron Collider experiment generates 27 terabytes of data per day and has its own network: https://en.wikipedia.org/wiki/Worldwide_LHC_Computing_Grid https://en.wikipedia.org/wiki/Worldwide_LHC_Computing_Grid

deleted 10y ago

[deleted]

azag0 10y ago

Mostly heavily parallel calculations I guess, where communication matters. Diagonalizing dense matrices, solving partial differential equations. Mostly scientific or engineering purposes.

azernik 10y ago
It's actually the opposite - heavily parallelizable calculations can easily be split across physically separated machines. Supercomputers are necessary for computations that are hard to parallelize, ie those where there's a lot of data dependency between individual subproblems and data transfer between computing nodes is a bottleneck.
- azag0 10y ago
  That's what I wrote. I didn't use the word "heavily" correctly.

metafunctor 10y ago

Moving data between processors is much faster in a specially built supercomputer.

fnord123 10y ago

There's plenty of reasons. Here's a bullet list with some thoughts. * Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day. * Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR. * Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality. * MR isn't an ideal paradigm for every calculation. MR comes from search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you. * Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat. * A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine. Also, the page says it's only a theoretical petaflop: "While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."

conjectures 10y ago
The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage in industry. There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.

CmdrSprinkles 10y ago

Different kinds of applications. Think of it like public transportation versus owning a car. For highly parallelizable and distributable applications (anything where mapreduce makes sense), commodity clouds are awesome. Just like relying on the subway in a city. You'll have a bit of a hassle when you need to go shopping, but your normal use case is such that it is worth the hassle. Now let's say you have a code where your tasks require much tighter coupling due to communication (basically "Science codes"). In these cases, what you want is the interconnect. You want to be able to get a physical block of nodes in a very short amount of time for a somewhat long amount of time. In terms of cars, imagine that you live outside of a city or tend to do a lot of shopping or long distance driving. And that is basically the logic here. It sounds like the Japanese rig is intended to be similar to the US's ORNL Titan in that it is primarily a capacity machine for science in general. So people with those kinds of jobs can buy time and run their large simulations with ease while still relying on stuff like AWS for postprocessing and the like. In that case, even though this is a 130 petaflop machine, it will likely only ever run one or two jobs at that scale and will instead operate as a "cloud" with a focus on large contiguous allocations. Whereas the other form of SuperComputer are the ones meant for large jobs for a small subset of users. Those are the ones that tend to be owned directly by governments and involve a fair amount of effort to get access to. Those run a mixture of the above kinds of jobs (for the target community) as well as actual large scale simulations. As for what they are simulating: Think about what would need really detailed simulations using codes and data that probably should never be near Amazon's servers.

dekhn 10y ago
Note that many of us who run science codes think the interconnect is being overbuilt. I always got pushback because my code of interest, AMBER, didn't scale above 128 nodes on typical supercomputers due to the communication requirements. I switched to running N independent simulations and pooled the stats, getting far more high quality results. I've pointed this out to people in SC community, they don't really care. Either they have a problem that only runs on a SC (IE, a science code with strong scaling) or they're just not interested in trying other approaches. Anyway, a supercomputer for deep learning makes sense. The communication patterns in deep learning can be very intense, with a lot of n2 calls, and so a good interconnect might speed up the code. However, tensorflow etc aren't designed for MPI SC architectures.

reacweb 10y ago

https://en.wikipedia.org/wiki/Psychohistory_(fictional) https://en.wikipedia.org/wiki/Psychohistory_(fictional) (keep it a secret, otherwise it stop working).

davidhyde 10y ago

Great question. I have a friend who used to work on the human brain project and I asked him a similar question. What surprised me is that the software that gets run on super computers is usually quite simple BUT the computational complexity is high or the volume of data is large. Fairly non-technical people (i.e. not necessarily computer scientists but maybe mathematicians) would typically submit small programs written in Python that would be automatically compiled into something that used the hardware to its maximum potential (instead of compiling it to something more portable). I believe that software running on super computers is closer to the wire than clustered computers. The operating systems are often very stripped down and custom too.

deleted 10y ago

[deleted]

rgbrenner 10y ago

pulled off a petaflop on AWS with 156k cores for $33k They ran it for 18 hours. So that's $44k/day for 1 petaflop. This supercomputer is 130 petaflops. So let's pretend it scales lineally.. that's $5.7m/day. In just 30 days on AWS, you could buy the $173m supercomputer in the article. This isn't a serious alternative.

aries1980 10y ago
That 130 petaflop is for specific applications, not in general computation.

Veratyr 10y ago

Others have mentioned practical issues like performance but there's also cost. When you're actually using a lot of capacity and doing so pretty consistently, Amazon is extremely expensive compared to dedicated hardware.

visarga 10y ago

> what's the use case for supercomputers these days Anime rendering? pretty rad action scenes!

davidf18 10y ago

HPC at centers like NCSA have a much lower latency interconnect than what is used by Amazon, Google, etc. and there are some computational problems that require the lower latency interconnect. Newer Intel chips will be integrating low-latency interconnects so in the future, Amazon, Google might be adopting low-latency interconnect computing.

sedachv 10y ago

For one, getting data into AWS is a problem for scientific computing. For example, the Large Hadron Collider experiment generates 27 terabytes of data per day and has its own network: https://en.wikipedia.org/wiki/Worldwide_LHC_Computing_Grid https://en.wikipedia.org/wiki/Worldwide_LHC_Computing_Grid