5 ms·

There's plenty of reasons. Here's a bullet list with some thoughts. * Using virtual machines isn't a bad idea for cases where you have a one off calculation fo

There's plenty of reasons. Here's a bullet list with some thoughts. * Using virtual machines isn't a bad idea for cases where you have a one off calculation for a paper that you need to publish. But for some groups they need to continually model and predict stuff. For example, weather forecasting needs to be done every day. * Hadoop MapReduce and friends are extremely slow. Virtualization slows things down as well. The person in your article was using software called Shrodinger which isn't MR. * Often, MapReduce performance is bottlenecked by the shuffle stage. This requires a lot of data to be passed around the network. Ethernet and http connections are not ideal for this if performance is your goal (though they are good for resilience). RDMA (remote direct memory access) with Infiniband is much better (offloading lots off the cpu, less chatter, etc). There exist RMDA plugins for Hadoop but they're of ymmv quality. * MR isn't an ideal paradigm for every calculation. MR comes from search indexing (google) and is useful in areas where you need to look at all the data and moving the data around is more intense than the calculations. If you can filter, feature extract, or otherwise reduce the amount of data being moved around, the benefits of using HPC style clusters win out. Places where you must look at all the data: Indexing, Adversary detection (insurance fraud, intruder detection), format change (mp4->webm, gz, json->bson), ingestion, backups. If you're not doing these things you might want to consider whether MR is right for you. * Even in search indexing, deep learning is becoming en vogue and this is extremely computationally expensive. This means the cost of moving the data around is again taking a back seat. * A large part of the benefits of Hadoop style MR was having local data. But network speeds have outpaced drive speeds so data locality is less of an issue. Coupled with deep learning, there's less need for HDFS type systems. GPFS or Object storage is fine. Also, the page says it's only a theoretical petaflop: "While Stowe says the Amazon cluster hit 1.21 petaflops, that's the theoretical peak speed rather than the actual performance. In the Linpack benchmark used to test supercomputer speeds, the theoretical peak is always reported, but the real-world results are what count when ranking the world's fastest machines."

conjectures 10y ago
The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage in industry. There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.
- fnord123 10y ago
  It depends on the problem you want to solve. Working hard on detail oriented work and not giving up until you're successful is always a transferrable experience. ut a lot of the data analysis systems at the moment seem to want an SQL interface. It's a good idea because it's a stable language that lots of people already know. But your work with MPI won't really help you directly. >There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases. It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it. Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.
logicallee 10y ago
(not OP) this was very interesting - thanks for writing it up!
dankohn1 10y ago
This is a fantastic answer. Could you or anyone else please point me to any examples of trying to run high-resolution weather forecasting, such as COAMPS [0] or WRF [1] on commodity cloud infrastructure instead of HPC? [0] http://www.nrlmry.navy.mil/coamps-web/web/home http://www.nrlmry.navy.mil/coamps-web/web/home [1] http://www.wrf-model.org/index.php http://www.wrf-model.org/index.php And fnord123, could you please email me?
- scott_s 10y ago
  Googling turns up some hits, but not much. The basic issue is that weather prediction is one of those problems that can always use more computational power. It is even a known problem that the models used by the National Weather Service are inferior to the models used by agencies in Europe, and one of the things holding us back is raw computational power. From "Why Isn't the U.S. Better at Prediction Extreme Weather?" (http://www.nytimes.com/2016/10/23/magazine/why-isnt-the-us-better-at-predicting-extreme-weather.html?_r=0 http://www.nytimes.com/2016/10/23/magazine/why-isnt-the-us-b...): While Mass is the most outspoken on the subject, many experts insist that if the Weather Service wants to meaningfully improve its predictions, it must employ a technique called ensemble forecasting. The basic premise is either to tweak the physics equations or to make repeated changes to a model’s variables: You might bump up the temperature slightly, for example, and then run the model again. After a half-dozen or so reruns, you get a set, or “ensemble,” of forecasts that can be compared with one another. When all the forecasts in an ensemble agree, it’s a reasonably sure bet that the predictions will pan out. Nobody I’ve spoken to doubts the superiority of ensembles. Yet they haven’t been widely adopted in the United States at the resolution required to forecast localized, or “mesoscale,” events — specifically, thunderstorms, flash floods and tornadoes — because high-resolution ensembles require more computing power than the National Weather Service can currently provide. Higher-resolution ensembles translate to greater accuracy in the same way that HDTVs are clearer than analog sets. I met with a scientist at the National Center for Atmospheric Research in Boulder who showed me a prototype mesoscale ensemble for the United States. But at the moment, he can’t exploit its full potential because the supercomputing cluster at the Weather Service simply couldn’t handle the load. HPC systems are essentially designed for this type of computational problem, and we still want more powerful versions of it. For this reason, I find it unlikely that commodity cloud infrastructure - which was not explicitly designed for this type of problem, and it almost assuredly going to perform worse - will have much success in this area. At least in the near future.

conjectures 10y ago

The points about Hadoop are interesting. I write code for a university HPC, and was wondering how these skills would compare to methods that seem to be the rage in industry. There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases.

fnord123 10y ago
It depends on the problem you want to solve. Working hard on detail oriented work and not giving up until you're successful is always a transferrable experience. ut a lot of the data analysis systems at the moment seem to want an SQL interface. It's a good idea because it's a stable language that lots of people already know. But your work with MPI won't really help you directly. >There've been several points where getting a detail right has seriously improved performance. These made me wonder how well the generic map reduce approach could really deliver on performance across use cases. It can't. A big selling point of Big Data systems is that you can analyse unstructured data that you haven't seen before and quickly dig in. But then if you try to reproduce the benchmarks you basically have to perform a parameter sweep to get the performance numbers that people are advertising. Maybe there are some super experts who can divine the parameters for their calculations ahead of time, but I've never been able to do it. Then on the HPC side you have people who look at the Big Data benchmarks and then take is as a challenge by writing some specific code to tackle the problem which trivially smashes the Big Data solution. Usually using a clever encoding. But this misses the point -but doesn't really since I think everyone was cheating/not-cheating anyway.

logicallee 10y ago

(not OP) this was very interesting - thanks for writing it up!

dankohn1 10y ago

This is a fantastic answer. Could you or anyone else please point me to any examples of trying to run high-resolution weather forecasting, such as COAMPS [0] or WRF [1] on commodity cloud infrastructure instead of HPC? [0] http://www.nrlmry.navy.mil/coamps-web/web/home http://www.nrlmry.navy.mil/coamps-web/web/home [1] http://www.wrf-model.org/index.php http://www.wrf-model.org/index.php And fnord123, could you please email me?

scott_s 10y ago
Googling turns up some hits, but not much. The basic issue is that weather prediction is one of those problems that can always use more computational power. It is even a known problem that the models used by the National Weather Service are inferior to the models used by agencies in Europe, and one of the things holding us back is raw computational power. From "Why Isn't the U.S. Better at Prediction Extreme Weather?" (http://www.nytimes.com/2016/10/23/magazine/why-isnt-the-us-better-at-predicting-extreme-weather.html?_r=0 http://www.nytimes.com/2016/10/23/magazine/why-isnt-the-us-b...): While Mass is the most outspoken on the subject, many experts insist that if the Weather Service wants to meaningfully improve its predictions, it must employ a technique called ensemble forecasting. The basic premise is either to tweak the physics equations or to make repeated changes to a model’s variables: You might bump up the temperature slightly, for example, and then run the model again. After a half-dozen or so reruns, you get a set, or “ensemble,” of forecasts that can be compared with one another. When all the forecasts in an ensemble agree, it’s a reasonably sure bet that the predictions will pan out. Nobody I’ve spoken to doubts the superiority of ensembles. Yet they haven’t been widely adopted in the United States at the resolution required to forecast localized, or “mesoscale,” events — specifically, thunderstorms, flash floods and tornadoes — because high-resolution ensembles require more computing power than the National Weather Service can currently provide. Higher-resolution ensembles translate to greater accuracy in the same way that HDTVs are clearer than analog sets. I met with a scientist at the National Center for Atmospheric Research in Boulder who showed me a prototype mesoscale ensemble for the United States. But at the moment, he can’t exploit its full potential because the supercomputing cluster at the Weather Service simply couldn’t handle the load. HPC systems are essentially designed for this type of computational problem, and we still want more powerful versions of it. For this reason, I find it unlikely that commodity cloud infrastructure - which was not explicitly designed for this type of problem, and it almost assuredly going to perform worse - will have much success in this area. At least in the near future.