3 ms·
I think the only thing that most HPC sites use which wouldn't be found in cloud infrastructure is the interconnects and a parallel file system. Back in 2004 I g
by fnord123 10y ago
I think the only thing that most HPC sites use which wouldn't be found in cloud infrastructure is the interconnects and a parallel file system. Back in 2004 I guess HPC sites were often using PPC on AIX. That's just gone. It's almost all Linux on x86_64 now aside from some interesting machines in Japan and China.
So the distinction of "commodity" that Google made in the original map reduce paper in 2004 is gone. And the benefits of data locality are all but gone. The paper was great when it came out but it's long in the tooth since the hardware of today is different.
To answer your actual question, I've never heard of anyone running WRF on cloud infrastructure. It doesn't have to be a terrible idea. If you're trying to come up with a competing model then the agility you get might be needed. i.e. being able to roll onto new hardware as it becomes available instead of having to work with budget cycles for a hardware refresh lets you move from tesla k80 to titans whenever AWS decided to make them available.
Searching around, I see Glenn Lockwood did some benchmarks in 2013 with AWS:
https://glennklockwood.blogspot.be/2013/12/high-performance-virtualization-sr-iov_14.html https://glennklockwood.blogspot.be/2013/12/high-performance-...
This lot seem to have done some more benchmarking in 2014 (tldr 44% overhead on VMs - but take the time to read it since glockwood is always a good read):
https://www.researchgate.net/publication/301407792_Performance_Characterisation_and_Evaluation_of_WRF_Model_on_Cloud_and_HPC_Architectures https://www.researchgate.net/publication/301407792_Performan...
I think the prevailing issue is that latency on virtualized nodes is poor so you have to work around it. Maybe fat nodes with gpgpu can overcome the issues.
If you need to talk to me privately then you can message me on reddit.