8 ms·
Supercomputer quietly puts U.S. weather resources back on top
- matt2000 11y agoIs there anyone here who works on weather models who would be able to explain why custom supercomputers are still needed vs. large commodity clusters? Thanks!
- GFK_of_xmaspast 11y agoI don't do weather, but probably it has a lot to do with inter-node bandwidth.
- mispeled 11y agoJust a guess... high fidelity weather modelling algorithms are highly constrained by node-to-node communication latency. So their supercomputer probably has some networked-direct-memory-access or another non-standard setup tuned toward passing data between adjacent nodes quickly.
- stephengillie 11y agoThe branding is still very valuable. This article makes them sound like common blades, ready for virtualization or containerization: http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_burst_buffer_tech_to_supercomputer_line http://m.theregister.co.uk/2014/09/30/cray_adds_haswell_and_...
- paulmd 11y agoModern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars. The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a specific or reconfigurable topology. Something like InfiniBand, where you can ensure very consistent, very favorable bounds on your latency and throughput. So think of it as being a "custom installation" or "custom cluster" instead. Nowadays there really are only a few real choices left in high-performance computing hardware: CPU vs GPU, processor architecture (eg POWER vs x86), and interconnection. Everything else is more or less irrelevant. I have to admit though, I've always wanted to see what you'd get if you built a Cray-2 with GPU chips (eg Pascal) or Xeon Phi as processing modules. The sheer density of chips Cray managed to cram in that case is impressive.
- thrownaway2424 11y agoWhat kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.
- semi-extrinsic 11y agoFor a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (< 32 kbytes) packet is less than 10 microseconds, down towards 1 microsecond for a 8 byte packet. Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs-dual-rail-qdr.html http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...
- bashinator 11y agoLess than one microsecond. https://en.wikipedia.org/wiki/InfiniBand#Performance https://en.wikipedia.org/wiki/InfiniBand#Performance
- mturmon 11y agoFor more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.
- superpc 11y agoDoes this count, just curious. It is a large single machine not cluster. https://www.sgi.com/pdfs/4555.pdf https://www.sgi.com/pdfs/4555.pdf
- sithadmin 11y agoA cursory glance at that PDF suggests the exact opposite of your claim.
- mturmon 11y agoI think the comments nearby about high communications bandwidth required by dense systems of equations are true -- basically they distinguish a tuned, integrated system sold by a vendor like Cray from a row of racks with commodity systems. The point I wanted to add is that ECMWF, generally acknowledged to be the leader in NWP, also uses a pair of large Cray supercomputers (http://www.ecmwf.int/en/computing/our-facilities/supercomputer http://www.ecmwf.int/en/computing/our-facilities/supercomput...). Note that the Cray XC30 uses Xeon processors and a custom interconnect. The old ECMWF system used IBM Power chips.
- mrow84 11y agoYes, it is the interconnects that are important. The UK Met Office has also recently upgraded to a Cray XC40 [0], and there are other XC30 systems in the UK NSS [1] and the US NERSC [2], both of which are used for weather/climate models, amongst other things. [0] http://www.metoffice.gov.uk/news/releases/archive/2014/new-hpc http://www.metoffice.gov.uk/news/releases/archive/2014/new-h... [1] http://www.archer.ac.uk/about-archer/ http://www.archer.ac.uk/about-archer/ [2] http://www.nersc.gov/systems/edison-cray-xc30/ http://www.nersc.gov/systems/edison-cray-xc30/
- al_biglan 11y agoI work for the company that provided the storage for this. Our product is not a collection of commodity components but instead something very much more like an "appliance" that provides the performance at scale that NOAA needs. In general, one of the trends we see/fight is the "Hey, we can get components ourselves and use Lustre to build what we need" And.... they are spot on right. What you need to consider doing this is the amount of maintaining and administering that comes with this solution. The trade-off we see is that some groups, "Academia" for example, will prefer the cheaper solution because they have Grad Students, Post Docs and Undergrad research folks to help make the people cost free. Others, who just want to buy something that works in their infrastructure without adding the people costs would prefer to buy something ready made that "just works" Bottom line: If you are using your system to get work done, the trend is to spend the money for reliability and availability. If you want to save money, it comes at a price (but is certainly do-able) Related: Look at Apple, Google, Facebook, Amazon, Microsoft... they are large enough to build their custom systems themselves. Not everyone operates at this scale. Individuals can buy a cheap PC and build a NAS... or they can buy one from Netgear, Buffalo, QNAP that "jsut works" at a price premium.
- khm 11y agoThere is no such thing as a 'large commodity cluster.' You can order a pile of commodity parts and spend the next five years fiddling with it as you watch your metrics fall into the toilet, or you can contract with a systems integrator with domain experience to make sure that you can actually accomplish the work you set out to do. When you're moving an established set of software from a known environment into a new cluster, it helps to have a wide array of technical experts and SLAs in place to ensure that the environment -- here meaning the physical compute infrastructure, storage service, and software installation -- is reliable and maintainable, so that your team can focus on their core mission -- in this case, the modeling software. "Chaos monkey" works fine when you're dealing with random cloudstuff and you are in a position to duck-punch your production code. When you have a large dataset that you need to process reliably and predictably, the mantra shifts to "fail never", and that requires engineering up front in combination with carefully-planned maintenance. Full disclosure: I've worked on this specific supercomputer. While I'm not on the admin team in question, I do work with them occasionally and I have a lot of respect for them -- their job is not an easy one.
- cryptoz 11y agoIt's a very exciting time in weather forecasting. We're getting all kinds of new observations available, from cheap LEO satellites to smartphone sensors to smart umbrellas and everything else. And faster computers! And more parallel computers! Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Weather forecasting accuracy is still constrained by both the available observations and the ability to run the forecast models. This new supercomputer won't be nearly enough to process all the new data that will arrive in the next few years, so the pendulum will swing back to the bottleneck being computing power soon enough. I think the most exciting things will happen over the next 5-10 years, as we end up with another 2-3 order of magnitude increase in the available observations to the models and we'll build new supercomputers to process them. I bet we can get incredible improvements, things like reliable 2-week weather forecasts.
- reubenmorais 11y ago> Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Sadly, until we have a breakthrough in battery technology, all of those will be parallel processors programmed to be asleep as often as they can.
- cryptoz 11y agoSure, that's true - but they can just as happily run their GPUs if they are plugged in and charging without any battery issues. Although for the moment you are right, they aren't that useful as a powerful distributed supercomputer. They ARE, however, really really good at collecting live weather data and sending that to the supercomputer to process. One step at a time.
- nradov 11y agoSmart phones are good at collecting live barometric pressure data but not much else. Temperature sensors aren't very useful because they are poorly calibrated, you can't reliably tell whether someone is outdoors, and the reading can be thrown off by the heat generated by the device or the person holding it. UV sensors are seldom pointed in the right direction. And I haven't seen a phone with a wind speed or precipitation sensor yet.
- nxzero 11y agoMaybe it's me, but having the backup in Orlando sounds like a bad idea. Anyone able to comment on the logic behind this?
- aaronbrethorst 11y agoI'd assume that one of Florida's senators, Bill Nelson or Marco Rubio, was responsible for ensuring that the backup system landed in their state. One person's pork is another's 'crucial jobs and infrastructure earmark.'
- chm 11y agoCould they have built it underground?
- munificent 11y ago"Underground" isn't very feasible in Florida. Most of the state is effectively a sandbar with a very high water table. Underground construction is uncommon.
- nradov 11y agoAnd even a slight sea level rise may make large parts of Florida effectively uninhabitable due to repeated flooding. If you want to build a reliable computing facility there then put it on stilts, not underground. http://www.newyorker.com/magazine/2015/12/21/the-siege-of-miami http://www.newyorker.com/magazine/2015/12/21/the-siege-of-mi...
- munificent 11y agoOrlando is 82' above sea level. If you're worried about it flooding, you should probably spend less time building stilts and more time dealing with the fact that Delaware no longer exists, nor does most of New Jersey, New York City, large swathes of the Eastern seaboard, 75% of Louisiana, the Texas coast all the way in to Houston, much of Los Angeles, and all of Sacramento. On the other hand, it will be nice to splash in the ocean at Joshua Tree National Park, and I'm sure Arkansas residents will be happy to finally be able to own beachfront property.
- dhsjhdjs 11y agoFor a newbie in this domain, how do we start doing our own toy forecasts (lets say for Bay area)?
- privong 11y ago> For a newbie in this domain, how do we start doing our own toy forecasts (lets say for Bay area)? There are some available weather modeling packages, such as WRF[0], as a mesoscale (few km to 1000 kms) model, that one can download and run. You'd obviously need to find a resource for the appropriate input data. I don't know how easy it is to get up and running, though. [0] http://www.wrf-model.org/index.php http://www.wrf-model.org/index.php
- goodbyegti 11y agoA really nice intro is this classic paper on numerical integration of the equation for barotropic vorticity: http://mathsci.ucd.ie/~plynch/eniac/CFvN-1950.pdf http://mathsci.ucd.ie/~plynch/eniac/CFvN-1950.pdf It's something you can solve in a few lines of python and run with minimal compute resources. A nice thing to try is to feed in some data from somewhere like below and watch what happens as you integrate it forwards in time. http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalys...
- acd 11y agoGreat that there is a more powerful computer helping predict the weather! One should know that weather is a chaotic system and after three days its very hard to predict the weather correctly. This is known by the lorenz equations. http://www.uvm.edu/~cdanfort/research/danforth-bates-thesis.pdf http://www.uvm.edu/~cdanfort/research/danforth-bates-thesis.... Here is the UK met office. "Most of the time the atmosphere behaves rather like the lower-left picture where we can predict with confidence for a few days and have to use probabilities thereafter. " http://research.metoffice.gov.uk/research/nwp/ensemble/concept.html http://research.metoffice.gov.uk/research/nwp/ensemble/conce...
- kkylin 11y agoIndeed! though the constraints that chaos imposes on predictability is a subtle question. See, e.g., ftp://mana.soest.hawaii.edu/pub/rlukas/LSASI/ENSO/Prediction/Predicability_a_Problem_2006.pdf
- jordache 11y agoWish they would stop using the library of congress' collection as a reference... I know it's big.. no idea how big.. it's not tangible to me.. never visited.. I'm sure most americans have not visited.
- superuser2 11y agoYou do not get an idea of how big it is by visiting; the stacks are off limits and much of the material must be ordered in from suburban warehouses.
- semi-extrinsic 11y agoThey probably also have most of the bookshelves on rails? The local uni library keeps their old books on a system like this, it's really quite cool. The room can fit, say, 54 bookshelves tight-packed with no room to enter anywhere. So they put 52 shelves in on rails, and between each pair there is a wire going across. Disconnect a wire between two shelves, and all the shelves slide apart so you can enter where you opened the wire.
- superuser2 11y agoI'd estimate most university libraries have a system like this, though the "disconnect a wire" part is strange. Usually they're just a button on the shelf.
- semi-extrinsic 11y agoI think the wire is a safety feature to avoid accidentally crushing people. If you try to disconnect a second wire, nothing happens, and you have to go find the first disconnected wire and reconnect it. Then you will see if there is still someone in there. (When you're finished in a row, you're supposed to reconnect behind you when you leave.)
- tyingq 11y agoThis helped a little: "The Library of Congress is the largest library in the world, with more than 162 million items on approximately 838 miles of bookshelves. The collections include more than 38 million books and other print materials, 3.6 million recordings, 14 million photographs, 5.5 million maps, 7.1 million pieces of sheet music and 70 million manuscripts."[1] [1] https://www.loc.gov/about/fascinating-facts/ https://www.loc.gov/about/fascinating-facts/
- SeanDav 11y ago> "Mass was highly critical of the federal government's lagging computing capacity in recent years.." Perhaps he should have been having a chat with the NSA boys...
- Sniffnoy 11y agoI'm bothered by how the article says "three years ago European models delivered a blow to the U.S. weather apparatus". Better models by Europeans do not make the models by Americans any worse. Improved models help everyone.
- CWuestefeld 11y agoThat was the first thing I noticed, too. The fact that the Euro computers got Hurricane Sandy right was NOT a bad thing - it helped us. There is no reason to look at this in any way adversarially. I guess it touches a nerve, because of the similar silly comments elsewhere, worrying about how the USA is "losing" if, say, the Chinese economy's size exceeds the size of our own. Other folks doing well doesn't make us worse. When it happens we should be happy: other people are doing well, and we've got an opportunity to learn and improve too!
- barney54 11y agoAnd even with the new computer the European model is still better at mid range forecasts.
- akie 11y agoI talked to a (very) senior scientist at the World Meteorological Organization not too long ago. He was of the opinion that the U.S. weather models were about 5 to 10 years behind on the European models, and that this was caused by structural underfunding over longer periods of time (i.e. past decades). As I understood it these weather models are basically humongous software programs that are developed in house, but based on publicly available science and fed by data from weather stations. Does your software become better if you run it on a faster computer? It will certainly help, but he instilled on me the impression that the US had a bit more catching up to do than just buying a new supercomputer.
- barney54 11y agoHe is correct. The European model still outperforms the US (GFS) model: https://twitter.com/RyanMaue/status/700216724067586049 https://twitter.com/RyanMaue/status/700216724067586049
- kuya 11y agofor anyone interested, this paper has a nice background on numerical weather prediction http://www.elsevierscitech.com/emails/physics/climate/the_origins_of_computer_weather_prediction.pdf http://www.elsevierscitech.com/emails/physics/climate/the_or...
- mediocrejoker 11y agoI got about 3 pages into this and felt like I was following the discussion until the discussion of an incorrect prediction result that: > In fact, the spurious tendencies are due to an imbalance between the pressure and wind fields resulting in large amplitude high frequency gravity wave oscillations. Since this paper was published within the last 20 years, I can't imagine what they were referring to by 'gravity waves' Do you know what is meant by this statement?
- cos2pi 11y agoGravity waves are waves in a fluid that obtain their restoring force from their buoyancy relative to the surrounding fluid. See [0] for the gory details. Some examples of gravity waves in the atmosphere: [1] [2] [0] http://glossary.ametsoc.org/wiki/Gravity_wave http://glossary.ametsoc.org/wiki/Gravity_wave [1] http://cimss.ssec.wisc.edu/goes/blog/archives/2051 http://cimss.ssec.wisc.edu/goes/blog/archives/2051 [2] https://www.youtube.com/watch?v=yXnkzeCU3bE https://www.youtube.com/watch?v=yXnkzeCU3bE
- Faint 11y agoAre you mixing gravity waves and gravitational waves? Gravity waves are any waves in media in which gravity acts as restoring force, like waves in ocean, or waves between layers with different densities in atmosphere, or, AFAIU, whole atmosphere (think of atmosphere as a big shallow pool, with air sloshing around in the pool instead of some liquid). https://en.wikipedia.org/wiki/Gravity_wave https://en.wikipedia.org/wiki/Gravity_wave
- divreg 11y agoIt's funny that you point this out, since some people in the field of astrophysics are quick to correct others when "gravity waves" are used when they really mean "gravitational waves", and sometimes vice versa. Basically, gravity waves (or g-waves) are a type of perturbation in stratified media where the restoring force on sound waves is buoyancy. The other scenario is where the restoring force is pressure, which are pressure waves (or p-waves). Gravity waves are important in not just planetary atmospheres but stellar media as well, particularly in the outer region of stars. A popular candidate for /gravitational/ wave sources is binary neutron star systems, so the two words aren't interchangeable there since they refer to two very different phenomena!
- ziedaniel1 11y agoWhat's the biggest bottleneck for better forecasting: better observations, better models, or better supercomputers?
- jackreichert 11y agoWell, this is happening just in time for all the weather patterns -- as we know them -- to change due to global warming. I guess it'll be not unlike understanding the universe according to Douglas Adams...
- barney54 11y agoEven though the US now has more computing power, the Euro model is still doing a better job at forecasting: https://twitter.com/RyanMaue/status/700216724067586049 https://twitter.com/RyanMaue/status/700216724067586049
- ansonhoyt 11y agoCliff Mass also blogged [1] about the NOAA supercomputer. While a day after the USA Today's article, many folks appreciate his perspective. He's posted on this several times in the past year [2]. [1] http://cliffmass.blogspot.com/2016/02/the-national-weather-services-new.html http://cliffmass.blogspot.com/2016/02/the-national-weather-s... [2] https://www.google.com/webhp?#q=supercomputer+site:+cliffmass.blogspot.com https://www.google.com/webhp?#q=supercomputer+site:+cliffmas...