7 ms·

Hurricane Joaquin Forecast: Why U.S. Weather Model Has Fallen Behind

78 points by jsm386 11y ago

rch 11y ago
I only have a few unrelated anecdotes to draw from, but my impression is that there is an outsized political influence on where in the US the currently allocated R&D funding is being spent.
jonknee 11y ago
When the party that controls Washington has not believing in climate change as a party platform I don't see a dramatic increase in budget for climate scientists coming anytime soon. Thankfully we can use the European model!
- happyscrappy 11y ago
  Please don't conflate climate and weather.
  - 67726e 11y ago
    Do the Republicans know the difference?
  - jonknee 11y ago
    The GOP has consistently wished to cut the budget of the National Weather Service: http://www.slate.com/blogs/weigel/2011/03/11/gop_s_continuing_resolution_cuts_funding_for_national_weather_service_fema.html http://www.slate.com/blogs/weigel/2011/03/11/gop_s_continuin...
    - deleted 11y ago
      [deleted]
  - Frondo 11y ago
    They aren't the same thing but they are related to one another, of course.
deciplex 11y ago
> Maybe it will be Joaquin’s false alarm, not another Hurricane Sandy, that gets America to make the G.F.S. great again. Or maybe it will prompt the usual suspects to "privatize" the whole thing i.e. cut funding under the assumption that private enterprise, or churches, or whatever the hell else, will pick up the slack.
- deleted 11y ago
  [deleted]
heimatau 11y ago
Anyone in this field says it's extremely complicated. NASA (or ESA) probably would solve this, if we didn't underfund them so much. To those advocating to privatizing, I don't see how the private sector could solve it by themselves. Just like McDonalds didn't land on the moon. It's extremely complicated. The task is going to require a lot more scientists and innovators to understand our weather patterns, it's my opinion that the best way to do that, is through the large governments of the world (not just USA's govt).
- karmelapple 11y ago
  Space exploration, nor the serving of food to those in space, was a goal of McDonald's. There was no sufficient market opportunity. However, is there a market opportunity for improved weather prediction? Who would benefit and make money?
  - heimatau 11y ago
    McDonalds is hyperbolic language. There is no market in space, even for SpaceX. Private incentives don't exist in a pure market fashion. Only the ultra-weathly would be the market and they are a small bunch (in the thousands of people) as opposed to the billions in the bottom/middle classes. Also, I wouldn't say the government is the market because that's inflated and not real (always a budget cut away from disappearing). Hence why private interests won't work. P.S. I think SpaceX has a difficult task to appeal to the public. When they can do that, on a regular basis, I'll change my position. Or if some weather company is trying to be disruptive with technology and innovation, then I'll modify my view. (But most of us know innovation in the private sector doesn't always reach the public)
  - greglindahl 11y ago
    ECMWF charges for their weather forecasts, which are more accurate than the free American ones. There are lots of small commercial weather forecasting firms who sell to people like securities traders. A single high-temperature day can reduce corn yield by 1%. Here's a startup that had a $1 billion exit to Monsanto: http://www.theverge.com/2013/10/10/4823004/monsanto-bets-the-farm-on-big-data http://www.theverge.com/2013/10/10/4823004/monsanto-bets-the... Their business is a combination of weather and climate. In short, there's a big market for this stuff.
deleted 11y ago
[deleted]
chimtim 11y ago
This was a fascinating read. I found the initial bit interesting that other models were ahead because they simply had more raw computational power, though it later adds that there may be other subtle differences.
- dhimes 11y ago
  They indicated that the other models were using the change of the atmospheric variables in their initial conditions. (Neumann boundary conditions?) The models may actually be fundamentally different.
aaronbrethorst 11y ago
If you live in the Pacific Northwest, the blog of the professor mentioned in the article, Cliff Mass, is a must-read: http://cliffmass.blogspot.com http://cliffmass.blogspot.com
- ZainRiz 11y ago
  Or if you want the abridged version check out the tldr feed: https://twitter.com/cliffmasstldr https://twitter.com/cliffmasstldr
- foofoo55 11y ago
  And if you live in the PNW, mariners (boats, not baseball) here find that the "superforecast" from windfinder.com is consistently far more accurate that the government forecasts, especially for wind on the water. The only big change to the government forecasts seems to have been putting some lipstick on their web pages, while the models don't seem to have changed in years.
lacker 11y ago
It seems quite strange if computing power is really the bottleneck here. What sort of architecture would you have for a weather prediction system where you would spend more on computing power than researchers' salaries?
- lsiq 11y ago
  There is a very good reason computing power is the bottleneck and it is because of how the weather has to be modeled. Weather models simulate the conditions in the cells of a lattice that is overlaid over geographic areas. To improve accuracy, the cells need to be shrunk to smaller sizes as they have been for a long time. However, every time the size of the cells in the lattices are halved you need 8 times the computing power to perform the calculations of the 3D space. Because of Moore's Law, it takes many years for it to make sense for weather centers to upgrade their infrastructure and see significant improvements in their predictions.
  - pinewurst 11y ago
    Cells also dynamically influence neighboring cells in all dimensions, making the highest performance, lowest latency interconnects essential. Interconnect pricing is a significant chunk of these large system costs. I occasionally hear statements wondering why weather modeling can't simply be distributed to desktop or cloud and can't, for some strange reason, get people to understand that connectivity is almost as important as computation itself for this.
    - metaobject 11y ago
      We have a few clusters at work that we use for numerical weather forecasting using the WRF weather model and some other custom applications. And, indeed, the InfiniBand interconnect was a hefty portion of the overall cost of the clusters. At one point there were people considering Gigabit Ethernet instead of IB, but thankfully it was shot down. The lower latency with IB has a greater impact on MPI-based applications than some people realize.
      - 11y ago
sandworm101 11y ago
Do those in charge actually want accurate predictions? The hurricane is going to go where it is going to go, the model has no impact on the realworld path. If we know the path with certainty then those in the path can take appropriate measures. But what is we enjoy those measures and want to do them regardless of where the storm will actually hit? What a storm might be approach the executive branch of the local government goes into emergency mode. Emergency funds are spent on preparations. Some government workers are sent home. Schools might be closed. Evacuation orders may be issued. Town mayors and, importantly, state governors all go on TV to talk about these preparations. The local executives get to do their lord-and-master thing. If the storm hits, they looks like heroes. If it doesn't, they still look like dutiful public servants concerned for the safety of their public. Some people enjoy a good disaster rehearsal, especially if they get to be the one in charge. Imho the private sector, specifically the insurance, travel and tourism industries are the ones who want accurate models. That isn't to say that prediction should be privatized. The government is probably in the better place to handle forecasting. But the demand on politicians for better models/funding arises from industry, not any noble desire to avoid widespread emergency declarations.
- ghaff 11y ago
  And if you do the disaster rehearsal thing too many times without anything happening, i.e. cry wolf, the next time no one pays any attention, Katrina or Sandy hits, and a lot of people die that didn't need to.
  - sandworm101 11y ago
    And then the politicians go on TV and lament that people should better listen to authorities, that they were the wise ones. My point is that an unpredictable offshore threat advantages local elected officials. Their careers are advanced by both the false alarms and the disasters.
    - ghaff 11y ago
      That is complete and utter nonsense. No one is basically ever happy with the government response to severe weather events. Leave aside hurricanes. Ask any number of Massachusetts officials who are no longer in their positions because of transit and other failures during blizzards last winter.

78 points by jsm386 11y ago

rch 11y ago

I only have a few unrelated anecdotes to draw from, but my impression is that there is an outsized political influence on where in the US the currently allocated R&D funding is being spent.

jonknee 11y ago

When the party that controls Washington has not believing in climate change as a party platform I don't see a dramatic increase in budget for climate scientists coming anytime soon. Thankfully we can use the European model!

happyscrappy 11y ago
Please don't conflate climate and weather.
- 67726e 11y ago
  Do the Republicans know the difference?
- jonknee 11y ago
  The GOP has consistently wished to cut the budget of the National Weather Service: http://www.slate.com/blogs/weigel/2011/03/11/gop_s_continuing_resolution_cuts_funding_for_national_weather_service_fema.html http://www.slate.com/blogs/weigel/2011/03/11/gop_s_continuin...
  - deleted 11y ago
    [deleted]
- Frondo 11y ago
  They aren't the same thing but they are related to one another, of course.

deciplex 11y ago

> Maybe it will be Joaquin’s false alarm, not another Hurricane Sandy, that gets America to make the G.F.S. great again. Or maybe it will prompt the usual suspects to "privatize" the whole thing i.e. cut funding under the assumption that private enterprise, or churches, or whatever the hell else, will pick up the slack.

deleted 11y ago
[deleted]

heimatau 11y ago

Anyone in this field says it's extremely complicated. NASA (or ESA) probably would solve this, if we didn't underfund them so much. To those advocating to privatizing, I don't see how the private sector could solve it by themselves. Just like McDonalds didn't land on the moon. It's extremely complicated. The task is going to require a lot more scientists and innovators to understand our weather patterns, it's my opinion that the best way to do that, is through the large governments of the world (not just USA's govt).

karmelapple 11y ago
Space exploration, nor the serving of food to those in space, was a goal of McDonald's. There was no sufficient market opportunity. However, is there a market opportunity for improved weather prediction? Who would benefit and make money?
- heimatau 11y ago
  McDonalds is hyperbolic language. There is no market in space, even for SpaceX. Private incentives don't exist in a pure market fashion. Only the ultra-weathly would be the market and they are a small bunch (in the thousands of people) as opposed to the billions in the bottom/middle classes. Also, I wouldn't say the government is the market because that's inflated and not real (always a budget cut away from disappearing). Hence why private interests won't work. P.S. I think SpaceX has a difficult task to appeal to the public. When they can do that, on a regular basis, I'll change my position. Or if some weather company is trying to be disruptive with technology and innovation, then I'll modify my view. (But most of us know innovation in the private sector doesn't always reach the public)
- greglindahl 11y ago
  ECMWF charges for their weather forecasts, which are more accurate than the free American ones. There are lots of small commercial weather forecasting firms who sell to people like securities traders. A single high-temperature day can reduce corn yield by 1%. Here's a startup that had a $1 billion exit to Monsanto: http://www.theverge.com/2013/10/10/4823004/monsanto-bets-the-farm-on-big-data http://www.theverge.com/2013/10/10/4823004/monsanto-bets-the... Their business is a combination of weather and climate. In short, there's a big market for this stuff.

deleted 11y ago

[deleted]

chimtim 11y ago

This was a fascinating read. I found the initial bit interesting that other models were ahead because they simply had more raw computational power, though it later adds that there may be other subtle differences.

dhimes 11y ago
They indicated that the other models were using the change of the atmospheric variables in their initial conditions. (Neumann boundary conditions?) The models may actually be fundamentally different.

aaronbrethorst 11y ago

If you live in the Pacific Northwest, the blog of the professor mentioned in the article, Cliff Mass, is a must-read: http://cliffmass.blogspot.com http://cliffmass.blogspot.com

ZainRiz 11y ago
Or if you want the abridged version check out the tldr feed: https://twitter.com/cliffmasstldr https://twitter.com/cliffmasstldr
foofoo55 11y ago
And if you live in the PNW, mariners (boats, not baseball) here find that the "superforecast" from windfinder.com is consistently far more accurate that the government forecasts, especially for wind on the water. The only big change to the government forecasts seems to have been putting some lipstick on their web pages, while the models don't seem to have changed in years.

lacker 11y ago

It seems quite strange if computing power is really the bottleneck here. What sort of architecture would you have for a weather prediction system where you would spend more on computing power than researchers' salaries?

lsiq 11y ago
There is a very good reason computing power is the bottleneck and it is because of how the weather has to be modeled. Weather models simulate the conditions in the cells of a lattice that is overlaid over geographic areas. To improve accuracy, the cells need to be shrunk to smaller sizes as they have been for a long time. However, every time the size of the cells in the lattices are halved you need 8 times the computing power to perform the calculations of the 3D space. Because of Moore's Law, it takes many years for it to make sense for weather centers to upgrade their infrastructure and see significant improvements in their predictions.
- pinewurst 11y ago
  Cells also dynamically influence neighboring cells in all dimensions, making the highest performance, lowest latency interconnects essential. Interconnect pricing is a significant chunk of these large system costs. I occasionally hear statements wondering why weather modeling can't simply be distributed to desktop or cloud and can't, for some strange reason, get people to understand that connectivity is almost as important as computation itself for this.
  - metaobject 11y ago
    We have a few clusters at work that we use for numerical weather forecasting using the WRF weather model and some other custom applications. And, indeed, the InfiniBand interconnect was a hefty portion of the overall cost of the clusters. At one point there were people considering Gigabit Ethernet instead of IB, but thankfully it was shot down. The lower latency with IB has a greater impact on MPI-based applications than some people realize.
    - 11y ago

sandworm101 11y ago

Do those in charge actually want accurate predictions? The hurricane is going to go where it is going to go, the model has no impact on the realworld path. If we know the path with certainty then those in the path can take appropriate measures. But what is we enjoy those measures and want to do them regardless of where the storm will actually hit? What a storm might be approach the executive branch of the local government goes into emergency mode. Emergency funds are spent on preparations. Some government workers are sent home. Schools might be closed. Evacuation orders may be issued. Town mayors and, importantly, state governors all go on TV to talk about these preparations. The local executives get to do their lord-and-master thing. If the storm hits, they looks like heroes. If it doesn't, they still look like dutiful public servants concerned for the safety of their public. Some people enjoy a good disaster rehearsal, especially if they get to be the one in charge. Imho the private sector, specifically the insurance, travel and tourism industries are the ones who want accurate models. That isn't to say that prediction should be privatized. The government is probably in the better place to handle forecasting. But the demand on politicians for better models/funding arises from industry, not any noble desire to avoid widespread emergency declarations.

ghaff 11y ago
And if you do the disaster rehearsal thing too many times without anything happening, i.e. cry wolf, the next time no one pays any attention, Katrina or Sandy hits, and a lot of people die that didn't need to.
- sandworm101 11y ago
  And then the politicians go on TV and lament that people should better listen to authorities, that they were the wise ones. My point is that an unpredictable offshore threat advantages local elected officials. Their careers are advanced by both the false alarms and the disasters.
  - ghaff 11y ago
    That is complete and utter nonsense. No one is basically ever happy with the government response to severe weather events. Leave aside hurricanes. Ask any number of Massachusetts officials who are no longer in their positions because of transit and other failures during blizzards last winter.