4 ms·

It's going to be a tangential comment but I work in science research that's adjacent to weather forecasting and I find the political/technical jockeying that is

It's going to be a tangential comment but I work in science research that's adjacent to weather forecasting and I find the political/technical jockeying that is happening with forecasting to be fascinating. It's a nexus of capitalism, federal government spending, politics, and technology that has very real implications for individual Americans. In summary: horrible oversight by the federal govt (read, congress) of our technical/scientific forecasting resources means that our forecasting ability is extremely fragmented and poorly organized. This has lead to a lot of companies being essentially resellers of public data. These companies claim to create a lot of value added products ('cleaner APIs', 'minutecasts', etc etc) that are either scientifically dubious or technically simple and then these companies walk away with huge profits based on being a portal to government data. It's so American it is almost laughable, all while the European ECMWF eats our lunch in terms of accuracy even for the CONUS. I've discussed this on technical internet forums often enough that I can practically already write the replies to my own comment. "What's the problem with that?" etc et al. But the reality of it is that it's emblematic of how politically broken the US is, in particular with regards to the agencies in charge of scientific products and funding. Not to mention the concrete problems with the forecast products themselves. Anyway. Good luck pirate weather and godspeed. Information was meant to be free and open, especially the forecast. It's such a laughably simple problem that could/should be so easily solved but, alas, there is money to be made!

jaybo_nomad 4y ago
A thousand times this!
billiam 4y ago
Have any nerdy Congresspeople wanted to solve this ridiculous inefficiency in recent years. Seems like a natural "the Europeans kick our ass at forecasting" message would go over well and potentially negate the lobbying from Weather Channel and others with a vested interest in perpetuating this stupid situation.
- counters 4y ago
  Yes, Congress passed the Weather Research and Forecasting Innovation Act back in 2017 which not only gave massive funding injection to model development activities, data procurement, and more, but also formalized the new next-generation community modeling initiatives which are focused on tackling the dominance of the EC models in the 2-5 day forecast space. The diagnosis of the problems of the American forecast modeling community here is based on flawed premises. There are three major factors which led to the ECMWF leap-frogging the US in day-ahead forecasting capability. The first is the consolidated investment in supercomputing resources; the WRFIA tackles this by earmarking a much larger allocation of funding for NOAA's next gen supercomputer, but this still pales in comparison to ECMWF investments. The second factor is the fragmentation of the research and operational weather modeling communities due to the divergent investment from NOAA and USAF in the 90's and 2000's; USAF in conjunction with NCAR sponsored the development of the WRF model which was widely adopted by the research community. NOAA continued investing the GFS lineage of models. The bifurcation of these communities slowed down the ability to matriculate advances in model developments to operations, and this was exacerbated by an old, closed-off approach by NOAA which made it extraordinarily difficult to run and develop the GFS on anything other than NOAA hardware. Finally, the ECMWF went all-in on 4DVAR data assimilation in the late 90's, whereas the American community pursued a diversity of other approaches ranging from 4DVAR to ensemble Kalman filters. 4DVAR necessitates advances to core weather model software (e.g. you need to write a model's adjoint or its tangent linear in order to actually use 4DVAR) adn the US' failure to adopt it led, imo to a "double edged sword" effect of (a) failing to provide impetus to greatly improve the US modeling software suite and supporting tools, and (b) being a worse assimilation technique unless advanced EnsKF techniques are employed using very large ensembles of models (expensive). The other problem as others have pointed out is that there is no accountability in the US private sector weather market. Virtually every player is re-transmitting raw model guidance or statistical post-processed forecasts using DiCast, _maybe_ with a some manual tweaking of forecast fields by humans. But this is not transparent, and many companies - if we're being charitable, here - are not honest about what they're actually doing to produce their forecasts. Put another way - there's a lot of BS claims out there, and it seems that investors have been more than happy to fund it over the past few years.
jmbwell 4y ago
100%. There is enormous data available at https://www.weather.gov/ https://www.weather.gov/ at no charge, including hourly and weekly forecasts, spot forecasts, radar (multiple layers, with/without animation) and satellite (multiple layers, with/without animation), plus storm watches, hurricane info, historical data, climate data… I guess it's nice that apps can do things like advise me that it might start raining in a few minutes, but often by the time I see those alerts, the water on my head has alerted me anyway. All other weather apps, it seems to me, are for little more than tracking my location and serving me relevant ads.
- Sightline 4y ago
  The most accurate and concise weather forecasts come from weather.gov, blows my mind that people don't know this.
baq 4y ago
As someone living in continental Europe there’s one thing I haven’t found anything remotely close to NHC quality forecasts. Curious what you think of that institution?
mesoman 4y ago
As one who wrote (over 10 years ago, still improving it) and sells a weather app targeted at storm chasers, weather enthusiasts and spotters... I find the NWS data to be fragmented, often based on ancient computing standards (look at how NEXRAD data is formatted - it's binary for a 16-bit computer), and very hard to find documentation on. Companies can add value by providing documented, consistent API's for data that, yes, is free from the government. NWS does not face the market incentives of a company, and it shows. And yes, a bunch of companies take that data and hype it, but that's not particularly new - it precedes apps. I've long seen claims of forecast accuracy from private companies that are, well, absurd, given the limits chaos (and other issues) place on forecasting very far into the future. The Big Data Project is a substantial improvement in terms of access, but the data itself is still in the legacy formats. Also, some data is not well suited for mobile access - it's in giant binary blobs (NEXRAD Level II) or requires multiple HTTP operations to acquire. But... at least it's available and free (except for lighting). I'm not talking about model data - I let others worry about that, and for personal use, I use sites like the excellent one from College of DuPage ( https://weather.cod.edu/forecast/ https://weather.cod.edu/forecast/ ). I've watched friends in the research and operational community describe their frustration at the decision process that went into GFS modernization, and how it was frustrating to see ECMWF beat it out in forecast skill (I'm not up to date on where that stands now).
- alexander0042 4y ago
  As the dev who set up the Pirate Weather source, I couldn't agree more. 90% of the challenge here was getting the grib files into a format I could quickly and easily access, since the provided format is about as difficult as it gets. I don't pretend to know more about the actual art of forecasting than NOAA, but wanted a better way to get the data they produce out there

jaybo_nomad 4y ago

A thousand times this!

billiam 4y ago

Have any nerdy Congresspeople wanted to solve this ridiculous inefficiency in recent years. Seems like a natural "the Europeans kick our ass at forecasting" message would go over well and potentially negate the lobbying from Weather Channel and others with a vested interest in perpetuating this stupid situation.

counters 4y ago
Yes, Congress passed the Weather Research and Forecasting Innovation Act back in 2017 which not only gave massive funding injection to model development activities, data procurement, and more, but also formalized the new next-generation community modeling initiatives which are focused on tackling the dominance of the EC models in the 2-5 day forecast space. The diagnosis of the problems of the American forecast modeling community here is based on flawed premises. There are three major factors which led to the ECMWF leap-frogging the US in day-ahead forecasting capability. The first is the consolidated investment in supercomputing resources; the WRFIA tackles this by earmarking a much larger allocation of funding for NOAA's next gen supercomputer, but this still pales in comparison to ECMWF investments. The second factor is the fragmentation of the research and operational weather modeling communities due to the divergent investment from NOAA and USAF in the 90's and 2000's; USAF in conjunction with NCAR sponsored the development of the WRF model which was widely adopted by the research community. NOAA continued investing the GFS lineage of models. The bifurcation of these communities slowed down the ability to matriculate advances in model developments to operations, and this was exacerbated by an old, closed-off approach by NOAA which made it extraordinarily difficult to run and develop the GFS on anything other than NOAA hardware. Finally, the ECMWF went all-in on 4DVAR data assimilation in the late 90's, whereas the American community pursued a diversity of other approaches ranging from 4DVAR to ensemble Kalman filters. 4DVAR necessitates advances to core weather model software (e.g. you need to write a model's adjoint or its tangent linear in order to actually use 4DVAR) adn the US' failure to adopt it led, imo to a "double edged sword" effect of (a) failing to provide impetus to greatly improve the US modeling software suite and supporting tools, and (b) being a worse assimilation technique unless advanced EnsKF techniques are employed using very large ensembles of models (expensive). The other problem as others have pointed out is that there is no accountability in the US private sector weather market. Virtually every player is re-transmitting raw model guidance or statistical post-processed forecasts using DiCast, _maybe_ with a some manual tweaking of forecast fields by humans. But this is not transparent, and many companies - if we're being charitable, here - are not honest about what they're actually doing to produce their forecasts. Put another way - there's a lot of BS claims out there, and it seems that investors have been more than happy to fund it over the past few years.

jmbwell 4y ago

100%. There is enormous data available at https://www.weather.gov/ https://www.weather.gov/ at no charge, including hourly and weekly forecasts, spot forecasts, radar (multiple layers, with/without animation) and satellite (multiple layers, with/without animation), plus storm watches, hurricane info, historical data, climate data… I guess it's nice that apps can do things like advise me that it might start raining in a few minutes, but often by the time I see those alerts, the water on my head has alerted me anyway. All other weather apps, it seems to me, are for little more than tracking my location and serving me relevant ads.

Sightline 4y ago
The most accurate and concise weather forecasts come from weather.gov, blows my mind that people don't know this.

baq 4y ago

As someone living in continental Europe there’s one thing I haven’t found anything remotely close to NHC quality forecasts. Curious what you think of that institution?

mesoman 4y ago

As one who wrote (over 10 years ago, still improving it) and sells a weather app targeted at storm chasers, weather enthusiasts and spotters... I find the NWS data to be fragmented, often based on ancient computing standards (look at how NEXRAD data is formatted - it's binary for a 16-bit computer), and very hard to find documentation on. Companies can add value by providing documented, consistent API's for data that, yes, is free from the government. NWS does not face the market incentives of a company, and it shows. And yes, a bunch of companies take that data and hype it, but that's not particularly new - it precedes apps. I've long seen claims of forecast accuracy from private companies that are, well, absurd, given the limits chaos (and other issues) place on forecasting very far into the future. The Big Data Project is a substantial improvement in terms of access, but the data itself is still in the legacy formats. Also, some data is not well suited for mobile access - it's in giant binary blobs (NEXRAD Level II) or requires multiple HTTP operations to acquire. But... at least it's available and free (except for lighting). I'm not talking about model data - I let others worry about that, and for personal use, I use sites like the excellent one from College of DuPage ( https://weather.cod.edu/forecast/ https://weather.cod.edu/forecast/ ). I've watched friends in the research and operational community describe their frustration at the decision process that went into GFS modernization, and how it was frustrating to see ECMWF beat it out in forecast skill (I'm not up to date on where that stands now).

alexander0042 4y ago
As the dev who set up the Pirate Weather source, I couldn't agree more. 90% of the challenge here was getting the grib files into a format I could quickly and easily access, since the provided format is about as difficult as it gets. I don't pretend to know more about the actual art of forecasting than NOAA, but wanted a better way to get the data they produce out there