4 ms·

> NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they

> NOAA/NWS, for example, is extremely underfunded so if they had to privilege to buy it they probably couldn't come to an agreement to buy it. As a result, they can't use that data to improve the accuracy of alerts/warnings/forcasts, the same exact tools that the big weather companies make all their money from. It's a shit cycle and totally unfair IMO. Huh? This is kind of an odd take for a few reasons. For starters, NOAA isn't "extremely underfunded"; with the possible exception of the current budgeting cycle, NOAA generally does pretty well and has strong bipartisan support. It could always use more money, but I wouldn't call it "underfunded. The reason NOAA doesn't buy more data is because most of the available data has limited value. Personal weather stations have substantial quality issues and add almost no value in areas where we already have high-quality surface observations. We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective. For what it's worth, few if any companies use proprietary observations to improve their forecasts. It's an open secret that the vast majority of companies out there are just applying proprietary statistical modeling / bias correction on top of publicly available data. Only a handful of companies actually have novel observations, and there's limited evidence it makes a significant difference in the forecast. At best, it can result in the way that those statistical corrections are applied to existing forecasts and ensembles - you can count on one hand the number of companies that actually run a vertically-integrated stack including data assimilation of proprietary observations and end-to-end numerical modeling. That isn't to say there isn't unique value in the observations. It's just that the industry flagrantly misleads about how they use them.

pinkmuffinere 2y ago
Referring to the parent comment, the data which NOAA isn’t able to buy is the government’s data, which is freely provided to non-government organizations. The parent comment doesn’t discuss personal data very much. I think this misunderstanding might be the root cause of your disagreement.
- counters 2y ago
  Parent comment is mis-informed. NOAA doesn't have to buy any data from other government agencies or organizations. It's all open and publicly available. There are challenges around the reliability and quality of data that NOAA doesn't take efforts itself to curate and maintain which limit their utility, but that's a separate issue. More importantly, NOAA explicitly funds the National Mesonet Program [1] to actively identify, acquire, consume, and ingest data from a wide variety of state and federal agencies across the country. The NMP itself partners with a major private sector company, Synoptic Data PBC [2] to perform the engineering necessary to acquire all this data. Synoptic actively maintains the infrastructure which consumes and publishes this data to MADIS for use by NOAA and any other stakeholder. [1]: https://nationalmesonet.us/ https://nationalmesonet.us/ [2]: https://synopticdata.com/ https://synopticdata.com/
zorm 2y ago
Very few companies run the vertically-integrated stack because it is prohibitively expensive to do so with current NWP versus what you can sell it for with only marginal forecast improvements. I know several companies have tried this with integrating their own observation sources and ended up with worse performing forecasts. Oops. I'm very interested to see how the ML modeling revolution changes this. The ability to perform global forecasts on a single GPU should make it cost competitive for more companies. I know several companies are already deriving their own weights for the forecasting component so that they can sell them. Google appears to be working on the next piece of the puzzle too with using ML for the data assimilation step, or skipping that altogether and using observations to go directly to forecasts.
- counters 2y ago
  There are a few groups working on leveraging observations more directly in the ML forecast models and skipping over the assimilation/analysis step. However, unlike the original ML forecasting problem (which, let's be honest - was grossly over-simplified by the existence of ERA-5, which has been treated as "ground truth" for the atmosphere and used to teach models how to simply go from state at t=1 to state at t=1+\delta t), there's reason to believe that incorporating the observations will be substantially more difficult, given the complexity and bounty of the observations themselves and the challenge of framing a tractable, useful ML problem on top of them.
amluto 2y ago
> We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective. I regularly notice that the NWS forecasts, even in the very short term, get the surface conditions rather wrong. (This is by comparison to a an inadvertent but, I think, quite accurate surface temperature and humidity measurement that I have.) I fully believe that the measurements aloft do a great job of predicting the conditions aloft, but I wonder whether the results would be further improved by even a fairly simple model to map the forecast results back to detailed surface conditions. After all, many of consumers of weather forecasts, e.g. people caring about personal comfort, climate control energy predictions and pre-heating/pre-cooling of buildings, etc. care about surface conditions more than they care about conditions aloft.
- counters 2y ago
  The measurements aloft constrain the entire system - things like vertical profiles of moisture and temperature, as well as the kinematic structure of the atmosphere (e.g. wind profiles) grossly constrain the evolution of the system. Put another way - the _information density_ of these observations is very high and they tend to constrain non-local features of the flow / structure of the atmosphere. Observations for the surface don't have this effect for two reasons: (1) they can be dominated by local influences (like local topography) that poorly constrain the background atmospheric state, and (2) the majority of numerical weather models do not directly model the planetary boundary layer (the layer of the atmosphere closest to the ground), and instead parameterize processes that occur here. What this means, practically, is that the information content of surface observations is low (1), and even when it isn't, there isn't a mechanism to effectively propagate this information outside of a single grid cell or even column in the actual forecast model (2). That's why observations are typically used to bias-correct forecast models - it's a form of localization or downscaling.

pinkmuffinere 2y ago

Referring to the parent comment, the data which NOAA isn’t able to buy is the government’s data, which is freely provided to non-government organizations. The parent comment doesn’t discuss personal data very much. I think this misunderstanding might be the root cause of your disagreement.

counters 2y ago
Parent comment is mis-informed. NOAA doesn't have to buy any data from other government agencies or organizations. It's all open and publicly available. There are challenges around the reliability and quality of data that NOAA doesn't take efforts itself to curate and maintain which limit their utility, but that's a separate issue. More importantly, NOAA explicitly funds the National Mesonet Program [1] to actively identify, acquire, consume, and ingest data from a wide variety of state and federal agencies across the country. The NMP itself partners with a major private sector company, Synoptic Data PBC [2] to perform the engineering necessary to acquire all this data. Synoptic actively maintains the infrastructure which consumes and publishes this data to MADIS for use by NOAA and any other stakeholder. [1]: https://nationalmesonet.us/ https://nationalmesonet.us/ [2]: https://synopticdata.com/ https://synopticdata.com/

zorm 2y ago

Very few companies run the vertically-integrated stack because it is prohibitively expensive to do so with current NWP versus what you can sell it for with only marginal forecast improvements. I know several companies have tried this with integrating their own observation sources and ended up with worse performing forecasts. Oops. I'm very interested to see how the ML modeling revolution changes this. The ability to perform global forecasts on a single GPU should make it cost competitive for more companies. I know several companies are already deriving their own weights for the forecasting component so that they can sell them. Google appears to be working on the next piece of the puzzle too with using ML for the data assimilation step, or skipping that altogether and using observations to go directly to forecasts.

counters 2y ago
There are a few groups working on leveraging observations more directly in the ML forecast models and skipping over the assimilation/analysis step. However, unlike the original ML forecasting problem (which, let's be honest - was grossly over-simplified by the existence of ERA-5, which has been treated as "ground truth" for the atmosphere and used to teach models how to simply go from state at t=1 to state at t=1+\delta t), there's reason to believe that incorporating the observations will be substantially more difficult, given the complexity and bounty of the observations themselves and the challenge of framing a tractable, useful ML problem on top of them.

amluto 2y ago

> We thin out and throw away a ton of surface observations already during the data assimilation process to initialize our forecast models anyways - data from aloft is far more valuable and impactful from a forecast impact perspective. I regularly notice that the NWS forecasts, even in the very short term, get the surface conditions rather wrong. (This is by comparison to a an inadvertent but, I think, quite accurate surface temperature and humidity measurement that I have.) I fully believe that the measurements aloft do a great job of predicting the conditions aloft, but I wonder whether the results would be further improved by even a fairly simple model to map the forecast results back to detailed surface conditions. After all, many of consumers of weather forecasts, e.g. people caring about personal comfort, climate control energy predictions and pre-heating/pre-cooling of buildings, etc. care about surface conditions more than they care about conditions aloft.

counters 2y ago
The measurements aloft constrain the entire system - things like vertical profiles of moisture and temperature, as well as the kinematic structure of the atmosphere (e.g. wind profiles) grossly constrain the evolution of the system. Put another way - the _information density_ of these observations is very high and they tend to constrain non-local features of the flow / structure of the atmosphere. Observations for the surface don't have this effect for two reasons: (1) they can be dominated by local influences (like local topography) that poorly constrain the background atmospheric state, and (2) the majority of numerical weather models do not directly model the planetary boundary layer (the layer of the atmosphere closest to the ground), and instead parameterize processes that occur here. What this means, practically, is that the information content of surface observations is low (1), and even when it isn't, there isn't a mechanism to effectively propagate this information outside of a single grid cell or even column in the actual forecast model (2). That's why observations are typically used to bias-correct forecast models - it's a form of localization or downscaling.