47 ms·

Eh, kriging is overkill relative to the skill you end up with since fine-grain spatial variability in weather can arise from a lot more than the static factors

Eh, kriging is overkill relative to the skill you end up with since fine-grain spatial variability in weather can arise from a lot more than the static factors you can pull into such a scheme. This is why objective analysis techniques are typically preferred. More importantly, high-resolution priors are universally available from weather models, so there is no need to just to ever naively interpolate between weather observation stations very far apart. One of the reasons that consumer weather apps perform poorly is because in many cases, the companies producing them arbitrarily choose some sort of spatial or temporal resolution requirements, and then throw the problem to an engineer or data scientist who probably doesn't know much more than these simple interpolation techniques (and I absolutely lump kriging into the 'simple' category these days). A modicum of domain knowledge applied here produces a much better product with substantially less effort.

crazygringo 3y ago
I'm curious if you could expand -- I researched this a while ago but what I'm describing is as deep as I got from various online tutorials. What kind of objective analysis techniques? And where does someone get high-resolution priors from? Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? (Which would be over 3 million points at that scale, compared to the ~10,000 US weather stations?)
- counters 3y ago
  Probably the two most common and simplest are Barnes and Cressman interpolation (see [1] for a modern implementation of Barnes), which use inverse distance weighting to combine observations within a neighborhood. An improvement to these techniques frequently used to grid statistical weather forecasts is the BCDG method (see [2]). The core idea of any objective analysis/interpolation scheme is that you model spatial relationships with some sort of function. Kriging exemplifies this - you fit a spatial covariance model and use that to predict values in the far field, using combinations of nearby values. I don't really have a good link or textbook reference at arms reach, but you can quickly re-frame this entire interpolation problem as a data assimilation one where you're attempting to approximate a solution to some 2D or 3D field using sparsely sampled observations; in the weather world, the Real-time Mesoscale Analysis used in the USA is a good example of a system which hybridizes a weather model and observations to create a high-resolution analysis [3]. A problem arises when the field you're analyzing can have shocks or variability much smaller than the scale that you're able to measure from your network. A front is a great example - across a front, temperature and wind direction will change over a very small distance, but most interpolation schemes will smear this out. The problem is that you really do care about that front and its location - that's where the interesting weather happens! > And where does someone get high-resolution priors from? A high-resolution weather forecast model, which uses physics to simulate the atmosphere and produce reference states. 3-km forecasts are readily and easily available in the US. > Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? No, this probably wouldn't be useful. Especially because we get a ~1-km mapping of lower troposphere temperatures from geostationary weather satellites every 10-15 minutes across the entire globe. May not be exactly "surface temperature", but it tells you a whole lot about high-frequency variability in temperature fields. [1]: https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-116.pdf https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-1... [2]: https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf2007080_1.xml https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf... [3]: https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-10-05037_1.xml https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-1...
  - crazygringo 3y ago
    Thank you so much, that's all so fascinating! And makes perfect sense that priors come from high resolution physics solutions. Much appreciated for all that info.

crazygringo 3y ago

I'm curious if you could expand -- I researched this a while ago but what I'm describing is as deep as I got from various online tutorials. What kind of objective analysis techniques? And where does someone get high-resolution priors from? Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? (Which would be over 3 million points at that scale, compared to the ~10,000 US weather stations?)

counters 3y ago
Probably the two most common and simplest are Barnes and Cressman interpolation (see [1] for a modern implementation of Barnes), which use inverse distance weighting to combine observations within a neighborhood. An improvement to these techniques frequently used to grid statistical weather forecasts is the BCDG method (see [2]). The core idea of any objective analysis/interpolation scheme is that you model spatial relationships with some sort of function. Kriging exemplifies this - you fit a spatial covariance model and use that to predict values in the far field, using combinations of nearby values. I don't really have a good link or textbook reference at arms reach, but you can quickly re-frame this entire interpolation problem as a data assimilation one where you're attempting to approximate a solution to some 2D or 3D field using sparsely sampled observations; in the weather world, the Real-time Mesoscale Analysis used in the USA is a good example of a system which hybridizes a weather model and observations to create a high-resolution analysis [3]. A problem arises when the field you're analyzing can have shocks or variability much smaller than the scale that you're able to measure from your network. A front is a great example - across a front, temperature and wind direction will change over a very small distance, but most interpolation schemes will smear this out. The problem is that you really do care about that front and its location - that's where the interesting weather happens! > And where does someone get high-resolution priors from? A high-resolution weather forecast model, which uses physics to simulate the atmosphere and produce reference states. 3-km forecasts are readily and easily available in the US. > Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? No, this probably wouldn't be useful. Especially because we get a ~1-km mapping of lower troposphere temperatures from geostationary weather satellites every 10-15 minutes across the entire globe. May not be exactly "surface temperature", but it tells you a whole lot about high-frequency variability in temperature fields. [1]: https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-116.pdf https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-1... [2]: https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf2007080_1.xml https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf... [3]: https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-10-05037_1.xml https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-1...
- crazygringo 3y ago
  Thank you so much, that's all so fascinating! And makes perfect sense that priors come from high resolution physics solutions. Much appreciated for all that info.