3 ms·

Yes - it’s much like now weather forecasting works. Data points may come from 2 places 100 mi apart, but what if someone lives between them? Temperature or ot

by pnpnp 3y ago

Yes - it’s much like now weather forecasting works. Data points may come from 2 places 100 mi apart, but what if someone lives between them? Temperature or other weather data is often an extrapolation based on official weather data collection points, and AQI data is no different. How that data gets extrapolated varies wildly by implementation. edit: Also a fun fact, I live 2000ft above and 50mi from the nearest forecasting center in a valley. “Weather apps” are about as accurate as throwing darts at a board up here, because the models used to extrapolate known data completely fall apart with all the variables. Our best source of data is the “short term weather discussion” by NOAA, which just gives some generalizations based on local weather knowledge.

aziaziazi 3y ago
Two differences between weather and AQI forecast: - AQI strongly varies with local human activities, e.g. a coal power plant in between the 2 places 100 miles appart. - temporal and hardly predictable human activities, e.g. a chem industry without good attenuation system that boost its production to fill a special command. Or an attenuation system that breaks.
crazygringo 3y ago
Not just forecasting, but reporting values right now. Spatial interpolation is much more non-obvious than you might initially assume, when the data points are coming from arbitrary locations and not a regular grid. For anyone curious, here [1] is a good place to start, or look up "Kriging interpolation". The basic implementation of any kind of continuous temperature estimation is to have a generic equation that relates how temperature naturally varies by altitude, use that normalize the temperature reported by weather stations according to altitude to create a "flat" temperature map, interpolate to create a smooth continuous temperature map using something like Kriging, and then de-normalize the temperature at a desired point according to the altitude of that location. There are always going to be hyper-local variations that won't account for (are you on the sunny side or shaded side of a mountain? is there an air current? how is vegetation/buildings/asphalt affecting it?), but it's the basic starting point. [1] https://geobgu.xyz/r-2020/spatial-interpolation-of-point-data.html https://geobgu.xyz/r-2020/spatial-interpolation-of-point-dat...
- counters 3y ago
  Eh, kriging is overkill relative to the skill you end up with since fine-grain spatial variability in weather can arise from a lot more than the static factors you can pull into such a scheme. This is why objective analysis techniques are typically preferred. More importantly, high-resolution priors are universally available from weather models, so there is no need to just to ever naively interpolate between weather observation stations very far apart. One of the reasons that consumer weather apps perform poorly is because in many cases, the companies producing them arbitrarily choose some sort of spatial or temporal resolution requirements, and then throw the problem to an engineer or data scientist who probably doesn't know much more than these simple interpolation techniques (and I absolutely lump kriging into the 'simple' category these days). A modicum of domain knowledge applied here produces a much better product with substantially less effort.
  - crazygringo 3y ago
    I'm curious if you could expand -- I researched this a while ago but what I'm describing is as deep as I got from various online tutorials. What kind of objective analysis techniques? And where does someone get high-resolution priors from? Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? (Which would be over 3 million points at that scale, compared to the ~10,000 US weather stations?)
    - counters 3y ago
      Probably the two most common and simplest are Barnes and Cressman interpolation (see [1] for a modern implementation of Barnes), which use inverse distance weighting to combine observations within a neighborhood. An improvement to these techniques frequently used to grid statistical weather forecasts is the BCDG method (see [2]). The core idea of any objective analysis/interpolation scheme is that you model spatial relationships with some sort of function. Kriging exemplifies this - you fit a spatial covariance model and use that to predict values in the far field, using combinations of nearby values. I don't really have a good link or textbook reference at arms reach, but you can quickly re-frame this entire interpolation problem as a data assimilation one where you're attempting to approximate a solution to some 2D or 3D field using sparsely sampled observations; in the weather world, the Real-time Mesoscale Analysis used in the USA is a good example of a system which hybridizes a weather model and observations to create a high-resolution analysis [3]. A problem arises when the field you're analyzing can have shocks or variability much smaller than the scale that you're able to measure from your network. A front is a great example - across a front, temperature and wind direction will change over a very small distance, but most interpolation schemes will smear this out. The problem is that you really do care about that front and its location - that's where the interesting weather happens! > And where does someone get high-resolution priors from? A high-resolution weather forecast model, which uses physics to simulate the atmosphere and produce reference states. 3-km forecasts are readily and easily available in the US. > Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? No, this probably wouldn't be useful. Especially because we get a ~1-km mapping of lower troposphere temperatures from geostationary weather satellites every 10-15 minutes across the entire globe. May not be exactly "surface temperature", but it tells you a whole lot about high-frequency variability in temperature fields. [1]: https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-116.pdf https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-1... [2]: https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf2007080_1.xml https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf... [3]: https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-10-05037_1.xml https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-1...

by pnpnp 3y ago

aziaziazi 3y ago

Two differences between weather and AQI forecast: - AQI strongly varies with local human activities, e.g. a coal power plant in between the 2 places 100 miles appart. - temporal and hardly predictable human activities, e.g. a chem industry without good attenuation system that boost its production to fill a special command. Or an attenuation system that breaks.

crazygringo 3y ago

Not just forecasting, but reporting values right now. Spatial interpolation is much more non-obvious than you might initially assume, when the data points are coming from arbitrary locations and not a regular grid. For anyone curious, here [1] is a good place to start, or look up "Kriging interpolation". The basic implementation of any kind of continuous temperature estimation is to have a generic equation that relates how temperature naturally varies by altitude, use that normalize the temperature reported by weather stations according to altitude to create a "flat" temperature map, interpolate to create a smooth continuous temperature map using something like Kriging, and then de-normalize the temperature at a desired point according to the altitude of that location. There are always going to be hyper-local variations that won't account for (are you on the sunny side or shaded side of a mountain? is there an air current? how is vegetation/buildings/asphalt affecting it?), but it's the basic starting point. [1] https://geobgu.xyz/r-2020/spatial-interpolation-of-point-data.html https://geobgu.xyz/r-2020/spatial-interpolation-of-point-dat...

counters 3y ago
Eh, kriging is overkill relative to the skill you end up with since fine-grain spatial variability in weather can arise from a lot more than the static factors you can pull into such a scheme. This is why objective analysis techniques are typically preferred. More importantly, high-resolution priors are universally available from weather models, so there is no need to just to ever naively interpolate between weather observation stations very far apart. One of the reasons that consumer weather apps perform poorly is because in many cases, the companies producing them arbitrarily choose some sort of spatial or temporal resolution requirements, and then throw the problem to an engineer or data scientist who probably doesn't know much more than these simple interpolation techniques (and I absolutely lump kriging into the 'simple' category these days). A modicum of domain knowledge applied here produces a much better product with substantially less effort.
- crazygringo 3y ago
  I'm curious if you could expand -- I researched this a while ago but what I'm describing is as deep as I got from various online tutorials. What kind of objective analysis techniques? And where does someone get high-resolution priors from? Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? (Which would be over 3 million points at that scale, compared to the ~10,000 US weather stations?)
  - counters 3y ago
    Probably the two most common and simplest are Barnes and Cressman interpolation (see [1] for a modern implementation of Barnes), which use inverse distance weighting to combine observations within a neighborhood. An improvement to these techniques frequently used to grid statistical weather forecasts is the BCDG method (see [2]). The core idea of any objective analysis/interpolation scheme is that you model spatial relationships with some sort of function. Kriging exemplifies this - you fit a spatial covariance model and use that to predict values in the far field, using combinations of nearby values. I don't really have a good link or textbook reference at arms reach, but you can quickly re-frame this entire interpolation problem as a data assimilation one where you're attempting to approximate a solution to some 2D or 3D field using sparsely sampled observations; in the weather world, the Real-time Mesoscale Analysis used in the USA is a good example of a system which hybridizes a weather model and observations to create a high-resolution analysis [3]. A problem arises when the field you're analyzing can have shocks or variability much smaller than the scale that you're able to measure from your network. A front is a great example - across a front, temperature and wind direction will change over a very small distance, but most interpolation schemes will smear this out. The problem is that you really do care about that front and its location - that's where the interesting weather happens! > And where does someone get high-resolution priors from? A high-resolution weather forecast model, which uses physics to simulate the atmosphere and produce reference states. 3-km forecasts are readily and easily available in the US. > Are you saying that companies have gone out and done a one-time mapping of temperature etc. at a e.g. 1-mile grid across the US? No, this probably wouldn't be useful. Especially because we get a ~1-km mapping of lower troposphere temperatures from geostationary weather satellites every 10-15 minutes across the entire globe. May not be exactly "surface temperature", but it tells you a whole lot about high-frequency variability in temperature fields. [1]: https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-116.pdf https://gmd.copernicus.org/preprints/gmd-2022-116/gmd-2022-1... [2]: https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf2007080_1.xml https://journals.ametsoc.org/view/journals/wefo/24/2/2008waf... [3]: https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-10-05037_1.xml https://journals.ametsoc.org/view/journals/wefo/26/5/waf-d-1...