4 ms·

Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Woul

Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....

darksaints 12y ago
Sometimes Machine Learning is the wrong approach. Direct mathematical modeling, if the underlying causal phenomenon are well known, bypasses the whole learning aspect of an explanatory system. Machine learning might help make sense of the residual noise that can't be accounted for with direct mathematical modeling. But at the level of sophistication that I've seen (predicting wind patterns using Finite Element Analysis methods mapped to topographical patterns fine-grained enough to model small buildings), I can't imagine any machine learning methods that could make any sense of the residual noise.
- davidwihl 12y ago
  I agree that Machine Learning is not always the right approach but wouldn't it be a relatively inexpensive experiment to see if it could be used for more accurate predictions? I'm surprised that topographical patterns are mapped that precisely when ASOS and AWOs stations are so far apart. Also the weather data from commercial airplanes is so far above the surface that modeling small buildings would be meaningless. I heard a talk from ex-Googler climate.com (now Monsanto) guys. Their forecasts were accurate enough to build a very profitable insurance business. I bet they used Bayesian rather than FEM techniques.
  - bainsfather 12y ago
    Is it that 'physics simulation' forecasting is best for short timescales (e.g. < 1 week in the uk), whilst ML can be better for estimating next year's weather statistics in some particular place? (Because your butterfly effect means that simulating the actual weather out to 1 year 'accurately' is not possible). Presumably any short-term ML forecasting would use the physics simulation as it's main input, and then try to improve slightly on it (e.g. maybe you observe that in a particular small area, the rainfall is on average 10% more than the physics simulation, so your ML method would add 10%, giving you a slight forecast improvement)?
desdiv 12y ago
ML techniques certainly are being used to predict the weather, maybe just not at NOAA or the Weather Channel. The weather derivative industry[0] is an $8 billion dollar industry, making it larger than the entire rocket launch industry (public and private combined). Just as there are Elon Musks in the rocket launch industry, there are maverick hedge funds out there making a killing by predicting the weather using the latest Big Data tools. [0] http://en.wikipedia.org/wiki/Weather_derivative http://en.wikipedia.org/wiki/Weather_derivative
GFK_of_xmaspast 12y ago
Probably because if it worked they'd already be doing it.

darksaints 12y ago

Sometimes Machine Learning is the wrong approach. Direct mathematical modeling, if the underlying causal phenomenon are well known, bypasses the whole learning aspect of an explanatory system. Machine learning might help make sense of the residual noise that can't be accounted for with direct mathematical modeling. But at the level of sophistication that I've seen (predicting wind patterns using Finite Element Analysis methods mapped to topographical patterns fine-grained enough to model small buildings), I can't imagine any machine learning methods that could make any sense of the residual noise.

davidwihl 12y ago
I agree that Machine Learning is not always the right approach but wouldn't it be a relatively inexpensive experiment to see if it could be used for more accurate predictions? I'm surprised that topographical patterns are mapped that precisely when ASOS and AWOs stations are so far apart. Also the weather data from commercial airplanes is so far above the surface that modeling small buildings would be meaningless. I heard a talk from ex-Googler climate.com (now Monsanto) guys. Their forecasts were accurate enough to build a very profitable insurance business. I bet they used Bayesian rather than FEM techniques.
- bainsfather 12y ago
  Is it that 'physics simulation' forecasting is best for short timescales (e.g. < 1 week in the uk), whilst ML can be better for estimating next year's weather statistics in some particular place? (Because your butterfly effect means that simulating the actual weather out to 1 year 'accurately' is not possible). Presumably any short-term ML forecasting would use the physics simulation as it's main input, and then try to improve slightly on it (e.g. maybe you observe that in a particular small area, the rainfall is on average 10% more than the physics simulation, so your ML method would add 10%, giving you a slight forecast improvement)?

desdiv 12y ago

ML techniques certainly are being used to predict the weather, maybe just not at NOAA or the Weather Channel. The weather derivative industry[0] is an $8 billion dollar industry, making it larger than the entire rocket launch industry (public and private combined). Just as there are Elon Musks in the rocket launch industry, there are maverick hedge funds out there making a killing by predicting the weather using the latest Big Data tools. [0] http://en.wikipedia.org/wiki/Weather_derivative http://en.wikipedia.org/wiki/Weather_derivative

GFK_of_xmaspast 12y ago

Probably because if it worked they'd already be doing it.