3 ms·

Do we have enough historical weather data to train a decent AI on? Maybe this is already being done and is just a quieter area of research (in the headline sens

Do we have enough historical weather data to train a decent AI on? Maybe this is already being done and is just a quieter area of research (in the headline sense)? I know for example every time a hurricane comes toward land all the weather places trot out a half dozen different model predictions saying different things. It would be great if there was more confidence in this area for planning.

boredumb 3y ago
Yes there are - This is also amusingly enough what I just started doing in an attempt to play with Candle in the next few weeks when I have some spare time. I'm in a hurricane prone area and we're in the season now so it's on my mind.
threeseed 3y ago
> the weather places trot out a half dozen different model predictions saying different things This should give you a hint that it's not as simple as having enough weather data. You are trying to predict a chaotic weather system that is impacted by everything from sea temperatures, sun activity, wind, volcanoes, bushfires, atmosphere changes etc. Much of which we don't even have enough data for or a complete understanding of.
- goosinmouse 3y ago
  Yep, the main problem is the bottleneck of data. I mean think about it, we still fly planes full of scientists directly into the largest storms ever created, just to get a few points of data. Satellites can only do so much.
em500 3y ago
> Do we have enough historical weather data to train a decent AI on? I have no particular insights about weather data or models, but I want to note that some prediction problems are just not limited by historical data (as you seem to assume implicitly). Consider the simplest example of a coin toss: the outcome of the next toss in unpredictable no matter how much historical coin toss data you collect and no matter how sophisticated your AI/ML model. (The only thing you can predict is the fraction or amount of heads/tails in a large batch of coin tosses, and you don't need much historical data for that.)
mkl 3y ago
No, not by many orders of magnitude. Even if you had sensors measuring temperature, pressure, humidity, wind velocity, etc. in every cubic metre of atmosphere, weather is chaotic (a tiny variation in starting conditions can lead to completely different outcomes) so you'd still only be able to predict a limited time ahead. Meteorologists have been working on this problem for a long time, and have thought of and tried every obvious idea (and many more). Weather prediction is literally where modern chaos theory started: https://en.wikipedia.org/wiki/Chaos_theory#History:~:text=His%20interest%20in%20chaos%20came%20about%20accidentally%20through%20his%20work%20on%20weather%20prediction%20in%201961 https://en.wikipedia.org/wiki/Chaos_theory#History:~:text=Hi...

boredumb 3y ago

Yes there are - This is also amusingly enough what I just started doing in an attempt to play with Candle in the next few weeks when I have some spare time. I'm in a hurricane prone area and we're in the season now so it's on my mind.

threeseed 3y ago

> the weather places trot out a half dozen different model predictions saying different things This should give you a hint that it's not as simple as having enough weather data. You are trying to predict a chaotic weather system that is impacted by everything from sea temperatures, sun activity, wind, volcanoes, bushfires, atmosphere changes etc. Much of which we don't even have enough data for or a complete understanding of.

goosinmouse 3y ago
Yep, the main problem is the bottleneck of data. I mean think about it, we still fly planes full of scientists directly into the largest storms ever created, just to get a few points of data. Satellites can only do so much.

em500 3y ago

> Do we have enough historical weather data to train a decent AI on? I have no particular insights about weather data or models, but I want to note that some prediction problems are just not limited by historical data (as you seem to assume implicitly). Consider the simplest example of a coin toss: the outcome of the next toss in unpredictable no matter how much historical coin toss data you collect and no matter how sophisticated your AI/ML model. (The only thing you can predict is the fraction or amount of heads/tails in a large batch of coin tosses, and you don't need much historical data for that.)

mkl 3y ago

No, not by many orders of magnitude. Even if you had sensors measuring temperature, pressure, humidity, wind velocity, etc. in every cubic metre of atmosphere, weather is chaotic (a tiny variation in starting conditions can lead to completely different outcomes) so you'd still only be able to predict a limited time ahead. Meteorologists have been working on this problem for a long time, and have thought of and tried every obvious idea (and many more). Weather prediction is literally where modern chaos theory started: https://en.wikipedia.org/wiki/Chaos_theory#History:~:text=His%20interest%20in%20chaos%20came%20about%20accidentally%20through%20his%20work%20on%20weather%20prediction%20in%201961 https://en.wikipedia.org/wiki/Chaos_theory#History:~:text=Hi...