4 ms·

Many folks, when hearing forecasts have improved, will disagree. They can think of many times when the forecast was wrong, perhaps hilariously or disruptively s

Many folks, when hearing forecasts have improved, will disagree. They can think of many times when the forecast was wrong, perhaps hilariously or disruptively so. This is textbook availability bias. You overestimate the failure rate because the failures were memorable, especially the major failures. But the far more frequent accurate predictions are not recalled, because they don't leave as much of an emotional impression. For the same reason folks regularly overestimate crime rates, deaths from terrorism or rare diseases. Meteorology has an entire subfield devoted to studying the quality of forecasts[0]. Before you bandy about numbers and anecdotes, stop and tell me what measure you are using. Is it percentage of degrees centigrade? Well you can't, degrees centigrade is an interval measure, not a ratio measure. If you use Kelvins, which has a mathematically meaningful zero, the percentage accuracy is suddenly very good. How do you count near misses of very intense weather? Direct hits by systems that were less intense than expected? How do you account for systems with very gradual gradients over very wide areas? How do you account for being early by an hour or late by an hour, but nailing the storm surge? How do you score for confidence? What's your rule for weighting false alarms? How important is mean error vs absolute error vs variance, and why? Solve these and dozens of other problems of describing "accuracy" and weather forecasters might take you seriously. But until then it's probably worth accepting that they are the most effective profession of their kind and that we have a lot we can learn from them. [0] http://www.cawcr.gov.au/projects/verification/ http://www.cawcr.gov.au/projects/verification/

grawprog 8y ago
I worked outside every day for 4 years at least. Weather forecasts are so unreliable I stopped listening. I had better luck trying to figure out if i'd need rain gear in a day by just waking up and looking out the window in the morning than looking up or listening to a weather forecast. You really start to notice how wrong they are when you spend every day outside. I don't notice any more working inside again. But when you rely on weather forecasts everyday to figure out how miserable you're going to be in a day...you really notice..they might as well just be making things up half the time.
- bainsfather 8y ago
  Similar experience in the uk, in terms of whether it will rain or not. 3 hour, 6 hour etc forecasts are of little use - no better than looking out the window. Meanwhile wind speed and temperature seem ok on that timescale, but are (usually) less important to me than precipitation. It depends a lot on what part of the world you are in - I think that the uk is particularly hard to forecast.
- brootstrap 8y ago
  Hmmm. If you are needing down to the minute accurate timing of when rain is going to start & stop, sure nobody can predict that, for every location in the USA, all the time. I have to question where you are getting your weather forecast information??? I am a bachelor degree meteorologist, I dont use any apps that try to be 'hyper local' (omg i hate that word, it is the weather version of AI/ML hype). Just go to the government site www.weather.gov We can't predict exactly when it will rain, but we can get daily trends pretty good for 3-5 days out. This means you are five days ahead of the weather. You cant know "it will rain from 5:00am to 9:30am on february 4th, 2019". But you can know if we have a synoptic scale system that is moving through your area on that day, and know if there is a probability of rain. Where do you work everyday outside? i would be happy to give you a 5 - 10 day forecast right now actually and can tell you it will be good. I am supposed to be writing some tests and documentation but i can drop some WX knowledge if you need.
  - mannykannot 8y ago
    I get the feeling that this "hyper-local" mentality contributed to NYC and surrounding areas being caught flat-footed by the first-of-season snowfall at the end of last year. I have little understanding of meteorology, but it was clear, just from the forecast maps, that a small change in the location and timing of the rain/snow line would make a big difference to the outcome, and that something more than was forecast was a distinct possibility. As with many things, having a little understanding of the big picture can help you get more out of weather forecasts.
    - scott_s 8y ago
accuranalysis 8y ago
ForecastWatch, a company solely dedicated to analyzing the accuracy of weather forecasts, has a good study available: https://www.forecastwatch.com/wp-content/uploads/High_Temperature_Accuracy_Study_12_Years.pdf https://www.forecastwatch.com/wp-content/uploads/High_Temper... You can also check accuracy for your local zip code (in US) at: https://www.forecastadvisor.com/ https://www.forecastadvisor.com/ tldr: for the best US consumer weather forecasters the mean error on high/low temp for 1-5 day forecasts is around 3F and they hit within 3F within 70% of the time. 7-day has declining accuracy and 10-day is not much better than statistical analysis on historical trends.
plopz 8y ago
You can use this site to compare the different models using statistics. http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/
brootstrap 8y ago
Validating the wx is extremely hard. For instance, if you make a forecast, what do you use as the 'truth'? A weather station? Trust me, I've worked with plenty of station data, and it is trash. So now you are comparing a weather forecast against some suspect trash validation set. Nobody knows! The fact is, predictions will keep getting gradually better, more high res. But the fact our atmosphere & earth system is insanely complex. The system is chaotic so tiny small changes in each model run produce different results... We have code to represent some basic physics but still dont understand many of the important things. We still dont know exactly the rain process & snow formation process. We are trying to pinpoint the exact moment when dust & water vapor magically form into a cloud droplet so we can better describe the rain process. THEN we have to translate all of this insane math and physics into code.
- labster 8y ago
  Hi fellow meterologist hacker. I too have a B.S. in atm sci and enough years in grad school that I'm still paying for it lol. Yeah, the extent to which models are verified against their own data is pretty interesting -- even the reanalysis data only represents a best guess of a place with physics similar to Earth. It includes a lot of advected observations, thousands of miles from the original observation site, but still useful because at least you know something about the air over the oceans. Even if we really do get a good handle on cloud physics, ultimately there are limits on how much we can model. Parameterizations have to happen, because of limits to computer time -- a 24h forecast completed in 26h is worthless. I'm honestly surprised that they've been pushing to get models to third order and higher approximations, because the errors in your initial conditions are larger, but apparently every little bit helps. But forecasts are pretty good considering that weather modeling is a initial value problem where we only know the initial conditions at a few observation points of varying quality, a boundary value problem where the boundaries can have complex interactions with the internal state, and simulation run time really matters.
tropo 8y ago
Often I can't even say for sure if the forecast is wrong or not. For example, what is a 10% chance of rain at 10? That could mean 10% chance that rain will occur at some point from 10 to 11, 10% chance that it will be raining at a randomly selected point between 10 and 11, or 10% chance that it will continuously rain from 10 to 11. Also, it could be any of those three, except from 9:30 to 10:30. Also, it could be a 10% chance of raining at exactly 10. So right there, I see 7 possible interpretations. Add in a map, and we can apply those to an area. It may be a 10% chance of raining at some place within this region, but any one place gets a smaller chance. It may be a 10% chance everywhere in the region, but a 99% chance that some part of the region gets rain.
- labster 8y ago
  What it really means in terms of exact timing is that an AWOS (weather station) will record measurable precip between the 1000 report and the 1100 report. Sometimes percent means forecast confidence, sometimes it means that they can't predict what area will get hit with rain. Like synoptic conditions will say "this region will definitely get a few thunderstorms", but when you drop down to mesoscale then you can't tell which city will actually get the rain. In longer-term forecasts, 10% usually means "I have no idea what's going to happen, so I guess precip is possible then." At mid range, like 3-5 days, it usually means "we haven't nailed down the timing of this thing yet". If you actually want to know what the logic is, go read the forecast discussions.
porpoisely 8y ago
Certainly weather forecasts are generally more right than wrong, but for a science, you expect it to be closer to 100% correct than 80% correct. We don't care if weather forecasters take us seriously. The weather forecasters should care whether we take them seriously. For example, just the other day in NY, we had a forecast of light rain. We got a snow squall instead. That sure was fun. People aren't complaining about a forecast of 80 degrees but it was 81 instead. It's those forecast that call for light rain and you get a nice fun snow squall.
kevin_thibedeau 8y ago
If you live near where the jetstream boundary usually lies, forecasts are generally less accurate because small, less predictable shifts can have a large impact on daily weather.
- saltcured 8y ago
  Yes! As a west coast native and sometime visitor to the midwest, I'd say that is true for the whole western US at least. If you learn to read the forecasts and maps you can actually start to grok it and intuit the mechanics. You can start to imagine the jet stream whipping like a hose and understanding that the exact rate of its swing might vary, causing weather fronts to sweep across the plains a little earlier or a little later, or steering the atmospheric river a little north or south as it sprays the coast. This is the mechanism behind those sudden wild changes in Chicago, when you switch between northern to southern weather in a single day. And in the short-term data, you can watch a radar animation and get pretty good at predicting when a rain wall or big thunder cell is going to cross your location, and what to expect on a quick errand. The only thing you cannot reason your way through is when the maps indicate large areas of instability and you really just have to set your mental threat condition and wait to see whether something erupts over your head or not. Afternoon thunderstorms in the mountains or coasts often have this uncertainty to them.
  - labster 8y ago
    Nah, midlatitude west coasts are worse because of the lack of good data over the oceans. Satellites can't tell you as much as you think, especially in the vertical direction. Knowing how much water is in a column of air is useful, but not as useful as if you knew precisely which layers it was in. A considerable amount of data in models for California is from observations that were made in China and Japan, that flow with the wind across the Pacific. Or conversely, you can look at it as a data hole that advected across the ocean, too. I'd like to note that of the few blown forecasts we get in Southern California, almost none of them have to deal with the jet stream. It's actually the cut-off lows offshore that don't have steering winds, which makes it difficult to predict where it will go next. The same is true with hurricanes.

grawprog 8y ago

I worked outside every day for 4 years at least. Weather forecasts are so unreliable I stopped listening. I had better luck trying to figure out if i'd need rain gear in a day by just waking up and looking out the window in the morning than looking up or listening to a weather forecast. You really start to notice how wrong they are when you spend every day outside. I don't notice any more working inside again. But when you rely on weather forecasts everyday to figure out how miserable you're going to be in a day...you really notice..they might as well just be making things up half the time.

bainsfather 8y ago
Similar experience in the uk, in terms of whether it will rain or not. 3 hour, 6 hour etc forecasts are of little use - no better than looking out the window. Meanwhile wind speed and temperature seem ok on that timescale, but are (usually) less important to me than precipitation. It depends a lot on what part of the world you are in - I think that the uk is particularly hard to forecast.
brootstrap 8y ago
Hmmm. If you are needing down to the minute accurate timing of when rain is going to start & stop, sure nobody can predict that, for every location in the USA, all the time. I have to question where you are getting your weather forecast information??? I am a bachelor degree meteorologist, I dont use any apps that try to be 'hyper local' (omg i hate that word, it is the weather version of AI/ML hype). Just go to the government site www.weather.gov We can't predict exactly when it will rain, but we can get daily trends pretty good for 3-5 days out. This means you are five days ahead of the weather. You cant know "it will rain from 5:00am to 9:30am on february 4th, 2019". But you can know if we have a synoptic scale system that is moving through your area on that day, and know if there is a probability of rain. Where do you work everyday outside? i would be happy to give you a 5 - 10 day forecast right now actually and can tell you it will be good. I am supposed to be writing some tests and documentation but i can drop some WX knowledge if you need.
- mannykannot 8y ago
  I get the feeling that this "hyper-local" mentality contributed to NYC and surrounding areas being caught flat-footed by the first-of-season snowfall at the end of last year. I have little understanding of meteorology, but it was clear, just from the forecast maps, that a small change in the location and timing of the rain/snow line would make a big difference to the outcome, and that something more than was forecast was a distinct possibility. As with many things, having a little understanding of the big picture can help you get more out of weather forecasts.
  - scott_s 8y ago

accuranalysis 8y ago

ForecastWatch, a company solely dedicated to analyzing the accuracy of weather forecasts, has a good study available: https://www.forecastwatch.com/wp-content/uploads/High_Temperature_Accuracy_Study_12_Years.pdf https://www.forecastwatch.com/wp-content/uploads/High_Temper... You can also check accuracy for your local zip code (in US) at: https://www.forecastadvisor.com/ https://www.forecastadvisor.com/ tldr: for the best US consumer weather forecasters the mean error on high/low temp for 1-5 day forecasts is around 3F and they hit within 3F within 70% of the time. 7-day has declining accuracy and 10-day is not much better than statistical analysis on historical trends.

plopz 8y ago

You can use this site to compare the different models using statistics. http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/ http://www.emc.ncep.noaa.gov/gmb/STATS_vsdb/

brootstrap 8y ago

Validating the wx is extremely hard. For instance, if you make a forecast, what do you use as the 'truth'? A weather station? Trust me, I've worked with plenty of station data, and it is trash. So now you are comparing a weather forecast against some suspect trash validation set. Nobody knows! The fact is, predictions will keep getting gradually better, more high res. But the fact our atmosphere & earth system is insanely complex. The system is chaotic so tiny small changes in each model run produce different results... We have code to represent some basic physics but still dont understand many of the important things. We still dont know exactly the rain process & snow formation process. We are trying to pinpoint the exact moment when dust & water vapor magically form into a cloud droplet so we can better describe the rain process. THEN we have to translate all of this insane math and physics into code.

labster 8y ago
Hi fellow meterologist hacker. I too have a B.S. in atm sci and enough years in grad school that I'm still paying for it lol. Yeah, the extent to which models are verified against their own data is pretty interesting -- even the reanalysis data only represents a best guess of a place with physics similar to Earth. It includes a lot of advected observations, thousands of miles from the original observation site, but still useful because at least you know something about the air over the oceans. Even if we really do get a good handle on cloud physics, ultimately there are limits on how much we can model. Parameterizations have to happen, because of limits to computer time -- a 24h forecast completed in 26h is worthless. I'm honestly surprised that they've been pushing to get models to third order and higher approximations, because the errors in your initial conditions are larger, but apparently every little bit helps. But forecasts are pretty good considering that weather modeling is a initial value problem where we only know the initial conditions at a few observation points of varying quality, a boundary value problem where the boundaries can have complex interactions with the internal state, and simulation run time really matters.

tropo 8y ago

Often I can't even say for sure if the forecast is wrong or not. For example, what is a 10% chance of rain at 10? That could mean 10% chance that rain will occur at some point from 10 to 11, 10% chance that it will be raining at a randomly selected point between 10 and 11, or 10% chance that it will continuously rain from 10 to 11. Also, it could be any of those three, except from 9:30 to 10:30. Also, it could be a 10% chance of raining at exactly 10. So right there, I see 7 possible interpretations. Add in a map, and we can apply those to an area. It may be a 10% chance of raining at some place within this region, but any one place gets a smaller chance. It may be a 10% chance everywhere in the region, but a 99% chance that some part of the region gets rain.

labster 8y ago
What it really means in terms of exact timing is that an AWOS (weather station) will record measurable precip between the 1000 report and the 1100 report. Sometimes percent means forecast confidence, sometimes it means that they can't predict what area will get hit with rain. Like synoptic conditions will say "this region will definitely get a few thunderstorms", but when you drop down to mesoscale then you can't tell which city will actually get the rain. In longer-term forecasts, 10% usually means "I have no idea what's going to happen, so I guess precip is possible then." At mid range, like 3-5 days, it usually means "we haven't nailed down the timing of this thing yet". If you actually want to know what the logic is, go read the forecast discussions.

porpoisely 8y ago

Certainly weather forecasts are generally more right than wrong, but for a science, you expect it to be closer to 100% correct than 80% correct. We don't care if weather forecasters take us seriously. The weather forecasters should care whether we take them seriously. For example, just the other day in NY, we had a forecast of light rain. We got a snow squall instead. That sure was fun. People aren't complaining about a forecast of 80 degrees but it was 81 instead. It's those forecast that call for light rain and you get a nice fun snow squall.

kevin_thibedeau 8y ago

If you live near where the jetstream boundary usually lies, forecasts are generally less accurate because small, less predictable shifts can have a large impact on daily weather.

saltcured 8y ago
Yes! As a west coast native and sometime visitor to the midwest, I'd say that is true for the whole western US at least. If you learn to read the forecasts and maps you can actually start to grok it and intuit the mechanics. You can start to imagine the jet stream whipping like a hose and understanding that the exact rate of its swing might vary, causing weather fronts to sweep across the plains a little earlier or a little later, or steering the atmospheric river a little north or south as it sprays the coast. This is the mechanism behind those sudden wild changes in Chicago, when you switch between northern to southern weather in a single day. And in the short-term data, you can watch a radar animation and get pretty good at predicting when a rain wall or big thunder cell is going to cross your location, and what to expect on a quick errand. The only thing you cannot reason your way through is when the maps indicate large areas of instability and you really just have to set your mental threat condition and wait to see whether something erupts over your head or not. Afternoon thunderstorms in the mountains or coasts often have this uncertainty to them.
- labster 8y ago
  Nah, midlatitude west coasts are worse because of the lack of good data over the oceans. Satellites can't tell you as much as you think, especially in the vertical direction. Knowing how much water is in a column of air is useful, but not as useful as if you knew precisely which layers it was in. A considerable amount of data in models for California is from observations that were made in China and Japan, that flow with the wind across the Pacific. Or conversely, you can look at it as a data hole that advected across the ocean, too. I'd like to note that of the few blown forecasts we get in Southern California, almost none of them have to deal with the jet stream. It's actually the cut-off lows offshore that don't have steering winds, which makes it difficult to predict where it will go next. The same is true with hurricanes.