6 ms·

Why the snow forecast for New York City was so bad

56 points by cryptoz 12y ago

cryptoz 12y ago
While this article mentions that the GFS model was more accurate than the others (ECMWF & NAM), it neglects to give the backstory as to why it was more accurate and less trusted. For years the GFS has been made fun of because of its poor performance when compared to the Euro - and it just got the major upgrades it needed this month! New supercomputers, new high-res models, all went online just before the storm arrived. So, very few trusted the model even though it was in the best spot to get everything right. In an unexpected move, The Weather Channel was much more accurate than the US NWS / AccuWeather / etc because of their use of the GFS as a trusted model. More info in this article: http://www.alternet.org/environment/why-almost-everyone-got-snowpocalypse-wrong http://www.alternet.org/environment/why-almost-everyone-got-... Edit: And of course, it's my opinion that all the models are currently suffering from a lack of input data. The "...and what should be done" part of the headline, in my opinion, should be answered with, "everyone download PressureNet and contribute your phone's sensor data to weather models", so that we can better predict difficult quantities like heavy snowfall and severe thunderstorms: https://play.google.com/store/apps/details?id=ca.cumulonimbus.barometernetwork https://play.google.com/store/apps/details?id=ca.cumulonimbu... ;)
- eitally 12y ago
  Do you know of any mobile apps that use GFS as a source?
  - VLM 12y ago
    Using the chrome app you can go to http://weather.rap.ucar.edu/model/?model=gfs http://weather.rap.ucar.edu/model/?model=gfs I never tried it on my phone until a couple seconds ago. It looks pretty good! You can create a shortcut to that URL on android, presumably legacy phones can do the same.
- potash 12y ago
  Do you know of any research about the accuracy of forecasts as a function of sensor data input? I.e. what kind of an improvement will we see if some number of Android users install PressureNet?
  - cryptoz 12y ago
    Here is a recently published paper by Cliff Mass that is the first attempt at forecast experiments with smartphone pressure data: http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-13-00188.1 http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-13-00188... The early results are good. There will be an improvement [1]. How much of an improvement? We don't know yet. Cliff thinks it could be a revolution for some types of forecasts, but we don't have the density of sensors yet to know for sure. For some comparisons, we sent cliff about 20,000 measurements per hour for his experiment in that paper. We're now delivering about 200,000 per hour to researchers, and that's not nearly enough. Our aim is for 2M per hour and I hope to reach that in the next 2-3 months. Around 1-2M per hour is probably sufficient to provide the "revolution" in accuracy that Cliff predicting. I should also note that 1-2M per hour is small. We should be able to get closer to 1B per hour, but it'll take a while to ramp up to that kind of scale (that'll be like, every smartphone + watch + car that has a barometer). The improvements will be slow and steady until we get massive scale and are able to run our models in real time. Until then, it's tough to guess how good the improvements will be. [1] In the linked paper above, I believe the results were a reduction in root mean square error of about 1deg C for a 3-hour temperature forecast in the Pacific Northwest.
jacques_chester 12y ago
Several years ago in Australia, a blogger called Possum Comitatus, aka Scott Steel, used to mock journalists for breathlessly reporting every twitch of the polls, even though most such movements were within the margin of error. So successful was this mockery that the way polls are reported actually changed. It become normal to report a change in polls with a remark about whether the change was within the margin. Political race-calling has moved onto less abuses of statistics. Better reporting can be done, and done with surprising ease, using a simple tool. Mockery of journalists. To gain status and prestige over their peers, some will adopt the better method. In a few months they'll all do it.
joshdick 12y ago
People don't like it when 14 inches is forecast and the actual snowfall is 2 inches, but they aren't going to like hearing forecasts of 2-16 inches. That sort of wide confidence interval is more accurate, but it's also basically useless. How does a 2-16 inch forecast help a parent or school district or mayor plan for anything?
- mikeash 12y ago
  Yes, it's hard to plan for 2-16 inches. But that's a fault with reality, and not a reason to give bad forecasts. It's really hard to plan for hurricanes, so let's just give a forecast of "it's going to miss us"! That's much easier to plan for! Sometimes life is uncertain. The answer is to account for the uncertainty when planning, not insist that the information indicate certainty when it's not actually there.
- ScottBurson 12y ago
  You make contingency plans for both ends of the range, and most importantly, you know you have to check for updates frequently.
- saalweachter 12y ago
  A single 9x% confidence interval like 2-16 inches gives the impression of a continuous bell-curve distribution, when you might have a multiple peaks based on a couple of simple factors. I wonder if it would be useful to give out a couple of probability buckets. If people heard "20% chance of 10 or more inches, 60% chance of less than 2 inches", would then know to be ready for a lot of snow, but not be surprised if it doesn't happen, or would they ignore the considerable chance of a heavy snowfall?
- adamtj 12y ago
  I disagree. When forecasters know with certainty that there will be 14 inches, that's what they should say. It's definitely more accurate. However, when they only know that it might go either way, they should say "2-16 inches", which is more useful and accurate in the face of real uncertainty. Even better would be "2-16 inches, probably 14". A school administrator who sees a forecast of "14 inches" may simply cancel class the evening before, giving people time to prepare with certainty. That's nice, but not at the cost of a lost school day and calendar disruptions. A forecast of "2-16 inches, probably 14" might cause that same administrator to wait and see. TV and radio would broadcast the news about possible school closings the evening before, so parents could prepare. They might have a bit more difficulty planning, but that's worth it if the school day can be saved.
jeffreyrogers 12y ago
Alternative hypothesis that also fits the evidence: You look bad if reality turns out worse than your forecast. You look less bad if reality turns out better than your forecast, thus there is a built in incentive to make forecasts more pessimistic than is justified. Imagine if 10in were forecast and then NYC actually got 2 feet. Then everyone would be complaining that the city wasn't adequately prepared for the 14 extra inches they received and would blame the forecasters for all the economic consequences of not being prepared.
- notatoad 12y ago
  >Imagine if 10in were forecast and then NYC actually got 2 feet. i'm imagining it, and not really seeing the problem. 10" and 2' are both enough snow that you probably don't want to drive in it unless you have to. It might make a difference for the snow plow operators or the skiers, but for the average person the only thing they need to know is "should i plan to drive to work in the morning" and the weather forecasters correctly predicted "no".
  - 100timesthis 12y ago
    Another incentive is the audience: people will watch the news much more if the forecast is really bad.
- spikels 12y ago
  If I'm understanding the article correctly, this seems correct. Of the many models almost everyone chose to emphasized the models with the more extreme output. Different parties had different motivations: selling news, avoiding political blame... Interesting things happen when science meets human nature. Science rarely wins especially if there is uncertainty.
  - jeffreyrogers 12y ago
    > Science rarely wins especially if there is uncertainty. I agree 100%. One of the most interesting books I've read lately has been The Black Swan by Nassim Taleb, which deals with this at length (ostensibly it is about unpredictability in financial markets, but the implications are far broader than that).
dougk16 12y ago
After watching the same ridiculous news reporting touting "historic", "storm of the century", "unprecedented", etc., year after year, the more cynical side of me suggests that there's a small supplemental reason besides flawed models: there's probably good money to be made from overhyping storms. At the very least news stations must get great ratings the day or two beforehand. Note that this doesn't preclude accurate estimating...you just have to pull a bunch of cheap psychological tricks to get people to focus on the worst case scenario. The end result of people freaking out is still largely the same. One reason this comes to mind is that I can't ever remember a storm reaching its worst-case snowfall predictions, and usually they barely make the minimums. A rule of thumb I use is to treat the minimum prediction as the actual maximum and zero as the actual minimum. Anecdotal, but seems to work well.
- icehawk219 12y ago
  The "play up the fear" stuff around weather reporting in recent years is why I've given up even attempting to use weather forecasts from anywhere but weather.gov which is sourced directly from The National Weather Service and NOAA. It's just a straight up information dump, no annoying ads or fear mongering for clicks. When I heard they were naming blizzards last year, and this year now, I was only momentarily shocked because it was the next logical step. This has far more severe consequences then just clickbaiting though. When Hurricane Irene was moving up the coast a few years ago they did the typical "storm of the century, omg a billion people might die!!!" routine and we got light wind and rain. The following year they did it again with Hurricane Sandy (I really hate when people call it Superstorm Sandy ... it was a small hurricane! We were just ill-prepared for it) and a whole lot of people didn't buy into it and paid the price. I'm only 27 but I have trouble remembering a time where every thunderstorm we got wasn't played up to be an extinction level event. At this point I have my small cache of supplies and otherwise completely ignore storm predictions for the most part.
  - maxerickson 12y ago
    It wasn't technically a hurricane anymore when it made landfall on the mainland US, which probably contributed to the superstorm name sticking.
acheron 12y ago
More to the point, who cares? The storm was still pretty bad: this isn't a case of a "bust" forecast. Long Island got hit, as well as much of New England. So the line on the map of where the heaviest snow would be was a bit off: the only reason this is "news" at all is because that line happened to affect New York City, and there are a lot of journalists who live there whose view of the world matches that New Yorker cover[1]. If the line had been anywhere else we wouldn't have heard about it at all. [1] https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Avenue https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Ave...
- VLM 12y ago
  Given the continual clickbait alarmist stuff about it being a snow-hurricane (say what?) I'm surprised the hurricane analogy hasn't stuck deeper. Technically hurricane Katrina never hit New Orleans, it hit Slidell. I can assure you no one cares, other than maybe people living in Slidell. The rest of the world feels apathy due to lack of geographic knowledge. Boston, NYC, Long Island, whatever. Similar to the Katrina example, I'm sure the difference between LI and NYC is extremely important to the status seeking locals, but...
- CPLX 12y ago
  Fair enough, but it's worth noting that line is the difference between empty ocean and twenty million people, threee major airports and the financial capital of the world. It's sort of the definition of a distinction denoting an actual difference.
  - mikeash 12y ago
    I think the problem is that people don't really understand nonlinearities in these things. Imagine somebody aims a gun at you and I say, "Watch out, he's going to shoot you!" Then he fires the gun, but it misses you by an inch. You wouldn't say "you idiot, the bullet didn't hit me at all." But the weather equivalent has people thinking forecasters are stupid.
    - RunningWild 12y ago
mikeash 12y ago
Note that this article is written by the Capital Weather Gang, a group of DC-area meteorologists who consistently put out accurate and nuanced short-range forecasts of upcoming weather in the area. They give out great info and they aren't afraid to state their uncertainties up front, nor are they afraid to admit it when they're wrong (which is not too frequent). They give an implicit answer to the question of "how could everybody do better?" which is "be more like us." And I think that would be great.
basseq 12y ago
Three problems: 1. Uncertainty isn't actionable. A statement like, "There's an 85% chance of snow with accumulation of 1–24" with peak probabilities of 28% and 17% at 2" and 18", respectively" is nonsense to most people. And moreover, it doesn't mean anything to me. Are we going to get snow that will affect my morning commute? Are we going to get so much snow that I need to take emergency action? A probability distribution doesn't answer those questions, even if that's the best that exists. 2. Weather is news. The cable news channels were dominated by snow predictions yesterday, and the Weather Channel exists as an "entertainment" venue, not a source of scientific information. If it doesn't fit into a soundbite, the public can't absorb it. And "Weather is hard to predict" doesn't garner viewership. 3. Prepare for the worst. While small changes in the environment can lead to large variability in snow accumulation, the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue. Can you imagine the outcry—not to mention the impact—if the situation was reversed (i.e., surprise 3' of snow)? Fairfax County Public Schools outside of DC got slammed in the media for not closing a few weeks back. The real question is: at what point of probability do you prepare for a potential outcome, and are you prepared to back that up (either way)? Do we prepare for record snowfall on 80% likelihood? 50%? 10%? (And as a broader point of discussion, this same calculation applies to other areas: TSA spend vs. terrorism; flood insurance cost vs. coverage; etc.)
- Goronmon 12y ago
  the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue. The thing is, for the Boston area at least, the crippling snowfall totals came true. Areas west of Boston were reporting well over two feet of snow earlier in the day.
  - emhart 12y ago
    Yeah, southwest of Boston we are getting wrecked. Still coming down hard, hoping it finishes up soon. I'm glad we had the warnings, and I'll be particularly glad if we lose power, which seems pretty likely at this point, since the warning gave me the chance to stock up, charge battery backups, etc.
Sharlin 12y ago
I was delighted to notice a while ago that the Finnish Meteorological Institute had added 50% and 80% confidence intervals to their 10-day online forecasts, for instance: http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long
fideloper 12y ago
I was curious about what kind of bump weather sites get in ad revenue as a result of storm hype. That's an interesting angle when wondering where the hype is driven from ... if you find the mental image of Murdoch twirling his mustache (this is my imagination, so why not add a mustache?) while making quiet phone calls from a back room a correct version of reality.
mathattack 12y ago
It seems like the costs are asymmetric on misses. If a bunch of people die, the mayor gets voted out of office. If they prematurely close the city for a day, the mayor just apologizes afterwards.
toephu2 12y ago
We are in the year of 2015, and yet we still cannot accurately predict the weather...
scott_s 12y ago
We should always strive to improve our predictions, but I see a fundamental dichotomy: do you prefer false-positives, or false-negatives? We don't get to pick no wrong results. When it comes to predictions like snow storms, hurricanes and tornadoes, you obviously want to favor false-positives. It's better to predict something bad will happen, pay the preparation costs and have nothing happen, than to have something bad happen when you're unprepared. So, yes, we should try to improve our predictions. And perhaps we should be better at communicating our uncertainty. But I don't blame the meteorologists, or the people in government who shut things down and told everyone to prepare.
JackFr 12y ago
How is it possible that millions of New Yorkers accurately discounted the probability of it being a 500-year storm and yet the mayor didn't? The responsibility must lie with either the Mayor or the meteorologists. And there is something to own up to -- shutting down the city is not without real cost to real people. The mayor and governor will stay it was for our own good and better safe than sorry. But put a few million people under house arrest (no using roads -- under threat of arrest, no public transport) for basically no reason. But a few things to consider: 1) The meteorologists with the most dire predictions are the one who are going to be picked up by the media -- the media needs to sell ads. 2) The bulk of the population accurately discounted the most dire predictions, because of well-calibrated but subconscious Bayesian calculation. 3) Bureaucrats, and to some extent politicians of both parties, have a well-intentioned, but costly and wrong, need to subvert the judgement of millions and replace it with that of a handful of experts. When the judgement proves wrong, they rest on their good intentions without assuming responsibility. 4) In subverting these judgements bureaucrats and politicians can and do rely on the states monopoly on force.

56 points by cryptoz 12y ago

While this article mentions that the GFS model was more accurate than the others (ECMWF & NAM), it neglects to give the backstory as to why it was more accurate and less trusted. For years the GFS has been made fun of because of its poor performance when compared to the Euro - and it just got the major upgrades it needed this month! New supercomputers, new high-res models, all went online just before the storm arrived. So, very few trusted the model even though it was in the best spot to get everything right. In an unexpected move, The Weather Channel was much more accurate than the US NWS / AccuWeather / etc because of their use of the GFS as a trusted model. More info in this article: http://www.alternet.org/environment/why-almost-everyone-got-snowpocalypse-wrong http://www.alternet.org/environment/why-almost-everyone-got-... Edit: And of course, it's my opinion that all the models are currently suffering from a lack of input data. The "...and what should be done" part of the headline, in my opinion, should be answered with, "everyone download PressureNet and contribute your phone's sensor data to weather models", so that we can better predict difficult quantities like heavy snowfall and severe thunderstorms: https://play.google.com/store/apps/details?id=ca.cumulonimbus.barometernetwork https://play.google.com/store/apps/details?id=ca.cumulonimbu... ;)

eitally 12y ago
Do you know of any mobile apps that use GFS as a source?
- VLM 12y ago
  Using the chrome app you can go to http://weather.rap.ucar.edu/model/?model=gfs http://weather.rap.ucar.edu/model/?model=gfs I never tried it on my phone until a couple seconds ago. It looks pretty good! You can create a shortcut to that URL on android, presumably legacy phones can do the same.
potash 12y ago
Do you know of any research about the accuracy of forecasts as a function of sensor data input? I.e. what kind of an improvement will we see if some number of Android users install PressureNet?
- cryptoz 12y ago
  Here is a recently published paper by Cliff Mass that is the first attempt at forecast experiments with smartphone pressure data: http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-13-00188.1 http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-13-00188... The early results are good. There will be an improvement [1]. How much of an improvement? We don't know yet. Cliff thinks it could be a revolution for some types of forecasts, but we don't have the density of sensors yet to know for sure. For some comparisons, we sent cliff about 20,000 measurements per hour for his experiment in that paper. We're now delivering about 200,000 per hour to researchers, and that's not nearly enough. Our aim is for 2M per hour and I hope to reach that in the next 2-3 months. Around 1-2M per hour is probably sufficient to provide the "revolution" in accuracy that Cliff predicting. I should also note that 1-2M per hour is small. We should be able to get closer to 1B per hour, but it'll take a while to ramp up to that kind of scale (that'll be like, every smartphone + watch + car that has a barometer). The improvements will be slow and steady until we get massive scale and are able to run our models in real time. Until then, it's tough to guess how good the improvements will be. [1] In the linked paper above, I believe the results were a reduction in root mean square error of about 1deg C for a 3-hour temperature forecast in the Pacific Northwest.

jacques_chester 12y ago

Several years ago in Australia, a blogger called Possum Comitatus, aka Scott Steel, used to mock journalists for breathlessly reporting every twitch of the polls, even though most such movements were within the margin of error. So successful was this mockery that the way polls are reported actually changed. It become normal to report a change in polls with a remark about whether the change was within the margin. Political race-calling has moved onto less abuses of statistics. Better reporting can be done, and done with surprising ease, using a simple tool. Mockery of journalists. To gain status and prestige over their peers, some will adopt the better method. In a few months they'll all do it.

joshdick 12y ago

People don't like it when 14 inches is forecast and the actual snowfall is 2 inches, but they aren't going to like hearing forecasts of 2-16 inches. That sort of wide confidence interval is more accurate, but it's also basically useless. How does a 2-16 inch forecast help a parent or school district or mayor plan for anything?

mikeash 12y ago
Yes, it's hard to plan for 2-16 inches. But that's a fault with reality, and not a reason to give bad forecasts. It's really hard to plan for hurricanes, so let's just give a forecast of "it's going to miss us"! That's much easier to plan for! Sometimes life is uncertain. The answer is to account for the uncertainty when planning, not insist that the information indicate certainty when it's not actually there.
ScottBurson 12y ago
You make contingency plans for both ends of the range, and most importantly, you know you have to check for updates frequently.
saalweachter 12y ago
A single 9x% confidence interval like 2-16 inches gives the impression of a continuous bell-curve distribution, when you might have a multiple peaks based on a couple of simple factors. I wonder if it would be useful to give out a couple of probability buckets. If people heard "20% chance of 10 or more inches, 60% chance of less than 2 inches", would then know to be ready for a lot of snow, but not be surprised if it doesn't happen, or would they ignore the considerable chance of a heavy snowfall?
adamtj 12y ago
I disagree. When forecasters know with certainty that there will be 14 inches, that's what they should say. It's definitely more accurate. However, when they only know that it might go either way, they should say "2-16 inches", which is more useful and accurate in the face of real uncertainty. Even better would be "2-16 inches, probably 14". A school administrator who sees a forecast of "14 inches" may simply cancel class the evening before, giving people time to prepare with certainty. That's nice, but not at the cost of a lost school day and calendar disruptions. A forecast of "2-16 inches, probably 14" might cause that same administrator to wait and see. TV and radio would broadcast the news about possible school closings the evening before, so parents could prepare. They might have a bit more difficulty planning, but that's worth it if the school day can be saved.

jeffreyrogers 12y ago

Alternative hypothesis that also fits the evidence: You look bad if reality turns out worse than your forecast. You look less bad if reality turns out better than your forecast, thus there is a built in incentive to make forecasts more pessimistic than is justified. Imagine if 10in were forecast and then NYC actually got 2 feet. Then everyone would be complaining that the city wasn't adequately prepared for the 14 extra inches they received and would blame the forecasters for all the economic consequences of not being prepared.

notatoad 12y ago
>Imagine if 10in were forecast and then NYC actually got 2 feet. i'm imagining it, and not really seeing the problem. 10" and 2' are both enough snow that you probably don't want to drive in it unless you have to. It might make a difference for the snow plow operators or the skiers, but for the average person the only thing they need to know is "should i plan to drive to work in the morning" and the weather forecasters correctly predicted "no".
- 100timesthis 12y ago
  Another incentive is the audience: people will watch the news much more if the forecast is really bad.
spikels 12y ago
If I'm understanding the article correctly, this seems correct. Of the many models almost everyone chose to emphasized the models with the more extreme output. Different parties had different motivations: selling news, avoiding political blame... Interesting things happen when science meets human nature. Science rarely wins especially if there is uncertainty.
- jeffreyrogers 12y ago
  > Science rarely wins especially if there is uncertainty. I agree 100%. One of the most interesting books I've read lately has been The Black Swan by Nassim Taleb, which deals with this at length (ostensibly it is about unpredictability in financial markets, but the implications are far broader than that).

dougk16 12y ago

After watching the same ridiculous news reporting touting "historic", "storm of the century", "unprecedented", etc., year after year, the more cynical side of me suggests that there's a small supplemental reason besides flawed models: there's probably good money to be made from overhyping storms. At the very least news stations must get great ratings the day or two beforehand. Note that this doesn't preclude accurate estimating...you just have to pull a bunch of cheap psychological tricks to get people to focus on the worst case scenario. The end result of people freaking out is still largely the same. One reason this comes to mind is that I can't ever remember a storm reaching its worst-case snowfall predictions, and usually they barely make the minimums. A rule of thumb I use is to treat the minimum prediction as the actual maximum and zero as the actual minimum. Anecdotal, but seems to work well.

icehawk219 12y ago
The "play up the fear" stuff around weather reporting in recent years is why I've given up even attempting to use weather forecasts from anywhere but weather.gov which is sourced directly from The National Weather Service and NOAA. It's just a straight up information dump, no annoying ads or fear mongering for clicks. When I heard they were naming blizzards last year, and this year now, I was only momentarily shocked because it was the next logical step. This has far more severe consequences then just clickbaiting though. When Hurricane Irene was moving up the coast a few years ago they did the typical "storm of the century, omg a billion people might die!!!" routine and we got light wind and rain. The following year they did it again with Hurricane Sandy (I really hate when people call it Superstorm Sandy ... it was a small hurricane! We were just ill-prepared for it) and a whole lot of people didn't buy into it and paid the price. I'm only 27 but I have trouble remembering a time where every thunderstorm we got wasn't played up to be an extinction level event. At this point I have my small cache of supplies and otherwise completely ignore storm predictions for the most part.
- maxerickson 12y ago
  It wasn't technically a hurricane anymore when it made landfall on the mainland US, which probably contributed to the superstorm name sticking.

acheron 12y ago

More to the point, who cares? The storm was still pretty bad: this isn't a case of a "bust" forecast. Long Island got hit, as well as much of New England. So the line on the map of where the heaviest snow would be was a bit off: the only reason this is "news" at all is because that line happened to affect New York City, and there are a lot of journalists who live there whose view of the world matches that New Yorker cover[1]. If the line had been anywhere else we wouldn't have heard about it at all. [1] https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Avenue https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Ave...

VLM 12y ago
Given the continual clickbait alarmist stuff about it being a snow-hurricane (say what?) I'm surprised the hurricane analogy hasn't stuck deeper. Technically hurricane Katrina never hit New Orleans, it hit Slidell. I can assure you no one cares, other than maybe people living in Slidell. The rest of the world feels apathy due to lack of geographic knowledge. Boston, NYC, Long Island, whatever. Similar to the Katrina example, I'm sure the difference between LI and NYC is extremely important to the status seeking locals, but...
CPLX 12y ago
Fair enough, but it's worth noting that line is the difference between empty ocean and twenty million people, threee major airports and the financial capital of the world. It's sort of the definition of a distinction denoting an actual difference.
- mikeash 12y ago
  I think the problem is that people don't really understand nonlinearities in these things. Imagine somebody aims a gun at you and I say, "Watch out, he's going to shoot you!" Then he fires the gun, but it misses you by an inch. You wouldn't say "you idiot, the bullet didn't hit me at all." But the weather equivalent has people thinking forecasters are stupid.
  - RunningWild 12y ago

mikeash 12y ago

Note that this article is written by the Capital Weather Gang, a group of DC-area meteorologists who consistently put out accurate and nuanced short-range forecasts of upcoming weather in the area. They give out great info and they aren't afraid to state their uncertainties up front, nor are they afraid to admit it when they're wrong (which is not too frequent). They give an implicit answer to the question of "how could everybody do better?" which is "be more like us." And I think that would be great.

basseq 12y ago

Three problems: 1. Uncertainty isn't actionable. A statement like, "There's an 85% chance of snow with accumulation of 1–24" with peak probabilities of 28% and 17% at 2" and 18", respectively" is nonsense to most people. And moreover, it doesn't mean anything to me. Are we going to get snow that will affect my morning commute? Are we going to get so much snow that I need to take emergency action? A probability distribution doesn't answer those questions, even if that's the best that exists. 2. Weather is news. The cable news channels were dominated by snow predictions yesterday, and the Weather Channel exists as an "entertainment" venue, not a source of scientific information. If it doesn't fit into a soundbite, the public can't absorb it. And "Weather is hard to predict" doesn't garner viewership. 3. Prepare for the worst. While small changes in the environment can lead to large variability in snow accumulation, the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue. Can you imagine the outcry—not to mention the impact—if the situation was reversed (i.e., surprise 3' of snow)? Fairfax County Public Schools outside of DC got slammed in the media for not closing a few weeks back. The real question is: at what point of probability do you prepare for a potential outcome, and are you prepared to back that up (either way)? Do we prepare for record snowfall on 80% likelihood? 50%? 10%? (And as a broader point of discussion, this same calculation applies to other areas: TSA spend vs. terrorism; flood insurance cost vs. coverage; etc.)

Goronmon 12y ago
the fact of the matter is that there was a not-unlikely chance (according to the models) that NYC, Boston, and other cities could have seen crippling snowfall. They were able to prepare for that eventuality. That it didn't come to pass is almost a non-issue. The thing is, for the Boston area at least, the crippling snowfall totals came true. Areas west of Boston were reporting well over two feet of snow earlier in the day.
- emhart 12y ago
  Yeah, southwest of Boston we are getting wrecked. Still coming down hard, hoping it finishes up soon. I'm glad we had the warnings, and I'll be particularly glad if we lose power, which seems pretty likely at this point, since the warning gave me the chance to stock up, charge battery backups, etc.

Sharlin 12y ago

I was delighted to notice a while ago that the Finnish Meteorological Institute had added 50% and 80% confidence intervals to their 10-day online forecasts, for instance: http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long

fideloper 12y ago

I was curious about what kind of bump weather sites get in ad revenue as a result of storm hype. That's an interesting angle when wondering where the hype is driven from ... if you find the mental image of Murdoch twirling his mustache (this is my imagination, so why not add a mustache?) while making quiet phone calls from a back room a correct version of reality.

mathattack 12y ago

It seems like the costs are asymmetric on misses. If a bunch of people die, the mayor gets voted out of office. If they prematurely close the city for a day, the mayor just apologizes afterwards.

toephu2 12y ago

We are in the year of 2015, and yet we still cannot accurately predict the weather...

scott_s 12y ago

We should always strive to improve our predictions, but I see a fundamental dichotomy: do you prefer false-positives, or false-negatives? We don't get to pick no wrong results. When it comes to predictions like snow storms, hurricanes and tornadoes, you obviously want to favor false-positives. It's better to predict something bad will happen, pay the preparation costs and have nothing happen, than to have something bad happen when you're unprepared. So, yes, we should try to improve our predictions. And perhaps we should be better at communicating our uncertainty. But I don't blame the meteorologists, or the people in government who shut things down and told everyone to prepare.

JackFr 12y ago

How is it possible that millions of New Yorkers accurately discounted the probability of it being a 500-year storm and yet the mayor didn't? The responsibility must lie with either the Mayor or the meteorologists. And there is something to own up to -- shutting down the city is not without real cost to real people. The mayor and governor will stay it was for our own good and better safe than sorry. But put a few million people under house arrest (no using roads -- under threat of arrest, no public transport) for basically no reason. But a few things to consider: 1) The meteorologists with the most dire predictions are the one who are going to be picked up by the media -- the media needs to sell ads. 2) The bulk of the population accurately discounted the most dire predictions, because of well-calibrated but subconscious Bayesian calculation. 3) Bureaucrats, and to some extent politicians of both parties, have a well-intentioned, but costly and wrong, need to subvert the judgement of millions and replace it with that of a handful of experts. When the judgement proves wrong, they rest on their good intentions without assuming responsibility. 4) In subverting these judgements bureaucrats and politicians can and do rely on the states monopoly on force.