14 ms·
Forecasts need to have error bars
- datadrivenangel 3y agoThe interesting example in this article is nowcasting! The art of forecasting the present or past while you're waiting for data to come in. It's sloppy science / statistics to not haven error ranges.
- RandomLensman 3y agoNot easy to always say what the benefit is: if you present in-model uncertainty from a stochastic model that might still say nothing about an estimation error vs the actual process. For forecasting to show actual uncertainty you need to be in a quite luxurious position to know the data generating process. You could try to fudge it with a lot of historical data where available - but still...
- aidenn0 3y agoMakes sense that such a thing would exist, though it renders less funny my joke: "I'm almost clairvoyant; I can predict things shortly after they happen"
- doubled112 3y agoI really thought that this was going to be about the weather.
- nullindividual 3y agoSame, but in a human context, are mundane atmospheric events so far off today that error bars would have any practical value and/or potentially introduce confusion?
- NegativeLatency 3y agoFor this reason I really enjoy reading the text products and area forecast discussion for interesting weather: https://forecast.weather.gov/product.php?site=NWS&issuedby=pqr&product=AFD&format=CI&glossary=1 https://forecast.weather.gov/product.php?site=NWS&issuedby=p...
- doubled112 3y agoAnybody happen to know if there's anything more detailed from Environment Canada than their forecast pages? https://weather.gc.ca/city/pages/on-143_metric_e.html https://weather.gc.ca/city/pages/on-143_metric_e.html I really like that discussion type forecast.
- yakubin 3y agoAbsolutely. 15 years ago I could reasonably trust forecasts regarding whether it’s going to rain in a given location 2 days in advance. Today I can’t trust forecasts about whether it’s raining currently.
- Smoosh 3y agoIt seems unlikely that the modelling and forecasting has become worse, so I guess there is some sort of change happening to the climate making it more unstable and less predictable?
- lispisok 3y ago>I guess there is some sort of change happening to the climate making it more unstable and less predictable? I've been seeing this question come up a lot lately. The answer is no, weather forecasting continues to improve. The rate is about 1 day improvement every 10 years so a 5 day forecast today is as good as a 4 day forecast 10 years ago.
- LexGray 3y agoI think that is a change in definition. 15 years ago it was only rain if you were sure to get drenched. Now rain means 1mm of water hit the ground in your general vicinity. I blame an abundance of data combined people who refuse to get damp and need an umbrella if there is any chance at all.
- yakubin 3y agoNo. I mean getting drenched, when forecast predicted no rain. 1mm isn’t really noticeable by humans.
- kqr 3y agoSure -- just a few day outs the forecast is not much better than the climatological average -- see e.g. https://charts.ecmwf.int/products/opencharts_meteogram?base_time=202312041200&epsgram=classical_15d_with_climate&lat=59.3326&lon=18.0649&station_name=Stockholm https://charts.ecmwf.int/products/opencharts_meteogram?base_... Up until that point, error bars increase. At least to me, there's a big difference between "1 mm rain guaranteed" and "90 % chance of no rain but 10 % chance of 10 mm rain" but both have the same average.
- jsmith45 3y agoThe other thing is that many forecasts have a large range. So it might be that the left third of the forecast area has like a 99.9% chance of zero precipitation, the right third has a 99.9% chance of some precipitation, and there is some uncertainly about where the border will be, (and near the border might have less precipitation than the edge). The result is that I will be told ~50% chance of precipitation, but the places I care about might well be in the essentially 0% or essentially 100% parts. The single forecast for a decent size city problem impacts other parts like forecasted highs/low. Even without a front, in some cities that have incorporated much of their suburbs, it is not uncommon for the eastmost part and westmost part to differ by 5-6 degrees, and that is ignoring the inherent temperature differences found in the downtown areas.
- dguest 3y agoMe too, and I was looking forward to the thread that talks about error bars in weather models, which is totally a thing! It turns out the ECMWF does do an ensamble model where they run 51 concurrent models, presumably with slightly different initial conditions, or they vary the model parameters within some envelope. From these 51 models you can get a decent confidence interval. But this is a lower resolution model, run less frequently. I assume they don't do this with their "HRES" model (which has twice the spacial resolution) in an ensemble because, well, it's really expensive. [1]: https://en.wikipedia.org/wiki/Integrated_Forecast_System#Variants https://en.wikipedia.org/wiki/Integrated_Forecast_System#Var...
- lispisok 3y agoA lot of weather agencies across the world run ensembles including US, Canada, and the UK. Ensembles are the future of weather forecasting but weather models are so computationally heavy models have a resolution/forecast length tradeoff which is even bigger when trying to run 20-50 ensemble members. You can have a high resolution model that runs to 2 days or so or have a longer range model at much coarser resolution. ECMWF recently upgraded their ensemble to run at the same resolution as the HRES. The HRES is basically the ensemble control member at this point [1] [1] https://www.ecmwf.int/en/about/media-centre/news/2023/model-upgrade-increases-skill-and-unifies-medium-range-resolutions https://www.ecmwf.int/en/about/media-centre/news/2023/model-...
- earthscienceman 3y agoI'm someone working in the field with weather modelers/forecasters but not a modeler myself. Reading this discussion has been an absolute treat ... The most fascinating thing about the concept of error in models, including in ensembles, is you can only calculate and propagate error for contributors that you can quantify. There are many unquantifiable sources of error. Imagine a physical process that you are unaware of that propagates as a bias, for example ice nucleation via aerosols. Perhaps you don't even model aerosols. How do you account for error here? What does error even mean? Ensembles only show you intramodel variability. Which is like error, sort of, but only really represents a combination of "real" variability in initial conditions and how that propagates through your physics/parameterizations. "models" the HN commentators make for their businesses surely have parallel concepts, but I don't see anyone talking about them. Only discussion about the errors you know when the ugliest errors are the ones that no one knows.
- iruoy 3y agoI've been using meteoblue for a while now and they tell you how sure they are of their predictions. Right now I can see that they rate their predictability as medium for tomorrow, but high for the day after. https://content.meteoblue.com/en/research-education/specifications/weather-variables/predictability https://content.meteoblue.com/en/research-education/specific...
- kqr 3y agoI'll give you one better. The ECMWF publishes their probabilistic ensemble forecasts with boxplots for numeric probabilities: https://charts.ecmwf.int/products/opencharts_meteogram?base_time=202311211200&epsgram=classical_10d&lat=59.3326&lon=18.0649&station_name=Stockholm https://charts.ecmwf.int/products/opencharts_meteogram?base_... They also have one for precipitation type distribution: https://charts.ecmwf.int/products/opencharts_ptype_meteogram?base_time=202312041200&lat=59.3326&lon=18.0649&station_name=Stockholm https://charts.ecmwf.int/products/opencharts_ptype_meteogram...
- amichal 3y agoI have, in my life as a web developer, had multiple "academics" urgently demand that i remove error bands, bars, notes about outliers, confidence intervals etc from graphics at the last minute so people are not "confused" Its depressing
- Maxion 3y agoThe depressing part is that many people actually need them removed in order to not be confused.
- nonethewiser 3y agoBut aren’t they still confused without the error bars? Or confidently incorrect? And who could blame them, when that’s the information they’re given? It seems like the options are: - no error bars which mislead everyone - error bars which confuse some people and accurately inform others
- alistairSH 3y agoYep. See also: Complaints about poll results in the last few rounds of elections in the US. "The polls said Hillary would win!!!" (no, they didn't). It's not just error margins, it's an absence of statistics of any sort in secondary school (for a large number of students).
- marcosdumay 3y agoYeah, when people remove that kind of information to not confuse people, they are aiming into making them confidently incorrect.
- Izkata 3y agoAfter a lot of back-and-forth some years ago, we settled on a third option: If the error bars would be too big (for whatever definition of "too big" we used back then), don't show the data and instead show a "not enough data points" message. Otherwise, if we were showing the data, show it without the error bars.
- 3y ago
- mrguyorama 3y agoIf you are forecasting both "Crime" and "Economy", it's VERY likely you have domain expertise for neither.
- bo1024 3y agoTwo things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmaking, and when it gets disconnected, it can be unclear what the value is. It sounds like Rosenfield is trying to use forecasting to give added weight to his statistical conclusions about past data, which I agree sounds suspect. (2) it's not clear what the "error bars" should mean. One is a confidence interval[1] (e.g. model gives 95% chance the output will be within these bounds). Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome). [1] acknowledged: not the correct term
- deleted 3y ago[deleted]
- m-murphy 3y agoThat's not what a confidence interval is. A confidence interval is a random variable that covers the true value 95% of the time (assuming the model is correctly specified).
- bo1024 3y agoOk, the 'reverse' of a confidence interval then -- I haven't seen a term for the object I described other than misuse of CI in the way I did. ("Double quantile"?)
- m-murphy 3y agoYou're probably thinking of a predictive interval
- borroka 3y agoIt is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech industry, with the usual exceptions, uncertainty intervals, for example on a graph, are interpreted by those making decisions as aesthetic components of the graph ("the gray bands look good here") and not as anything even marginally related to a prediction.
- deleted 3y ago[deleted]
- esafak 3y agoUncertainty quantification is a neglected aspect of data science and especially machine learning. Practitioners do not always have the statistical background, and the ML crowd generally has a "predict first and asks questions later" mindset that precludes such niceties. I always demand error bars.
- figassis 3y agoSo is it really science? These are concepts from stats 101. And the reasons and need, and the risks of not having them are very clear. But you have millions being put into models without these pre-requisites, and being sold to people as solutions, and waved away as "if people buy is it's bc it has value". People also pay fraudsters.
- nradov 3y agoMostly not. Very few data "scientists" working in industry actually follow the scientific method. Instead they just mess around with various statistical techniques (including AI/ML) until they get a result that management likes.
- marcinzm 3y agoMost decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.
- rented_mule 3y agoYes, please! I was part of an org that ran thousands of online experiments over the course of several years. Having some sort of error bars when comparing the benefit of a new treatment gave a much better understanding. Some thought it clouded the issue. For example, when a new treatment caused a 1% "improvement", but the confidence interval extended from -10% to 10%, it was clear that the experiment didn't tell us how that metric was affected. This makes the decision feel more arbitrary. But that is exactly the point - the decision is arbitrary in that case, and the confidence interval tells us that, allowing us to focus on other trade-offs involved. If the confidence interval is 0.9% to 1.1%, we know that we can be much more confident in the effect. A big problem with this is that meaningful error bars can be extremely difficult to come by in some cases. For example, imagine having something like that for every prediction made by an ML model. I would love to have that, but I'm not aware of any reasonable way to achieve it for most types of models. The same goes for online experiments where a complicated experiment design is required because there isn't a way to do random allocation that results in sufficiently independent cohorts. On a similar note, regularly look at histograms (i.e., statistical distributions) for all important metrics. In one case, we were having speed issues in calls to a large web service. Many calls were completing in < 50 ms, but too many were tripping our 500 ms timeout. At the same time, we had noticed the emergence of two clear peaks in the speed histogram (i.e., it was a multimodal distribution). That caused us to dig a bit deeper and see that the two peaks represented logged-out and logged-in users. That knowledge allowed us to ignore wide swaths of code and spot the speed issues in some recently pushed personalization code that we might not have suspected otherwise.
- kqr 3y ago> This makes the decision feel more arbitrary. This is something I've started noticing more and more with experience: people really hate arbitrary decisions. People go to surprising lengths to add legitimacy to arbitrary decisions. Sometimes it takes the shape of statistical models that produce noise that is then paraded as signal. Often it comes from pseudo-experts who don't really have the methods and feedback loops to know what they are doing but they have a socially cultivated air of expertise so they can lend decisions legitimacy. (They used to be called witch-doctors, priests or astrologers, now they are management consultants and macroeconomists.) Me? I prefer to be explicit about what's going on and literally toss a coin. That is not the strategy to get big piles of shiny rocks though.
- clircle 3y agoEvery estimate/prediction/forecast/interpolation/extrapolation should have a confidence/prediction/ or tolerance interval (application dependent) that incorporates the assumptions that the team is putting into the problem.
- mightybyte 3y agoCompletely agree with this idea. And I would add a corollary...date estimates (i.e. deadlines) should also have error bars. After all, a date is a forecast. If a stakeholder asks for a date, they should also specify what kind of error bars they're looking for. A raw date with no estimate of uncertainty is meaningless. And correspondingly, if an engineer is giving a date to some other stakeholder, they should include some kind of uncertainty estimate with it. There's a huge difference between saying that something will be done by X date with 90% confidence versus three nines confidence.
- niebeendend 3y agoA deadline implies the upper limit of error bar cannot exceed it. That means you need to appropriately buffer to hit the deadline.
- mightybyte 3y agoI don't think that's the way it works out in practice. The fact of the matter is that deadlines are missed all the time. In many cases, there is no such thing as 100% certainty that you'll hit a "deadline"--there are always circumstances outside your control (global pandemics anyone?). There's just some implicit confidence threshold or other assumptions lurking around that probably need to be communicated. Do you want three 9s of confidence? Five 9s? Those things are very different and the cost to actually achieve the latter can often be prohibitive. Everyone benefits if we make explicit our pre-conceived idea of precisely what "cannot exceed" means.
- staunton 3y agoGoing even further, deadlines are often a tool for signaling "the organization is trying really hard to achieve this fast". The shorter the deadline (as long as it's at least somewhat in theory plausible), the harder you're trying. Often most people involved (even those deciding on the date for the deadline) know from the very start that the deadline will almost certainly be missed. I regularly see two kinds of deadlines. "Planning deadlines" describe an estimate when something will be done. "Signalling deadlines" signal priorities and motivation to employees or clients. Sometimes both exist in parallel for the same task and there is a subset of people who know both.
- Animats 3y agoLooking at the graph, changes in this decade are noise. But what happened back in 1990?
- netsharc 3y agoProbably no simple answer, but here's a long paper I just found: https://pubs.aeaweb.org/doi/pdf/10.1257/089533004773563485 https://pubs.aeaweb.org/doi/pdf/10.1257/089533004773563485 Another famous hypothesis is the phasing out of lead fuel: https://en.wikipedia.org/wiki/Lead%E2%80%93crime_hypothesis https://en.wikipedia.org/wiki/Lead%E2%80%93crime_hypothesis
- hcks 3y ago[flagged]
- predict_addict 3y agoLet me suggest a solution https://github.com/valeman/awesome-conformal-prediction https://github.com/valeman/awesome-conformal-prediction
- sobriquet9 3y agoThat's not a solution, but more like lots of different solutions, many too complicated to understand. I would suggest split conformal first.
- xchip 3y agoAnd also claims that say "x improves y", should include std and avg in the title.
- CalChris 3y agoI'm reminded of Walter Lewin's analogous point about measurements from his 8.01 lectures: any measurement that you make without any knowledge of the uncertainty is meaningless https://youtu.be/6htJHmPq0Os https://youtu.be/6htJHmPq0Os You could say that forecasts are measurements you make about the future.
- lnwlebjel 3y agoTo that point, similarly: "Being able to quantify uncertainty, and incorporate it into models, is what makes science quantitative, rather than qualitative. " - Lawrence M. Krauss From https://www.edge.org/response-detail/10459 https://www.edge.org/response-detail/10459
- _hyttioaoa_ 3y agoForecasts can also be useful without error bars. Sometimes all one needs is a point prediction to inform actions. But sometimes full knowledge of the predictive distribution is helpful or needed to make good decisions. "Point forecasts will always be wrong" - true that for continuous data but if you can predict that some stock will go to 2.01x it's value instead of 2x that's still helpful.
- lagrange77 3y agoThis is a great advantage of Gaussian Process Regression aka. Kriging. https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_process_prediction,_or_Kriging https://en.wikipedia.org/wiki/Gaussian_process#Gaussian_proc...
- BOOSTERHIDROGEN 3y agoWhat is the best explanations to error bars ?
- sobriquet9 3y agoIf you keep doing that for a long time, actuals will fall within the error bars about 95% of the time.
- amai 3y agosee https://errorbars.streamlit.app/ https://errorbars.streamlit.app/
- nothrowaways 3y agoLinear error rate
- Arrath 3y agoI'm just imagining adding error bars to my schedule forecasting (with schedules that are typically one the optimistic side thanks to management), with bars pointing in the bad direction, and seeing management still insist it'll take too long.
- y1zhou 3y agoFor prediction intervals that are guaranteed coverage, check out conformal prediction [1]. Works great especially for time series data. [1] https://github.com/valeman/awesome-conformal-prediction https://github.com/valeman/awesome-conformal-prediction
- KingOfCoders 3y agoTake the error bars for Scrum estimations, 3, 5, 8 - people treat them as real things although they have huge errors and are very course.
- jalapenos 3y agoDoesn't work. For instance in a business setting, if I say "it'll be done in 10 days +/- 4 days", they'll immediately say "ok so you're saying it'll be done in 14 days tops then". More effective to sound as unsure as possible, disclaim everything in slippery language, and promise to give updates to your predictions as soon as you realise they've changed (granted this wouldn't work as well for an anonymous reader situation like in this article).
- lordgrenville 3y agoIs it wrong of them to interpret you that way? What is the correct interpretation?
- ur-whale 3y agoNot just forecasts. Accounting should do it too in their reporting. I would love to see a balance sheet with a proper 'certainty range' around the values in there.
- amai 3y agoNot only forecast need error bars. Every statistic needs error bars. But even then most people interpret error bars wrongly, see e.g. https://errorbars.streamlit.app/ https://errorbars.streamlit.app/
- aurelien 3y agotwo inches deeper, is too much money! So Alphabet prefer to loose the idea, the contract, the business plan rather than make the things solid and works for a long time. a $ is a dollars, a project, is just a piece of fun. Alphabet LOL XD
- borg16 3y agoReminds me of this paper[1] > An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability > Traditionally, scientists have placed more emphasis on communicating inferential uncertainty (i.e., the precision of statistical estimates) compared to outcome variability (i.e., the predictability of individual outcomes). Here, we show that this can lead to sizable misperceptions about the implications of scientific results. Specifically, we present three preregistered, randomized experiments where participants saw the same scientific findings visualized as showing only inferential uncertainty, only outcome variability, or both and answered questions about the size and importance of findings they were shown. Our results, composed of responses from medical professionals, professional data scientists, and tenure-track faculty, show that the prevalent form of visualizing only inferential uncertainty can lead to significant overestimates of treatment effects, even among highly trained experts. In contrast, we find that depicting both inferential uncertainty and outcome variability leads to more accurate perceptions of results while appearing to leave other subjective impressions of the results unchanged, on average. [1] https://www.microsoft.com/en-us/research/publication/an-illusion-of-predictability-in-scientific-results-even-experts-confuse-inferential-uncertainty-and-outcome-variability/ https://www.microsoft.com/en-us/research/publication/an-illu...
- 3abiton 3y agoVery interesting reference, there is a whole field about Uncertainty modelling.