6 ms·

Two things I think are interesting here, one discussed by the author and one not. (1) As mentioned at the bottom, forecasting usually should lead to decisionmak

• deleted 3y ago
  [deleted]
• m-murphy 3y ago
  That's not what a confidence interval is. A confidence interval is a random variable that covers the true value 95% of the time (assuming the model is correctly specified).
  • bo1024 3y ago
    Ok, the 'reverse' of a confidence interval then -- I haven't seen a term for the object I described other than misuse of CI in the way I did. ("Double quantile"?)
    • m-murphy 3y ago
      You're probably thinking of a predictive interval
      • borroka 3y ago
        It is a very common misconception and one of my technical crusades. I keep fighting, but I think I have lost. Not knowing what the "uncertainty interval" represents (is it, loosely speaking, an expectation about a mean/true value or about the distribution of unobserved values?) could be even more dangerous, in theory, than using no uncertainty interval at all. I say in theory because, in my experience in the tech industry, with the usual exceptions, uncertainty intervals, for example on a graph, are interpreted by those making decisions as aesthetic components of the graph ("the gray bands look good here") and not as anything even marginally related to a prediction.
        • m-murphy 3y ago
          Agreed! I also think it's extremely important as practitioners to know what we're even trying to estimate. Expected value (i.e. least squares regression) is the usual first thing to go for, does that even matter? We're probably actually interested in something like an upper quantile for planning purposes. And then the whole model component of it, the interval that's being simultaneously estimated is model driven and if that's wrong, then the interval is meaningless. There's a lot of space for super interesting and impactful work in this area IMO, once you (the practitioner) think more critically about the objective. And then don't even get me started on interventions and causal inference...
          • fjkdlsjflkds 3y ago
            > We're probably actually interested in something like an upper quantile for planning purposes. True. But a conditional quantile is much harder to accurately estimate from data than a conditional expectation (particularly if you are talking about extreme quantiles).
            • m-murphy 3y ago
              Oh absolutely, so it's all the more important to be precise in what we're estimating and for what purpose, and to be honest about our ability to estimate it with appropriate uncertainty quantification (such as by using conformal prediction methods/bootstrapping).
          • borroka 3y ago
            From a statistical point of view, I agree that there is a lot of interesting and impactful work to be done on estimating predictive intervals, more in ML than in traditional statistical modeling. I have more doubts when it comes to actions taken when considering properly estimated predictive intervals. Even I, who have a good knowledge of statistical modeling, after hearing "the median survival time for this disease is 5 years," do not stop to think that the median is calculated/estimated on an empirical distribution, so there are people who presumably die after 2 years, others after 8. Well, that depends on the variance. But if I am so strongly drawn to a central estimate, is there any chance for others not so used to thinking about distributions?
      • bo1024 3y ago
        Yes, that term captures what I'm talking about.
    • cubefox 3y ago
      "Credible interval": https://en.wikipedia.org/wiki/Credible_interval https://en.wikipedia.org/wiki/Credible_interval
      • bo1024 3y ago
        No, predictive interval is more precise, since we are dealing with predicting an observation rather than forming a belief about a parameter.
        • cubefox 3y ago
          What's a predictive interval?
          • bo1024 3y ago
            I don't normally use that term, but someone else in reply to me did, and it captures what I wanted to say: https://en.wikipedia.org/wiki/Prediction_interval https://en.wikipedia.org/wiki/Prediction_interval
• ramblenode 3y ago
  > Another is a standard deviation (i.e. you are pretty much predicting the squared difference between your own point forecast and the outcome). What you probably want is the standard error, because you are not interested in how much your data differ from each other but in how much your data differ from the true population.
  • bo1024 3y ago
    I don't see how standard error applies here. You are only going to get one data point, e.g. "violent crime rate in 2023". What I mean is a prediction, not only of what you think the number is, but also of how wrong you think your prediction will be.
    • nonameiguess 3y ago
      Standard error is exactly what the statsmodels ARIMA.PredictionResults object actually gives you and the confidence interval in this chart is constructed from a formula that uses the standard error. ARIMA is based on a few assumptions. One, there exists some "true" mean value for the parameter you're trying to estimate, in this case violent crime rate. Two, the value you measure in any given period will be this true mean plus some random error term. Three, the value you measure in successive periods will regress back toward the mean. The "true mean" and error terms are both random variables, not a single value but a distribution of values, and when you add them up to get the predicted measurement for future periods, that is also a random variable with a distribution of values, and it has a standard error and confidence intervals and these are exactly what the article is saying should be included in any graphical report of the model output. This is a characteristic of the model. What you're asking for, "how wrong do you think the model is," is a reasonable thing to ask for, but different and much harder to quantify.
      • bo1024 3y ago
        Thanks for explaining how it works - I don't use R (I assume this is R). This does not seem like a good way to produce "error bars" around a forecast like the one in this case study. It seems more like a note about how much volatility there has been in the past.
• hgomersall 3y ago
  Error bars in forecasts can only mean uncertainty your model has. Without error bars over models, you can say nothing about how good your model is. Even with them, your hypermodel may be inadequate.
  • bo1024 3y ago
    To me, this comes back to the question of skin in the game. If you have skin in the game, then you produce the best uncertainty estimates you can (by any means). If you don't, you just sit back and say "well these are the error bars my model came up with".
    • PeterisP 3y ago
      There are ways of scoring forecasts that reward accurate-and-certain forecasts in a manner where it's provably optimal to provide the most accurate estimates for your (un)certainty as you can.
      • bo1024 3y ago
        Yes, of course. I don't see that as very related to my point. For example, consider how 538 or The Economist predict elections. They might claim they'll use squared error or log score, but when it comes down to a big mistake, they'll blame it on factors outside their models.
        • eru 3y ago
          Well, but at least 538 has a reputation to defend as an accurate forecaster. So they have some skin in the game. (Of course, that's not as good as betting money.)
    • hgomersall 3y ago
      It's worse than that. Oftentimes the skin in the game provides a motivation to mislead. C.f. most of the economics profession.
• pacbard 3y ago
  As far as error bars are concerned, you could report some% credible intervals calculated from taking the some%tile out of your results. It’s somewhat Bayesian thinking but it will work better than confidence intervals. The intuition would be that some% of your forecasts are between the bounds of the credible interval.
• mnky9800n 3y ago
  Recently someone on hacker news described statistics as trying to measure how surprised you should be when you are wrong. Big fat error bars would give you the idea that you should expect to be wrong. Skinny ones would highlight that it might be somewhat upsetting to find out you are wrong. I don't think this is an exhaustive description of statistics but I do find it useful when thinking about forecasts.
  • nonameiguess 3y ago
    This can be true but depends on other sources of error being small enough. Standard error is just a formula and it varies inversely with the square of the sample size, so you trivially narrow a confidence interval by sampling more often. In this specific case, imagine you had daily measures of violent crime instead of only annual. You'd get much tighter error bars. Does that mean you should be more surprised if your predictions are wrong? It depends. You've only reduced model error, but this is the classic precision versus accuracy problem. You can very precisely estimate the wrong number. Does the model really reflect the underlying data generating process? Are the inputs you're giving reliable measurements? If both answers are yes, then your more precise model should be converging toward a better prediction of the true value, but if not, you're only getting a better prediction of the wrong thing. We can ask these questions with this very example. Clearly, ARIMA is not a causally realistic model. Criminals don't look at last year's crime rates and decide whether to commit a crime based on that. The assumption is that, whatever actually does cause crime, it tends to happen at fairly similar levels year to year, that is, 2020 should be more different than 2010 than it is different from 2019. We may not know what the causually relevant factors really are or we may not be able to measure them, but we at least assume they follow that kind of rule. This sounds plausible to me, but is it true? We can backtest by making predictions of past years and seeing how close they are to the measured value, but the possibility of this even working depends upon the answer to the second question. So then the second question. Is the national violent crime data actually reliable? I don't know the answer to that, but it certainly isn't perfect. There is a real crime rate for every crime, but it isn't exactly the reported number. Recording and reporting standards vary from jurisdiction to jurisdiction. Many categories of crime go undereported, the extent of which can change over time. Changes may reflect different policing emphasis as much as or more than changes in the underlying true rate. I believe the way the FBI even collects and categorizes data has changed in the past, so I'm not sure a measurement from 1960 can be meaningfully compared to a measurement from 2020. Ultimately, "how surprised you should be when you are wrong" needs to take all of these sources of error into account, not just the model's coefficient uncertainty.
• lr1970 3y ago
  Error bars (either confidence interval or standard deviations) are of little use because they do not tell you how the probability is distributed within the confidence interval band. The holy grail of forecasting is the Probabilistic Forecast that predicts the entire posterior distribution so that you can sample from it generating scenarios or realizations of the underlying random process.
  • americast 3y ago
    While I agree, we can always have multiple overlapping error bars to understand how the probability is distributed. I am not sure how a probabilistic forecast method is able to perform this better because the confidence interval is always generated through sampling in either situation. Though probabilistic forecasting methods may have a Bayesian approach, it is Monte Carlo sampling that helps generate the confidence intervals. Feel free to correct me if I am wrong! Thanks :)
• ISL 3y ago
  A position espoused by Bill Phillips [1], and to which I now adhere: "You should be willing to take either side of the bet that confidence interval implies." (paraphrasing; he says it better). For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. Aside from being generally correct, this approach is immediately actionable by making the meaning more visceral in discussions of uncertainty. Done right, it pushes you to assign uncertainties that are neither too conservative nor too optimistic. If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible. Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) [1] https://www.nobelprize.org/prizes/physics/1997/phillips/facts/ https://www.nobelprize.org/prizes/physics/1997/phillips/fact...
  • eru 3y ago
    > Edit for OP's explicit question: One standard-deviation errorbars are 68% confidence intervals. Two standard deviations are 95% confidence intervals. (assuming you're a frequentist, of course) Also assuming normal distribution, I think? > If the notion of letting your reader take either side of the bet makes your stomach a little queasy, you're on the right track. The feeling will subside when you're pretty sure you got the errorbar right and your reasoning is documented and defensible. > For a concrete example, with a 95% confidence interval, you should be as willing to accept the 19:1 odds that the true value is outside the interval as you are the 1:19 odds that the true value is inside the interval. I would like to build some edge into my bets. If a reader takes both sides of your example, they would be come out exactly even. But since readers are not forced to take any side at all, they will only take the bet if one of the sides has an advantage. So I would like to be able to say, 20:1 payout the true value is inside the error bar, and 1:10 payout it's outside the error bar (or something like that). The tighter the spread I am willing to quote, the more confident I am that I got the error estimates right. (I'm not sure how you translate these spreads back into the language of statistics.)