5 ms·

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it m

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.

nabla9 10y ago
True model is the probability distribution that generates the observed data.
- DiabloD3 10y ago
  In other words, what was actually observed?
  - lottin 10y ago
    The stochastic process that generated the observed data.
  - irremediable 10y ago
    Not really. What is observed is data, not a model/distribution. One can fit a given model to this data, but they do not, on their own, tell you which model generated them.
  - enthdegree 10y ago
    Maybe you've gotten it but I don't know. Here is a maybe over-tired coin-flipping example: Say you have a coin that might be unfair, and you want to estimate its bias. You flip it a bunch of times, and it mostly lands heads. Predictions of the coin's bias based on this observed data are usually going to be that it's biased some % towards heads. (unless maybe you have a strong prior, but that's a different topic) But there is also a chance that really the coin is fair, or even biased towards tails and you just got unlucky in your flips. There's a mismatch between the true model (the coins actual bias) and the observed data (the result of your flips) because of this chance of `unluckiness'
  - nabla9 10y ago
    No. "true model", "true distribution" and "true population" is what generates the data.
    - jsprogrammer 10y ago
      Which is a misnomer, because the data probably isn't generated by a model.
      - tomrod 10y ago
        Semantics at this point. Data-generating process is a term that's also used. A model seeks to mimic or match the real process to a reasonable approximation. Hence "true model" as a scoring engine or data relationship that represents reality completely.
        jsprogrammer 10y ago
        Except that we can't access reality completely. Assuming that there is a "true model" generating data is just that: an assumption with no real basis. Semantics is the essence of communication.
        pessimizer 10y ago
        Unless you have data, that is. The data is the basis for assuming that a process has generated data. Either that, or the data has existed for all eternity, and therefore could never have been collected.
        jsprogrammer 10y ago
        "data" is your own perception, however. Can you give an example of a data generating process?
        tomrod 10y ago
        We can create them! Suppose I take the function y = log(x) and add random white noise. The function log() and the parameters on the random white noise process are the data generating process. We could then fit a model y = \beta X + \epsilon, and then compare the "true" (first) model to our second model. When the natural world generates our data, the idea behind all this is the same: there is a process which generates the data, and the data reveals information about that process to an approximate degree. Some reads: [1] http://www.rimini.unibo.it/fanelli/econometric_models2_2012.pdf http://www.rimini.unibo.it/fanelli/econometric_models2_2012.... [2] https://en.wikipedia.org/wiki/Data_generating_process https://en.wikipedia.org/wiki/Data_generating_process
- unfamiliar 10y ago
  No, the "true model" he seems to be referring to is the mean of the distribution of the noisy observations. In the example, the distribution is the normal distribution and the mean is log(x), which he is referring to as the "true model". The notation is pretty sloppy for someone handing out statistical advice.
  - nabla9 10y ago
    I don't think his usage of these terms is sloppy at all. The main point of his argument is that R² compares to observed mean and not into true model behind the data. Terms "true model", "true distribution" and "true population" are well defined concepts in statistics – unless Jorma Rissanen is nearby.
  - bunderbunder 10y ago
    The true model is y = log(x) + u Where u is an error/noise term, and in this case takes a normal distribution. The error term is meant to Hoover up all the stuff you can't account for because it's impossible to measure everything. As ever, what you're trying to estimate is the parameters for everything but the error term. He's maybe being a little sloppy with the notation, but TBH it's a pedantic distinction that I haven't seen anyone make a big deal of since I last took intro stats. In R, for example, you'd specify the model as "y ~ log(x)" and leave it at that.
yiyus 10y ago
In the example, log(x) is the true model. It is generally unknown.
tel 10y ago
It's a bit of a philosophical thing. The data arose because of results of some large set of processes, all unobserved. Assuming a deterministic world—just for convenience of argument here—there's no actual underlying distribution which caused the data. Just a big equation with too many unknowns. In practice, though, these systems are often well-described by distributions. In common parlance for statistics, people thus sometimes like to talk about that large set of unknowns as though it really were just some arbitrarily complex distribution which we don't know: the true model. From a frequentist perspective, the idea of the true model is very important. Many arguments arise by considering it and trying to understand how to reduce the distance between your estimators and the unknown truth. Bayesian foundations avoid this entirely, refusing to posit an unknown and instead talking about refinements of what we know.
- obastani 10y ago
  It's not just philosophical -- there are examples of practical situations where there is a true model (especially in more traditional applications of statistics): - Polling for presidential election: The "true model" is the voting preferences of all 300 million Americans. A (uniformly) random sample of N Americans can be used to estimate the true model roughly with standard error 1/sqrt(N). - Particle physics: The "true model" is the decay probabilities of various particles as computed by quantum mechanics. Different models (and parameters) yield different decay probabilities, and experiments can be used to choose between models and/or estimate model parameters. Of course, oftentimes the true model is philosophical as you describe.
  - tel 10y ago
    That's reasonably fair, but it's worth noting that even in these situations the true circumstance might be a little more nuanced than the idea of a "true model" suggests. - In polling, it's a bit of an ideal world idea to think that the true voting preferences of all Americans are (1) fixed, (2) consistently measurable, or (3) even relevant given a lot of people won't vote. These are all sources of unknowns and variances which make me pretty unhappy with the idea of the idea of a true model even here. - In physics there's definitely a notion of a true model taken under the assumption that one set of equations and models is "correct" (which is its own philosophical problem!) but even then experiments don't measure this true model perfectly. They're also based on correction for variance in measurement and tooling which is assumed to be eventually ignorable. In any case, this has a much more definite notion of a true model, but I still find it difficult to swallow all together.

nabla9 10y ago

True model is the probability distribution that generates the observed data.

DiabloD3 10y ago
In other words, what was actually observed?
- lottin 10y ago
  The stochastic process that generated the observed data.
- irremediable 10y ago
  Not really. What is observed is data, not a model/distribution. One can fit a given model to this data, but they do not, on their own, tell you which model generated them.
- enthdegree 10y ago
  Maybe you've gotten it but I don't know. Here is a maybe over-tired coin-flipping example: Say you have a coin that might be unfair, and you want to estimate its bias. You flip it a bunch of times, and it mostly lands heads. Predictions of the coin's bias based on this observed data are usually going to be that it's biased some % towards heads. (unless maybe you have a strong prior, but that's a different topic) But there is also a chance that really the coin is fair, or even biased towards tails and you just got unlucky in your flips. There's a mismatch between the true model (the coins actual bias) and the observed data (the result of your flips) because of this chance of `unluckiness'
- nabla9 10y ago
  No. "true model", "true distribution" and "true population" is what generates the data.
  - jsprogrammer 10y ago
    Which is a misnomer, because the data probably isn't generated by a model.
    - tomrod 10y ago
      Semantics at this point. Data-generating process is a term that's also used. A model seeks to mimic or match the real process to a reasonable approximation. Hence "true model" as a scoring engine or data relationship that represents reality completely.
      - jsprogrammer 10y ago
        Except that we can't access reality completely. Assuming that there is a "true model" generating data is just that: an assumption with no real basis. Semantics is the essence of communication.
        pessimizer 10y ago
        Unless you have data, that is. The data is the basis for assuming that a process has generated data. Either that, or the data has existed for all eternity, and therefore could never have been collected.
        jsprogrammer 10y ago
        "data" is your own perception, however. Can you give an example of a data generating process?
        tomrod 10y ago
        We can create them! Suppose I take the function y = log(x) and add random white noise. The function log() and the parameters on the random white noise process are the data generating process. We could then fit a model y = \beta X + \epsilon, and then compare the "true" (first) model to our second model. When the natural world generates our data, the idea behind all this is the same: there is a process which generates the data, and the data reveals information about that process to an approximate degree. Some reads: [1] http://www.rimini.unibo.it/fanelli/econometric_models2_2012.pdf http://www.rimini.unibo.it/fanelli/econometric_models2_2012.... [2] https://en.wikipedia.org/wiki/Data_generating_process https://en.wikipedia.org/wiki/Data_generating_process
unfamiliar 10y ago
No, the "true model" he seems to be referring to is the mean of the distribution of the noisy observations. In the example, the distribution is the normal distribution and the mean is log(x), which he is referring to as the "true model". The notation is pretty sloppy for someone handing out statistical advice.
- nabla9 10y ago
  I don't think his usage of these terms is sloppy at all. The main point of his argument is that R² compares to observed mean and not into true model behind the data. Terms "true model", "true distribution" and "true population" are well defined concepts in statistics – unless Jorma Rissanen is nearby.
- bunderbunder 10y ago
  The true model is y = log(x) + u Where u is an error/noise term, and in this case takes a normal distribution. The error term is meant to Hoover up all the stuff you can't account for because it's impossible to measure everything. As ever, what you're trying to estimate is the parameters for everything but the error term. He's maybe being a little sloppy with the notation, but TBH it's a pedantic distinction that I haven't seen anyone make a big deal of since I last took intro stats. In R, for example, you'd specify the model as "y ~ log(x)" and leave it at that.

yiyus 10y ago

In the example, log(x) is the true model. It is generally unknown.

tel 10y ago

It's a bit of a philosophical thing. The data arose because of results of some large set of processes, all unobserved. Assuming a deterministic world—just for convenience of argument here—there's no actual underlying distribution which caused the data. Just a big equation with too many unknowns. In practice, though, these systems are often well-described by distributions. In common parlance for statistics, people thus sometimes like to talk about that large set of unknowns as though it really were just some arbitrarily complex distribution which we don't know: the true model. From a frequentist perspective, the idea of the true model is very important. Many arguments arise by considering it and trying to understand how to reduce the distance between your estimators and the unknown truth. Bayesian foundations avoid this entirely, refusing to posit an unknown and instead talking about refinements of what we know.

obastani 10y ago
It's not just philosophical -- there are examples of practical situations where there is a true model (especially in more traditional applications of statistics): - Polling for presidential election: The "true model" is the voting preferences of all 300 million Americans. A (uniformly) random sample of N Americans can be used to estimate the true model roughly with standard error 1/sqrt(N). - Particle physics: The "true model" is the decay probabilities of various particles as computed by quantum mechanics. Different models (and parameters) yield different decay probabilities, and experiments can be used to choose between models and/or estimate model parameters. Of course, oftentimes the true model is philosophical as you describe.
- tel 10y ago
  That's reasonably fair, but it's worth noting that even in these situations the true circumstance might be a little more nuanced than the idea of a "true model" suggests. - In polling, it's a bit of an ideal world idea to think that the true voting preferences of all Americans are (1) fixed, (2) consistently measurable, or (3) even relevant given a lot of people won't vote. These are all sources of unknowns and variances which make me pretty unhappy with the idea of the idea of a true model even here. - In physics there's definitely a notion of a true model taken under the assumption that one set of equations and models is "correct" (which is its own philosophical problem!) but even then experiments don't measure this true model perfectly. They're also based on correction for variance in measurement and tooling which is assumed to be eventually ignorable. In any case, this has a much more definite notion of a true model, but I still find it difficult to swallow all together.