4 ms·

Well, to be fair, if I give you the data set: a=1 b=2 d=4 e=5 f=6 and you predict that c=3, you are not committing fraud. The gold standard would be testing

Well, to be fair, if I give you the data set: a=1 b=2 d=4 e=5 f=6 and you predict that c=3, you are not committing fraud. The gold standard would be testing on some more couples and finding the model he made is predictive. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. I'm not a statistician - does anyone know if there is a measure of how over-fitted a model is based on degrees of freedom and size of sample?

drunkpotato 17y ago
To properly use your analogy, if I first saw your data, then "predicted" that "a = 1.1, b = 2.1, d = 3.99, e = 5.01, f = 3.9999" and claimed that I had "predicted" your data with 90% (or whatever) accuracy, that is substantially misrepresenting what I have done. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. That is a problem. That should not be enough to convince you. It is enough to make it an avenue worth exploring.
- Robin_Message 17y ago
  I don't think so. I think my analogy was that given the data, if I can create a model that accurately predicts the data given, and is plausibly simple, i.e. for X=Y, Y=ordinal(X), then I can have some faith in the model. To put it another way, if Gottman said: Add up these 5000 variables multiplied by these factors to get the score then I wouldn't believe it. But I think he said something much closer to (on p. 13 of the 98 paper): We tried a couple of combinations of variables, that we had hypotheses for. One combination gives a very accurate predictor (Table 1, Husband high negativity). Two others are also accurate (Table 1, low negativity by husband or wife.) I think where the paper went wrong, and what the Slate article is criticising is the claim made in the second column of p. 16, where they do seem to put all the variables into a model and pull the answer from nowhere. So, I'll give you that the 82% prediction is not justifiable. But negativity as a predictor for divorce seems well supported. --- If I drop 30 monkeys out of different height trees and I measure 100 things about each monkey, each second, as it falls, and then I give you a model that purports to tell you a monkey's speed from those 100 measurements, then you won't be surprised to received an over-fitted piece of junk. If instead I give you speed=constant * t^2, would I need to drop another 30 monkeys?
  - Retric 17y ago
    This is one of those things where the size of the result set let's you easily over fit data. Take 50 people born in 1970 and you could make a fairly simple model based on their birth day to predict if they were living or dead because you only need 50 bits of information and 10 of them can be wrong. There is even a strong bias where most people born in that year are still alive. Edit: As a simple rule of thumb compare the complexity of the formula with the number of accurate bits in your output compared to the most simple model possible. If the formula is anywhere near as complex as your results it’s probably over fitting the data.

drunkpotato 17y ago

To properly use your analogy, if I first saw your data, then "predicted" that "a = 1.1, b = 2.1, d = 3.99, e = 5.01, f = 3.9999" and claimed that I had "predicted" your data with 90% (or whatever) accuracy, that is substantially misrepresenting what I have done. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. That is a problem. That should not be enough to convince you. It is enough to make it an avenue worth exploring.

Robin_Message 17y ago
I don't think so. I think my analogy was that given the data, if I can create a model that accurately predicts the data given, and is plausibly simple, i.e. for X=Y, Y=ordinal(X), then I can have some faith in the model. To put it another way, if Gottman said: Add up these 5000 variables multiplied by these factors to get the score then I wouldn't believe it. But I think he said something much closer to (on p. 13 of the 98 paper): We tried a couple of combinations of variables, that we had hypotheses for. One combination gives a very accurate predictor (Table 1, Husband high negativity). Two others are also accurate (Table 1, low negativity by husband or wife.) I think where the paper went wrong, and what the Slate article is criticising is the claim made in the second column of p. 16, where they do seem to put all the variables into a model and pull the answer from nowhere. So, I'll give you that the 82% prediction is not justifiable. But negativity as a predictor for divorce seems well supported. --- If I drop 30 monkeys out of different height trees and I measure 100 things about each monkey, each second, as it falls, and then I give you a model that purports to tell you a monkey's speed from those 100 measurements, then you won't be surprised to received an over-fitted piece of junk. If instead I give you speed=constant * t^2, would I need to drop another 30 monkeys?
- Retric 17y ago
  This is one of those things where the size of the result set let's you easily over fit data. Take 50 people born in 1970 and you could make a fairly simple model based on their birth day to predict if they were living or dead because you only need 50 bits of information and 10 of them can be wrong. There is even a strong bias where most people born in that year are still alive. Edit: As a simple rule of thumb compare the complexity of the formula with the number of accurate bits in your output compared to the most simple model possible. If the formula is anywhere near as complex as your results it’s probably over fitting the data.