7 ms·

He fitted a curve to the training data and called it a prediction. How is this not fraud?

by drunkpotato 17y ago

ig1 17y ago
Fraud implies he did it on purpose, it seems likely in this case that he didn't realize that his methods were invalid.
- michael_dorfman 17y ago
  I agree, it's not fraud, strictly speaking However, to create a "predictive model", and then not check it on fresh data, is so staggeringly inept as to be reckless. He should have known better, and the zillions of folks who parroted the results really should have read the study carefully.
  - Deestan 17y ago
    It is a bit disappointing. At my university, we were taught of this fallacy in the mandatory introduction to science course. I remember a beatiful analogy which went something like: As long as both your sample data and your test data are collected between January and May, you can make a model that accurately predicts the time the sun sets based on the length of your hair.
    - jerf 17y ago
      The evidence would abundantly suggest that merely being exposed to this idea in a course is not sufficient to guarantee that the resulting scientist will have any clue. I see two problems here: First, by the time you get to that level of schooling, you've developed a "resistance" (for lack of a better term) to schooling. That is, you've long since internalized the fact that there are two worlds, the one described in school and the real world, and the overlap between the two is tenuous at best. Merely being told about this stuff in the school world isn't enough, you somehow need it to be penetrated down to the "real" world. Computer scientists get a bit of an advantage here in that after being told about this, their homework assignment will generally be to implement the situations described and verify the results. Computer courses are rather good at bridging this gap; most disciplines lack the equivalent and a shocking amount of basic stuff is safely locked away in the "schooling" portion of the world for most people. This is a result of the second part of the problem, which is that the teachers themselves help create this environment. I went up the computer science grad school approach for this stuff, where not only do they show you the fundamental math, they make you actually implement it (like I said above). I got to watch my wife go up through the biology stats course, and they don't do that. At the time I didn't notice how bad it was, but the stats courses they do are terrible; they teach a variety of "leaf node" techniques (that is, the final results of the other math), but they never teach the actual prerequisites for using the techniques, and in the end it is nothing but voodoo. Students who consign this stuff to "in the school world but not real" are, arguably, being more rational than those who don't; things like confidence interval and p-values are only applicable in Gaussian domains, and I know they say that at some point but they don't ever cover anything else from what I can see. By never connecting it to the real world (very well) and indeed treating in the courses as a bit of voodoo ("wave this statistical chicken over the data and it will squawk the answer, which you will then write in your paper") they contribute to the division between school and real. So, I'd say two things: Those who say this stuff is taught in "any college stats course" are probably actually literally wrong in that it is possible to take a lot of stats without really covering this, and secondly, even if covered, it ends up categorized away unless great efforts are taken to "make it real" for the student, which, short of forcing implementation on the students (impractical in general) I'm not sure how to do. Most scientists seem to come out of PhD programs with a very thin understanding of this stuff.
- drunkpotato 17y ago
  He was a math major and should certainly know better and thoroughly know his methods. I am unwilling to give the benefit of the doubt in this case. Update: Also, he did it twice, and the article implies he has continued doing it. That's way beyond simple methodical error.
  - deleted 17y ago
    [deleted]
- jaiwithani 17y ago
  "A math major at MIT before he switched to psychology" How does someone go into a math major at MIT without a basic grasp of statistics? And to then spend years studying the field and other lab reports? How could he fail to pick this up?
yungchin 17y ago
I'd say it would only be fraud if he misreported his methods in his papers. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so.
- drunkpotato 17y ago
  I'd say it would only be fraud if he misreported his methods in his papers. Fair point, you are right. Many authors do, in fact, misreport in this fashion, but I have no evidence that John Gottman has done so. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. No, she doesn't just say that. This is not a difference of opinion, it's a matter of fact of what is written in the actual article. "I think" does not apply. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. That is not what I said. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so. That is a valid methodology, but not the one the article says he used. What was described in the article was pure curve-fitting to the training data, which is not predictive. You can only claim predictiveness by using the generated curve on a test set and seeing how well it does, which is exactly what the article says he did not do. You are arguing in circles! You should, in fact, expect to fit a curve to a training set with higher accuracy than its general predictive power! You can only test its predictive power on test data.
  - yungchin 17y ago
    > That is not what I said. Right, sorry, I should have been clearer that I was reacting to the article and not to you there. Or actually, I was reacting to what I thought the article said, because indeed that's not what it said! Thanks.
- araneae 17y ago
Robin_Message 17y ago
Well, to be fair, if I give you the data set: a=1 b=2 d=4 e=5 f=6 and you predict that c=3, you are not committing fraud. The gold standard would be testing on some more couples and finding the model he made is predictive. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. I'm not a statistician - does anyone know if there is a measure of how over-fitted a model is based on degrees of freedom and size of sample?
- drunkpotato 17y ago
  To properly use your analogy, if I first saw your data, then "predicted" that "a = 1.1, b = 2.1, d = 3.99, e = 5.01, f = 3.9999" and claimed that I had "predicted" your data with 90% (or whatever) accuracy, that is substantially misrepresenting what I have done. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. That is a problem. That should not be enough to convince you. It is enough to make it an avenue worth exploring.
  - Robin_Message 17y ago
    I don't think so. I think my analogy was that given the data, if I can create a model that accurately predicts the data given, and is plausibly simple, i.e. for X=Y, Y=ordinal(X), then I can have some faith in the model. To put it another way, if Gottman said: Add up these 5000 variables multiplied by these factors to get the score then I wouldn't believe it. But I think he said something much closer to (on p. 13 of the 98 paper): We tried a couple of combinations of variables, that we had hypotheses for. One combination gives a very accurate predictor (Table 1, Husband high negativity). Two others are also accurate (Table 1, low negativity by husband or wife.) I think where the paper went wrong, and what the Slate article is criticising is the claim made in the second column of p. 16, where they do seem to put all the variables into a model and pull the answer from nowhere. So, I'll give you that the 82% prediction is not justifiable. But negativity as a predictor for divorce seems well supported. --- If I drop 30 monkeys out of different height trees and I measure 100 things about each monkey, each second, as it falls, and then I give you a model that purports to tell you a monkey's speed from those 100 measurements, then you won't be surprised to received an over-fitted piece of junk. If instead I give you speed=constant * t^2, would I need to drop another 30 monkeys?
CWuestefeld 17y ago
At least as this article describes it, it's not even predictive on its face. He did not assess "fresh" couples, and come back a few years later to see how they fared. Rather, he interviewed people who were already either having trouble or they weren't. Although I haven't studied it for proof, I have to assume that the interactions of a honeymoon couple will be different from a couple in trouble, even if the honeymoon couple is destined to have problems. Thus, he's not able to interview the honeymoon couple and determine what their prospects are.

He fitted a curve to the training data and called it a prediction. How is this not fraud?

ig1 17y ago

Fraud implies he did it on purpose, it seems likely in this case that he didn't realize that his methods were invalid.

michael_dorfman 17y ago
I agree, it's not fraud, strictly speaking However, to create a "predictive model", and then not check it on fresh data, is so staggeringly inept as to be reckless. He should have known better, and the zillions of folks who parroted the results really should have read the study carefully.
- Deestan 17y ago
  It is a bit disappointing. At my university, we were taught of this fallacy in the mandatory introduction to science course. I remember a beatiful analogy which went something like: As long as both your sample data and your test data are collected between January and May, you can make a model that accurately predicts the time the sun sets based on the length of your hair.
  - jerf 17y ago
    The evidence would abundantly suggest that merely being exposed to this idea in a course is not sufficient to guarantee that the resulting scientist will have any clue. I see two problems here: First, by the time you get to that level of schooling, you've developed a "resistance" (for lack of a better term) to schooling. That is, you've long since internalized the fact that there are two worlds, the one described in school and the real world, and the overlap between the two is tenuous at best. Merely being told about this stuff in the school world isn't enough, you somehow need it to be penetrated down to the "real" world. Computer scientists get a bit of an advantage here in that after being told about this, their homework assignment will generally be to implement the situations described and verify the results. Computer courses are rather good at bridging this gap; most disciplines lack the equivalent and a shocking amount of basic stuff is safely locked away in the "schooling" portion of the world for most people. This is a result of the second part of the problem, which is that the teachers themselves help create this environment. I went up the computer science grad school approach for this stuff, where not only do they show you the fundamental math, they make you actually implement it (like I said above). I got to watch my wife go up through the biology stats course, and they don't do that. At the time I didn't notice how bad it was, but the stats courses they do are terrible; they teach a variety of "leaf node" techniques (that is, the final results of the other math), but they never teach the actual prerequisites for using the techniques, and in the end it is nothing but voodoo. Students who consign this stuff to "in the school world but not real" are, arguably, being more rational than those who don't; things like confidence interval and p-values are only applicable in Gaussian domains, and I know they say that at some point but they don't ever cover anything else from what I can see. By never connecting it to the real world (very well) and indeed treating in the courses as a bit of voodoo ("wave this statistical chicken over the data and it will squawk the answer, which you will then write in your paper") they contribute to the division between school and real. So, I'd say two things: Those who say this stuff is taught in "any college stats course" are probably actually literally wrong in that it is possible to take a lot of stats without really covering this, and secondly, even if covered, it ends up categorized away unless great efforts are taken to "make it real" for the student, which, short of forcing implementation on the students (impractical in general) I'm not sure how to do. Most scientists seem to come out of PhD programs with a very thin understanding of this stuff.
drunkpotato 17y ago
He was a math major and should certainly know better and thoroughly know his methods. I am unwilling to give the benefit of the doubt in this case. Update: Also, he did it twice, and the article implies he has continued doing it. That's way beyond simple methodical error.
- deleted 17y ago
  [deleted]
jaiwithani 17y ago
"A math major at MIT before he switched to psychology" How does someone go into a math major at MIT without a basic grasp of statistics? And to then spend years studying the field and other lab reports? How could he fail to pick this up?

yungchin 17y ago

I'd say it would only be fraud if he misreported his methods in his papers. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so.

drunkpotato 17y ago
I'd say it would only be fraud if he misreported his methods in his papers. Fair point, you are right. Many authors do, in fact, misreport in this fashion, but I have no evidence that John Gottman has done so. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. No, she doesn't just say that. This is not a difference of opinion, it's a matter of fact of what is written in the actual article. "I think" does not apply. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. That is not what I said. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so. That is a valid methodology, but not the one the article says he used. What was described in the article was pure curve-fitting to the training data, which is not predictive. You can only claim predictiveness by using the generated curve on a test set and seeing how well it does, which is exactly what the article says he did not do. You are arguing in circles! You should, in fact, expect to fit a curve to a training set with higher accuracy than its general predictive power! You can only test its predictive power on test data.
- yungchin 17y ago
  > That is not what I said. Right, sorry, I should have been clearer that I was reacting to the article and not to you there. Or actually, I was reacting to what I thought the article said, because indeed that's not what it said! Thanks.
araneae 17y ago

Robin_Message 17y ago

Well, to be fair, if I give you the data set: a=1 b=2 d=4 e=5 f=6 and you predict that c=3, you are not committing fraud. The gold standard would be testing on some more couples and finding the model he made is predictive. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. I'm not a statistician - does anyone know if there is a measure of how over-fitted a model is based on degrees of freedom and size of sample?

drunkpotato 17y ago
To properly use your analogy, if I first saw your data, then "predicted" that "a = 1.1, b = 2.1, d = 3.99, e = 5.01, f = 3.9999" and claimed that I had "predicted" your data with 90% (or whatever) accuracy, that is substantially misrepresenting what I have done. However, I would be convinced by a strong correlation (i.e. the model is not overfitted), coupled with a plausible model. That is a problem. That should not be enough to convince you. It is enough to make it an avenue worth exploring.
- Robin_Message 17y ago
  I don't think so. I think my analogy was that given the data, if I can create a model that accurately predicts the data given, and is plausibly simple, i.e. for X=Y, Y=ordinal(X), then I can have some faith in the model. To put it another way, if Gottman said: Add up these 5000 variables multiplied by these factors to get the score then I wouldn't believe it. But I think he said something much closer to (on p. 13 of the 98 paper): We tried a couple of combinations of variables, that we had hypotheses for. One combination gives a very accurate predictor (Table 1, Husband high negativity). Two others are also accurate (Table 1, low negativity by husband or wife.) I think where the paper went wrong, and what the Slate article is criticising is the claim made in the second column of p. 16, where they do seem to put all the variables into a model and pull the answer from nowhere. So, I'll give you that the 82% prediction is not justifiable. But negativity as a predictor for divorce seems well supported. --- If I drop 30 monkeys out of different height trees and I measure 100 things about each monkey, each second, as it falls, and then I give you a model that purports to tell you a monkey's speed from those 100 measurements, then you won't be surprised to received an over-fitted piece of junk. If instead I give you speed=constant * t^2, would I need to drop another 30 monkeys?

CWuestefeld 17y ago

At least as this article describes it, it's not even predictive on its face. He did not assess "fresh" couples, and come back a few years later to see how they fared. Rather, he interviewed people who were already either having trouble or they weren't. Although I haven't studied it for proof, I have to assume that the interactions of a honeymoon couple will be different from a couple in trouble, even if the honeymoon couple is destined to have problems. Thus, he's not able to interview the honeymoon couple and determine what their prospects are.