3 ms·

I'd say it would only be fraud if he misreported his methods in his papers. I think the Slate author just says that she assumed his methods to have been differe

I'd say it would only be fraud if he misreported his methods in his papers. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so.

drunkpotato 17y ago
I'd say it would only be fraud if he misreported his methods in his papers. Fair point, you are right. Many authors do, in fact, misreport in this fashion, but I have no evidence that John Gottman has done so. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. No, she doesn't just say that. This is not a difference of opinion, it's a matter of fact of what is written in the actual article. "I think" does not apply. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. That is not what I said. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so. That is a valid methodology, but not the one the article says he used. What was described in the article was pure curve-fitting to the training data, which is not predictive. You can only claim predictiveness by using the generated curve on a test set and seeing how well it does, which is exactly what the article says he did not do. You are arguing in circles! You should, in fact, expect to fit a curve to a training set with higher accuracy than its general predictive power! You can only test its predictive power on test data.
- yungchin 17y ago
  > That is not what I said. Right, sorry, I should have been clearer that I was reacting to the article and not to you there. Or actually, I was reacting to what I thought the article said, because indeed that's not what it said! Thanks.
araneae 17y ago
Not testing the model on a validation set doesn't mean it doesn't have predictive power... but if you don't test it, there's no way you can say it does have predictive power. And saying it predicts the data 87% of the time when you're using the training set is in fact fraudulent. I can't believe that no one called him on this before, or that he got away with this! Anyone who has take a college statistics course knows that you have to have separate sets for training and for validation. There are even a ton of statistical methods to deal with this problem with a small data set, i.e. bootstrapping your training set. There's no excuse. In fact, I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me?
- eru 17y ago
  > In fact, I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me? It's interesting that we still long for citations of some authority figure --- even when we all agree. (And I, too, would like to see what Feynman said.)
  - araneae 17y ago
    It's more of a longing to figure out what the heck I'm remembering. This happens to me all the time with people, too... someone will look really familiar but I can't place them at all, and it drives me crazy.
- tokenadult 17y ago
  I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me? Perhaps you are referring to this passage of "Cargo Cult Science": "There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition." http://calteches.library.caltech.edu/51/02/CargoCult.pdf http://calteches.library.caltech.edu/51/02/CargoCult.pdf (Caltech lecture series is primary source) http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm (There are several online reprints of this lecture text in HTML, more searchable than .PDF format.) See http://en.wikipedia.org/wiki/Cargo_cult_science http://en.wikipedia.org/wiki/Cargo_cult_science for background and both of those citations.
breck 17y ago
> I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. I did too, because I used to think Gladwell was a reliable source. Oops!

drunkpotato 17y ago

I'd say it would only be fraud if he misreported his methods in his papers. Fair point, you are right. Many authors do, in fact, misreport in this fashion, but I have no evidence that John Gottman has done so. I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. No, she doesn't just say that. This is not a difference of opinion, it's a matter of fact of what is written in the actual article. "I think" does not apply. Also, it seems to me that if you make a model that fits the data, that doesn't at all mean that it doesn't have predictive power. That is not what I said. You could fit it with one data-set, and test it on another one. It just means that you now have a purely empirical model, with no built-in assumptions on why it is so. That is a valid methodology, but not the one the article says he used. What was described in the article was pure curve-fitting to the training data, which is not predictive. You can only claim predictiveness by using the generated curve on a test set and seeing how well it does, which is exactly what the article says he did not do. You are arguing in circles! You should, in fact, expect to fit a curve to a training set with higher accuracy than its general predictive power! You can only test its predictive power on test data.

yungchin 17y ago
> That is not what I said. Right, sorry, I should have been clearer that I was reacting to the article and not to you there. Or actually, I was reacting to what I thought the article said, because indeed that's not what it said! Thanks.

araneae 17y ago

Not testing the model on a validation set doesn't mean it doesn't have predictive power... but if you don't test it, there's no way you can say it does have predictive power. And saying it predicts the data 87% of the time when you're using the training set is in fact fraudulent. I can't believe that no one called him on this before, or that he got away with this! Anyone who has take a college statistics course knows that you have to have separate sets for training and for validation. There are even a ton of statistical methods to deal with this problem with a small data set, i.e. bootstrapping your training set. There's no excuse. In fact, I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me?

eru 17y ago
> In fact, I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me? It's interesting that we still long for citations of some authority figure --- even when we all agree. (And I, too, would like to see what Feynman said.)
- araneae 17y ago
  It's more of a longing to figure out what the heck I'm remembering. This happens to me all the time with people, too... someone will look really familiar but I can't place them at all, and it drives me crazy.
tokenadult 17y ago
I remember reading some Feynman lecture in which he specifically mentions how bad this is... anyone got a citation for me? Perhaps you are referring to this passage of "Cargo Cult Science": "There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition." http://calteches.library.caltech.edu/51/02/CargoCult.pdf http://calteches.library.caltech.edu/51/02/CargoCult.pdf (Caltech lecture series is primary source) http://www.lhup.edu/~DSIMANEK/cargocul.htm http://www.lhup.edu/~DSIMANEK/cargocul.htm (There are several online reprints of this lecture text in HTML, more searchable than .PDF format.) See http://en.wikipedia.org/wiki/Cargo_cult_science http://en.wikipedia.org/wiki/Cargo_cult_science for background and both of those citations.

breck 17y ago

> I think the Slate author just says that she assumed his methods to have been different than they were, after reading Gladwell's story about them. I did too, because I used to think Gladwell was a reliable source. Oops!