5 ms·

Individual researchers have huge incentives for fraud, but large universities are the ones who most need to look inward here. HBS could become a beacon for stro

Individual researchers have huge incentives for fraud, but large universities are the ones who most need to look inward here. HBS could become a beacon for stronger research practices in this area - requiring study preregistration and data collection methods that carefully trace provenance and changes. If HBS has fewer but better quirky marketing psychology studies, their reputation will do well. However for an individual researcher, the benefits to using extremely rigorous practices are not as clear, while the costs are high. A random grad student is not going to be able to establish a norm of data traceability. I worked in this area (at Harvard as well) a bit as a grad student and can absolutely understand the temptation for the lighter version of this. If you can drop one outlier group, you get a cool story and a job at a top 10 b-school. Or keep it, get a muddled result, and try again for a better paper next year. I ended up just leaving the field entirely as the whole "our system massively rewards dodgy practices" vibe really bummed me out.

IggleSniggle 3y ago
For similar reasons, I left academia. I joined academia for the pursuit of truth, and forth the glory of Knowing. But it turned out that academia doesn’t really do this anymore, it just sells itself as doing these things. Academia is entirely a “reputation” system, as it turns out, from paper publishing to student-evaluations to “public rankings.” As it turns out, the capitalist marketplace must also contend with these issues, but imho (and somewhat counterintuitively), it does so somewhat more honestly.
- bwestergard 3y ago
  I'd like to hear more about how you've found more honesty in the private sector. "academia doesn’t really do this anymore" I've heard this from more than one current or former academic. Some argue it's due to market logic entering academia. It sounds like you take a different view?
  - 93po 3y ago
    I have no idea what I'm talking about but I would speculate that looking at how.much.freaking.money these schools have provides us a window into what these schools actually focus on. The focus seems to be making and having a ton of money. This compounded by the trends we see in fewer tenure positions, worse pay for associate profs, a glut of admin positions and admin pay, etc. So yeah, I'd say the observation that market logic is entering academia has some surface level reasons to support it.
    - otikik 3y ago
      Ah, you meant it the same way that a thief is more honest that a corrupt politician. At least the first one is honest about what it does.
      - IggleSniggle 3y ago
        The person you are responding to is not me, but they captured the essence of my intent, more or less.
  - deleted 3y ago
    [deleted]
  - eru 3y ago
    > I'd like to hear more about how you've found more honesty in the private sector. vs > Some argue it's due to market logic entering academia. People's behaviour is shaped by incentives (amongst other things). People in academia are rewarded for pumping out quirky studies that find interesting things, regardless of whether they reproduce. So that's what you get. People in business mostly just want to make money selling some gadget (or providing some service). So they need methods that help reliably produce gadgets that are more cheaper or perform better. Because businesses want reliability, you get your reproducibility as a side effect. (Of course, businesses also produce the occasional study where they use the study itself as marketing material. I expect those studies to be just as unreliable and full of wishful thinking as anything academia pumps out: the incentives aren't any better.)
  - red-iron-pine 3y ago
    > I'd like to hear more about how you've found more honesty in the private sector. no intellectual wrangling, just make it rain. and in my experience, the type-A super-go-hards are respectful enough to stab you in the front instead of in the back.
- greazy 3y ago
  Wait a second, the blog which is bringing to light these issues is by academics who ARE seeking the truth. As someone in academia (very different field than Behavioral psych), I'm very cautious of people who say overgeneralised hyperbole such as: > I joined academia for the pursuit of truth, and forth the glory of Knowing. Because doing science is not clear concise straight forward exercise people envision. (Science has major systemic issues but that's besides my point)
  - operatingthetan 3y ago
    > I joined academia for the pursuit of truth, and forth the glory of Knowing. I assume they said this from a position of naïveté and then after joining academia realized it was incorrect. I'm not sure we should use that to suggest a defect in character.
    - greazy 3y ago
      > I'm not sure we should use that to suggest a defect in character. Apologies for the above poster if this was taken as an attack on their character, it wasn't.
  - deleted 3y ago
    [deleted]
  - IggleSniggle 3y ago
    To be fair, my field was not in the sciences. Regardless, I left the tenure track due to disillusionment. I don’t presume that others had the same experience, and it is my earnest hope that most people do not. I was heavily encouraged to give high grades for high reviews and to publish high-volume, quality be damned. I hope that my leaving had some effect on the decision makers at my particular institution. But my impression was that the issue was systemic, and I became very bitter. I thought it better to leave than to remain and further institutionalize that bitterness.
    - greazy 3y ago
      I think your experience is on par with a lot of scientists and academics, the issues you describe are present across nearly every field of science and academia. My comment was not meant to reduce your experience. However, my main concern is that people enter academia due to some idealistic, romantic and preconceived ideas. Science (and academia) isn't above the fray of humanities ugly side. It's done by people for people, its messy inefficient monster. I wish it weren't but that means taking the humanity out of it. These are my crazy ideas. I am probably wrong.
- eru 3y ago
  > But it turned out that academia doesn’t really do this anymore, it just sells itself as doing these things. What do you mean by 'anymore'. Was there some golden age of bygone glory?
  - TX81Z 3y ago
    To some degree yes. As the world got more globalized and the number of prestigious universities basically stayed the same while the number of people getting PhDs skyrocketed, the competition for jobs at even third-tier schools became outrageous. You either play the game and survive - often at the cost of your integrity. Or you walk. I walked. No regrets.
  - hgsgm 3y ago
    Academics used to be rich people (entrepreneurs or aristocrats) who funded their own work. Then (late 1800s?) it changed to a profession, and the focus became extracting funding from ignorant patrons.
  - kneebonian 3y ago
    I'd argue there was to some extent. I mean psych has always had issues, but it seems from the late 1800's to the mid 1970's would characterize the golden age. During this time science that produced results and was verifiable and reproducible experienced a huge boom. In part because during this time period almost every nation on earth got involved at some point in a life or death struggle which turns to do a pretty good job at cutting through science that doesn't hold water when it's the difference between your country winning and your countrymen being slaughtered. As proof of this most of the open question in fundamental physics are the same ones that were open at the time of Einstein's death.
    - eru 3y ago
      My null hypothesis for that golden age from late 1800s til mid-1970s is that they just found all the low hanging fruit? See https://slatestarcodex.com/2016/11/17/the-alzheimer-photo/ https://slatestarcodex.com/2016/11/17/the-alzheimer-photo/ for background. Compare also how even second rate people could do first rate work in the last fifteen years by applying deep learning to new domains. You mention that war has a certain tendency to cut through bullshit. Ideally, market competition also does some of that. See https://gwern.net/backstop https://gwern.net/backstop for a more general exploration: > I suggest interpreting phenomenon as multi-level nested optimization paradigm: many systems can be usefully described as having two (or more) levels where a slow sample-inefficient but ground-truth ‘outer’ loss such as death, bankruptcy, or reproductive fitness, trains & constrains a fast sample-efficient but possibly misguided ‘inner’ loss which is used by learned mechanisms such as neural networks or linear programming group selection perspective.
- TX81Z 3y ago
  Ha, did the same and literally had an hour long talk with an emerita faculty member today about how totally skewed the incentive structure is. People only care about lines on the CV - and are willing to do anything to add another under the “peer reviewed” column. Including, lie, cheat, and steal (ideas).
- Xen9 3y ago
  The society makes individuals with high intellectual potential to attach their ambitions to the itself and this process begins usually even before the individuals can see the flaws and untrustworthiness of the society. Those who eventually break their ties to the system are outcast by everyone else.
- DrScientist 3y ago
  I think this is a little unfair - academia is stuffed full of people seeking truth and doing good work. However as a PhD student I decided that science academia wasn't for me - not because of out and out fraud - but because I could see success depended on marketing, networking and self-promotion - simply doing excellent quality work was required, but not sufficient. Some of this is fair - part of an academics job is not just to discover new stuff, but to educate about it's discovery. Anyway I decided I didn't have a loud enough trumpet and sharp enough elbows. I'd agree that you find less politics in industry.
lumost 3y ago
I suspect this is also why large industrial ML labs have mixed productivity. My observation has been that people decide that a problem is important because someone high up thought it was important. The researchers using the methodology that shows the largest offline impact are rewarded - not those with the most rigorous finding. This creates a strong incentive to fudge the results and never ship anything. I knew of one scientist who figured out you could just fake the entire thing and skip the work part. He coasted for 5 years doing this.
- golergka 3y ago
  What happened after 5 years? Did he get caught? Or just founded his own unicorn ai startup
  - lumost 3y ago
    I think around year 4 a manager made the mistake of believing the research and pushing hard on it. This in turn became a tracked goal with a multi-year timeline. The individual left before things could fall on them too heavily.
    - mmmmpancakes 3y ago
      how do you work 4 years on a project before having a tracked goal. absurd. sounds like management's fault.
      - dchftcs 3y ago
        The goals before the 4 years were probably based on results claimed, not making a product. Research management is difficult and efficiently running a lab requires some degree of trust that researchers don't lie. I think the bigger part of management's fault here is that they should have spent some time on verifying the research before dedicating more resources.
        lumost 3y ago
pyuser583 3y ago
I’ve been involved in discussions over what numbers to give to the Board of Trustees. CFOs go to prison over this. It’s funny how greedy, for-profit companies are strict about honest accounting, while non-profit universities are fine with dodgy research.
- zetazzed 3y ago
  While I strongly think HBS should do much more to prevent fraud, I would also note that they have suspended Gino and seem likely to fire her. So it’s not like they are completely ok with this…
- namdnay 3y ago
  Well you said it yourself, CFOs go to prison for it. I’m sure if as a society we treated academic fraud like financial fraud the situation would be different. I don’t think it’s linked to companies being for profit or not
civilized 3y ago
> If HBS has fewer but better quirky marketing psychology studies, their reputation will do well. Provided that there are valuable things to discover in the field of quirky marketing psychology. But if all they want to do is provide a regular supply of ammo to both sides of internal corporate conflicts on marketing strategy, and promote the consulting brand of their profs, their current approach is optimal.

IggleSniggle 3y ago

For similar reasons, I left academia. I joined academia for the pursuit of truth, and forth the glory of Knowing. But it turned out that academia doesn’t really do this anymore, it just sells itself as doing these things. Academia is entirely a “reputation” system, as it turns out, from paper publishing to student-evaluations to “public rankings.” As it turns out, the capitalist marketplace must also contend with these issues, but imho (and somewhat counterintuitively), it does so somewhat more honestly.

bwestergard 3y ago
I'd like to hear more about how you've found more honesty in the private sector. "academia doesn’t really do this anymore" I've heard this from more than one current or former academic. Some argue it's due to market logic entering academia. It sounds like you take a different view?
- 93po 3y ago
  I have no idea what I'm talking about but I would speculate that looking at how.much.freaking.money these schools have provides us a window into what these schools actually focus on. The focus seems to be making and having a ton of money. This compounded by the trends we see in fewer tenure positions, worse pay for associate profs, a glut of admin positions and admin pay, etc. So yeah, I'd say the observation that market logic is entering academia has some surface level reasons to support it.
  - otikik 3y ago
    Ah, you meant it the same way that a thief is more honest that a corrupt politician. At least the first one is honest about what it does.
    - IggleSniggle 3y ago
      The person you are responding to is not me, but they captured the essence of my intent, more or less.
- deleted 3y ago
  [deleted]
- eru 3y ago
  > I'd like to hear more about how you've found more honesty in the private sector. vs > Some argue it's due to market logic entering academia. People's behaviour is shaped by incentives (amongst other things). People in academia are rewarded for pumping out quirky studies that find interesting things, regardless of whether they reproduce. So that's what you get. People in business mostly just want to make money selling some gadget (or providing some service). So they need methods that help reliably produce gadgets that are more cheaper or perform better. Because businesses want reliability, you get your reproducibility as a side effect. (Of course, businesses also produce the occasional study where they use the study itself as marketing material. I expect those studies to be just as unreliable and full of wishful thinking as anything academia pumps out: the incentives aren't any better.)
- red-iron-pine 3y ago
  > I'd like to hear more about how you've found more honesty in the private sector. no intellectual wrangling, just make it rain. and in my experience, the type-A super-go-hards are respectful enough to stab you in the front instead of in the back.
greazy 3y ago
Wait a second, the blog which is bringing to light these issues is by academics who ARE seeking the truth. As someone in academia (very different field than Behavioral psych), I'm very cautious of people who say overgeneralised hyperbole such as: > I joined academia for the pursuit of truth, and forth the glory of Knowing. Because doing science is not clear concise straight forward exercise people envision. (Science has major systemic issues but that's besides my point)
- operatingthetan 3y ago
  > I joined academia for the pursuit of truth, and forth the glory of Knowing. I assume they said this from a position of naïveté and then after joining academia realized it was incorrect. I'm not sure we should use that to suggest a defect in character.
  - greazy 3y ago
    > I'm not sure we should use that to suggest a defect in character. Apologies for the above poster if this was taken as an attack on their character, it wasn't.
- deleted 3y ago
  [deleted]
- IggleSniggle 3y ago
  To be fair, my field was not in the sciences. Regardless, I left the tenure track due to disillusionment. I don’t presume that others had the same experience, and it is my earnest hope that most people do not. I was heavily encouraged to give high grades for high reviews and to publish high-volume, quality be damned. I hope that my leaving had some effect on the decision makers at my particular institution. But my impression was that the issue was systemic, and I became very bitter. I thought it better to leave than to remain and further institutionalize that bitterness.
  - greazy 3y ago
    I think your experience is on par with a lot of scientists and academics, the issues you describe are present across nearly every field of science and academia. My comment was not meant to reduce your experience. However, my main concern is that people enter academia due to some idealistic, romantic and preconceived ideas. Science (and academia) isn't above the fray of humanities ugly side. It's done by people for people, its messy inefficient monster. I wish it weren't but that means taking the humanity out of it. These are my crazy ideas. I am probably wrong.
eru 3y ago
> But it turned out that academia doesn’t really do this anymore, it just sells itself as doing these things. What do you mean by 'anymore'. Was there some golden age of bygone glory?
- TX81Z 3y ago
  To some degree yes. As the world got more globalized and the number of prestigious universities basically stayed the same while the number of people getting PhDs skyrocketed, the competition for jobs at even third-tier schools became outrageous. You either play the game and survive - often at the cost of your integrity. Or you walk. I walked. No regrets.
- hgsgm 3y ago
  Academics used to be rich people (entrepreneurs or aristocrats) who funded their own work. Then (late 1800s?) it changed to a profession, and the focus became extracting funding from ignorant patrons.
- kneebonian 3y ago
  I'd argue there was to some extent. I mean psych has always had issues, but it seems from the late 1800's to the mid 1970's would characterize the golden age. During this time science that produced results and was verifiable and reproducible experienced a huge boom. In part because during this time period almost every nation on earth got involved at some point in a life or death struggle which turns to do a pretty good job at cutting through science that doesn't hold water when it's the difference between your country winning and your countrymen being slaughtered. As proof of this most of the open question in fundamental physics are the same ones that were open at the time of Einstein's death.
  - eru 3y ago
    My null hypothesis for that golden age from late 1800s til mid-1970s is that they just found all the low hanging fruit? See https://slatestarcodex.com/2016/11/17/the-alzheimer-photo/ https://slatestarcodex.com/2016/11/17/the-alzheimer-photo/ for background. Compare also how even second rate people could do first rate work in the last fifteen years by applying deep learning to new domains. You mention that war has a certain tendency to cut through bullshit. Ideally, market competition also does some of that. See https://gwern.net/backstop https://gwern.net/backstop for a more general exploration: > I suggest interpreting phenomenon as multi-level nested optimization paradigm: many systems can be usefully described as having two (or more) levels where a slow sample-inefficient but ground-truth ‘outer’ loss such as death, bankruptcy, or reproductive fitness, trains & constrains a fast sample-efficient but possibly misguided ‘inner’ loss which is used by learned mechanisms such as neural networks or linear programming group selection perspective.
TX81Z 3y ago
Ha, did the same and literally had an hour long talk with an emerita faculty member today about how totally skewed the incentive structure is. People only care about lines on the CV - and are willing to do anything to add another under the “peer reviewed” column. Including, lie, cheat, and steal (ideas).
Xen9 3y ago
The society makes individuals with high intellectual potential to attach their ambitions to the itself and this process begins usually even before the individuals can see the flaws and untrustworthiness of the society. Those who eventually break their ties to the system are outcast by everyone else.
DrScientist 3y ago
I think this is a little unfair - academia is stuffed full of people seeking truth and doing good work. However as a PhD student I decided that science academia wasn't for me - not because of out and out fraud - but because I could see success depended on marketing, networking and self-promotion - simply doing excellent quality work was required, but not sufficient. Some of this is fair - part of an academics job is not just to discover new stuff, but to educate about it's discovery. Anyway I decided I didn't have a loud enough trumpet and sharp enough elbows. I'd agree that you find less politics in industry.

lumost 3y ago

I suspect this is also why large industrial ML labs have mixed productivity. My observation has been that people decide that a problem is important because someone high up thought it was important. The researchers using the methodology that shows the largest offline impact are rewarded - not those with the most rigorous finding. This creates a strong incentive to fudge the results and never ship anything. I knew of one scientist who figured out you could just fake the entire thing and skip the work part. He coasted for 5 years doing this.

golergka 3y ago
What happened after 5 years? Did he get caught? Or just founded his own unicorn ai startup
- lumost 3y ago
  I think around year 4 a manager made the mistake of believing the research and pushing hard on it. This in turn became a tracked goal with a multi-year timeline. The individual left before things could fall on them too heavily.
  - mmmmpancakes 3y ago
    how do you work 4 years on a project before having a tracked goal. absurd. sounds like management's fault.
    - dchftcs 3y ago
      The goals before the 4 years were probably based on results claimed, not making a product. Research management is difficult and efficiently running a lab requires some degree of trust that researchers don't lie. I think the bigger part of management's fault here is that they should have spent some time on verifying the research before dedicating more resources.
      - lumost 3y ago

pyuser583 3y ago

I’ve been involved in discussions over what numbers to give to the Board of Trustees. CFOs go to prison over this. It’s funny how greedy, for-profit companies are strict about honest accounting, while non-profit universities are fine with dodgy research.

zetazzed 3y ago
While I strongly think HBS should do much more to prevent fraud, I would also note that they have suspended Gino and seem likely to fire her. So it’s not like they are completely ok with this…
namdnay 3y ago
Well you said it yourself, CFOs go to prison for it. I’m sure if as a society we treated academic fraud like financial fraud the situation would be different. I don’t think it’s linked to companies being for profit or not

civilized 3y ago

> If HBS has fewer but better quirky marketing psychology studies, their reputation will do well. Provided that there are valuable things to discover in the field of quirky marketing psychology. But if all they want to do is provide a regular supply of ammo to both sides of internal corporate conflicts on marketing strategy, and promote the consulting brand of their profs, their current approach is optimal.