5 ms·

I had such high regard for academia while I was in school. Glad I didn’t end up going down that path, it seems like every day recently there have been stories l

I had such high regard for academia while I was in school. Glad I didn’t end up going down that path, it seems like every day recently there have been stories like this posted to Hacker News. Behavioral economics especially seems hard hit.

pocketsand 3y ago
Seems like behavioral economics really just receives splash damage from what is the massive fraud at the heart of experimental social psychology. At this point I basically assume any catchy finding from the field is at best HARKed or p-hacked and at worst completely fraudulent.
- chongli 3y ago
  Why aren't journals requiring pre-registration? It's an absolute catastrophe and no one seems to care!
  - pocketsand 3y ago
    Pre-registration protects against HARKing and p-hacking, but it unfortunately doesn't protect at all against outright fraud, which we appear to see from the likes of Ariely and Gino.
    - chongli 3y ago
      Right. Outright fraud needs to be dealt with much more severely. Total disgrace, loss of tenure, and potential prison time.
- FreshStart 3y ago
  The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You can even do ethical virtual experiments with the data of 3 billion people. Just specify the situation, the person and what's to happen.. Voila. All things happen regularly at some point..
  - esafak 3y ago
    > The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You really need to clarify this.
    - cool_dude85 3y ago
      I think it's pretty clear what the post is saying. It doesn't need to be clarified, it needs to be supported, though the nature of the claim is such that it would be tough to provide direct evidence.
    - meepmorp 3y ago
      I’m going with: > The government is trying to control your mind through the internet
    - scrps 3y ago
      Same techniques (largely derived from psychology, statistics, sociology, etc) used in social media marketing, intent tracking, etc has uses in national security. Same thing that might let meta know you might be interested in batman pajamas can be used to sort out if you are being radicalized to action by some group. Interesting example is fractalrabbit. Group of people who both have the theoretical training and practical experience is probably pretty small so private/public crossover isn't unreasonable.
passwordoops 3y ago
Me too, then following my Ph.D at a relatively small school went off to a post-doc at a major research university where the grants are millions or you're worthless... That environment essentially forces you to commit fraud so I got out pretty quick
cbsmith 3y ago
Consider how many studies are being produced each day. That there is a story every day like this suggests that academia is making an effort to identify bad actors, but the frequency of discovery is actually quite low.
- tnecniv 3y ago
  Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. I would think instances of fabrication are quite low — it’s high-risk, medium reward. Even if it goes undiscovered, academia is a marketing game, and you can’t control what other people feel is impactful or not. It’s very easy to do bad science without falsifying data, which is probably the deadliest sin in academia.
  - cbsmith 3y ago
    > Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. Yes, which is why I said the frequency of discovery is low. Seeing a story in the paper about fraud every day doesn't really tell us anything about the rate of fraudulence, which I was hoping would become transparent with my point.
lolinder 3y ago
To be fair, industry has a similar problem: every day there's a new company that gets in trouble for abusing customer data, losing customer data, abusing customer trust, making it hard to cancel a subscription, etc, etc. People in general tend to be dishonest if they think they can get away with it. I'm still glad I didn't go into academia for many other reasons, but I don't think I dodged an ethics bullet.
- notahacker 3y ago
  yeah, this Reputation probably matters more for academia (people go into it because they think the research is important, not for the money, and they think they're standing on the shoulder of giants and not amiable bullshitters taking the credit for findings actually created by their overworked research assistants kludging the data), but if you think academia is full of lies and half truths relative to other professions, you haven't encountered many startups!
usednet 3y ago
Could somebody provide me some links to other recent scandals in academia/research? Very interested.
- pocketsand 3y ago
  Check out the datacolada web site. https://datacolada.org/ https://datacolada.org/
- tnecniv 3y ago
  I forget the guys name (he’s German iirc), but a very impactful one was at Bell Labs in the 00’s. STEM PhD programs now all have a research ethics component because of him. While you can’t teach someone to be ethical, it at least hammers home how many people you are defrauding by falsifying data, which includes everyone from your funding organizations to the lowly PhD students whose advisors are making them iterate on your study and don’t understand why, after three years, it still isn’t working.
- dmvdoug 3y ago
  Retraction Watch, maybe? https://retractionwatch.com/ https://retractionwatch.com/
tnecniv 3y ago
While fields are small, academia is a big place. Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line, and from an emotional perspective, their academic work is tied to their self-image. It’s also arguably a lot harder to do in the hard sciences but there have still been examples. In the hard sciences, I feel like the solution is some method to increase how much people at different labs talk to one another (in addition to including all raw data and processing code from experiments). If an impactful article comes out, labs will try to build on it. Not every lab will have the equipment but those in the sub area might. If you’re trying to iterate on a study, and you can’t reproduce the base result before adding your modifications, you might just think you’re dumb or doing something dumb. There’s likely nobody outside maybe your research group who can help you. However, if it was easier to talk to other researchers in the field (like a web forum), you might find out that nobody has been able to get the original experiment working! In the social sciences, I’m not sure what you do besides running automated checks on submitted data. Humans are bad at generating things that look random (for example, we use more 3s and 7s than nature), and you can probably flag warnings. Either that or include paper trails like a lab notebook in chemistry — which is considered a legal document and thus must be kept in pen. That way you can establish a chain of custody for the data from collection to publication.
- pevey 3y ago
  > Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line In my experience, this is absolutely not true. I mostly only have first-hand observations in the economics, finance, and marketing disciplines. They left me EXTREMELY cynical about all academics. I have seen: 1) A well-known professor at an Ivy-league B-school openly talking to his class about how the NAR funded his latest study on the real estate market, which (of course) found that the market was not overvalued. This was 2007. The co-author on the paper was the dean of the B-school. He is no longer the dean, but only because he seems to have moved on to more lucrative consulting roles. 2) An older professor, who was one of the first to be in the group labeled "behavioral economics," when asked why he wasn't writing a book focused on more mainstream readers like some other such big names were at the time, STRONGLY hinting if you filled in the blanks although not outright saying that their ethics were questionable, they were all forming hypotheses that were useful for for consulting dollars and dredging up data to support it, and he was too old school (and too old) to be interested in playing that game. 3) A full cohort of marketing grad students (6-7) talking openly about how the system is messed up, and to get a job, you absolutely have to have a good dissertation with a strong results, but to avoid p-hacking allegations, you have to shoot in the dark. So you're left in this situation where you commit financial and career suicide if you DON'T make up data to support your thesis. It is not rare. It is an open secret. No one likes it. They hate it. But they also feel like they are the stupid idiot if they try to fight against it alone. It is how the system works. Only a true saint would accept data that actually does NOT support their hypothesis because it makes it unlikely they will get a job in a very competitive market. Some people do report non-findings, but they are rare. IMO, academic departments should specifically seek out graduates who were willing to report that and stuck by it. But that is not the way the world works. As another poster pointed out, we all like stories, and no significance means no good story to tell.
rayiner 3y ago
Charlatans took advantage of the reputation of “science” in the 20th century. Thankfully, public perception is already adjusting.
scrum-treats 3y ago
Experimentation in moral psychology without robust, independently verified design is an incredibly slippery slope. This is why certain programs require interdisciplinary approaches (e.g., philosophy, psychology, linguistics, math, physics, neuroscience, biology, history). Otherwise people just make stuff up. And if there's no one to run independent replication studies, and no journals require replication studies prior to publication, then we get... this. Just wait for people to dive into the false equivalence of scales and measures used in political research. It's very much "make it up and see if it sticks." The goal appears to be Psychology Today, not scientific rigor. I wrote about this in graduate school, as part of my hefty statistics coursework, it bothered me so much.

pocketsand 3y ago

Seems like behavioral economics really just receives splash damage from what is the massive fraud at the heart of experimental social psychology. At this point I basically assume any catchy finding from the field is at best HARKed or p-hacked and at worst completely fraudulent.

chongli 3y ago
Why aren't journals requiring pre-registration? It's an absolute catastrophe and no one seems to care!
- pocketsand 3y ago
  Pre-registration protects against HARKing and p-hacking, but it unfortunately doesn't protect at all against outright fraud, which we appear to see from the likes of Ariely and Gino.
  - chongli 3y ago
    Right. Outright fraud needs to be dealt with much more severely. Total disgrace, loss of tenure, and potential prison time.
FreshStart 3y ago
The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You can even do ethical virtual experiments with the data of 3 billion people. Just specify the situation, the person and what's to happen.. Voila. All things happen regularly at some point..
- esafak 3y ago
  > The field migrated to were the data is. Some communication dark departments within Meta and Google, sponsored by three letters. You really need to clarify this.
  - cool_dude85 3y ago
    I think it's pretty clear what the post is saying. It doesn't need to be clarified, it needs to be supported, though the nature of the claim is such that it would be tough to provide direct evidence.
  - meepmorp 3y ago
    I’m going with: > The government is trying to control your mind through the internet
  - scrps 3y ago
    Same techniques (largely derived from psychology, statistics, sociology, etc) used in social media marketing, intent tracking, etc has uses in national security. Same thing that might let meta know you might be interested in batman pajamas can be used to sort out if you are being radicalized to action by some group. Interesting example is fractalrabbit. Group of people who both have the theoretical training and practical experience is probably pretty small so private/public crossover isn't unreasonable.

passwordoops 3y ago

Me too, then following my Ph.D at a relatively small school went off to a post-doc at a major research university where the grants are millions or you're worthless... That environment essentially forces you to commit fraud so I got out pretty quick

cbsmith 3y ago

Consider how many studies are being produced each day. That there is a story every day like this suggests that academia is making an effort to identify bad actors, but the frequency of discovery is actually quite low.

tnecniv 3y ago
Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. I would think instances of fabrication are quite low — it’s high-risk, medium reward. Even if it goes undiscovered, academia is a marketing game, and you can’t control what other people feel is impactful or not. It’s very easy to do bad science without falsifying data, which is probably the deadliest sin in academia.
- cbsmith 3y ago
  > Well we don’t know how many people falsify data because, if it goes undiscovered, we don’t know it happened. It could easily be that the rate at which data is falsified is quite low and we are good at catching it, or it is high and we are bad at catching it. Yes, which is why I said the frequency of discovery is low. Seeing a story in the paper about fraud every day doesn't really tell us anything about the rate of fraudulence, which I was hoping would become transparent with my point.

lolinder 3y ago

To be fair, industry has a similar problem: every day there's a new company that gets in trouble for abusing customer data, losing customer data, abusing customer trust, making it hard to cancel a subscription, etc, etc. People in general tend to be dishonest if they think they can get away with it. I'm still glad I didn't go into academia for many other reasons, but I don't think I dodged an ethics bullet.

notahacker 3y ago
yeah, this Reputation probably matters more for academia (people go into it because they think the research is important, not for the money, and they think they're standing on the shoulder of giants and not amiable bullshitters taking the credit for findings actually created by their overworked research assistants kludging the data), but if you think academia is full of lies and half truths relative to other professions, you haven't encountered many startups!

usednet 3y ago

Could somebody provide me some links to other recent scandals in academia/research? Very interested.

pocketsand 3y ago
Check out the datacolada web site. https://datacolada.org/ https://datacolada.org/
tnecniv 3y ago
I forget the guys name (he’s German iirc), but a very impactful one was at Bell Labs in the 00’s. STEM PhD programs now all have a research ethics component because of him. While you can’t teach someone to be ethical, it at least hammers home how many people you are defrauding by falsifying data, which includes everyone from your funding organizations to the lowly PhD students whose advisors are making them iterate on your study and don’t understand why, after three years, it still isn’t working.
dmvdoug 3y ago
Retraction Watch, maybe? https://retractionwatch.com/ https://retractionwatch.com/

tnecniv 3y ago

While fields are small, academia is a big place. Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line, and from an emotional perspective, their academic work is tied to their self-image. It’s also arguably a lot harder to do in the hard sciences but there have still been examples. In the hard sciences, I feel like the solution is some method to increase how much people at different labs talk to one another (in addition to including all raw data and processing code from experiments). If an impactful article comes out, labs will try to build on it. Not every lab will have the equipment but those in the sub area might. If you’re trying to iterate on a study, and you can’t reproduce the base result before adding your modifications, you might just think you’re dumb or doing something dumb. There’s likely nobody outside maybe your research group who can help you. However, if it was easier to talk to other researchers in the field (like a web forum), you might find out that nobody has been able to get the original experiment working! In the social sciences, I’m not sure what you do besides running automated checks on submitted data. Humans are bad at generating things that look random (for example, we use more 3s and 7s than nature), and you can probably flag warnings. Either that or include paper trails like a lab notebook in chemistry — which is considered a legal document and thus must be kept in pen. That way you can establish a chain of custody for the data from collection to publication.

pevey 3y ago
> Most academics won’t falsify data — from a pragmatic perspective, that’s their career on the line In my experience, this is absolutely not true. I mostly only have first-hand observations in the economics, finance, and marketing disciplines. They left me EXTREMELY cynical about all academics. I have seen: 1) A well-known professor at an Ivy-league B-school openly talking to his class about how the NAR funded his latest study on the real estate market, which (of course) found that the market was not overvalued. This was 2007. The co-author on the paper was the dean of the B-school. He is no longer the dean, but only because he seems to have moved on to more lucrative consulting roles. 2) An older professor, who was one of the first to be in the group labeled "behavioral economics," when asked why he wasn't writing a book focused on more mainstream readers like some other such big names were at the time, STRONGLY hinting if you filled in the blanks although not outright saying that their ethics were questionable, they were all forming hypotheses that were useful for for consulting dollars and dredging up data to support it, and he was too old school (and too old) to be interested in playing that game. 3) A full cohort of marketing grad students (6-7) talking openly about how the system is messed up, and to get a job, you absolutely have to have a good dissertation with a strong results, but to avoid p-hacking allegations, you have to shoot in the dark. So you're left in this situation where you commit financial and career suicide if you DON'T make up data to support your thesis. It is not rare. It is an open secret. No one likes it. They hate it. But they also feel like they are the stupid idiot if they try to fight against it alone. It is how the system works. Only a true saint would accept data that actually does NOT support their hypothesis because it makes it unlikely they will get a job in a very competitive market. Some people do report non-findings, but they are rare. IMO, academic departments should specifically seek out graduates who were willing to report that and stuck by it. But that is not the way the world works. As another poster pointed out, we all like stories, and no significance means no good story to tell.

rayiner 3y ago

Charlatans took advantage of the reputation of “science” in the 20th century. Thankfully, public perception is already adjusting.

scrum-treats 3y ago

Experimentation in moral psychology without robust, independently verified design is an incredibly slippery slope. This is why certain programs require interdisciplinary approaches (e.g., philosophy, psychology, linguistics, math, physics, neuroscience, biology, history). Otherwise people just make stuff up. And if there's no one to run independent replication studies, and no journals require replication studies prior to publication, then we get... this. Just wait for people to dive into the false equivalence of scales and measures used in political research. It's very much "make it up and see if it sticks." The goal appears to be Psychology Today, not scientific rigor. I wrote about this in graduate school, as part of my hefty statistics coursework, it bothered me so much.