9 ms·

Please Commit More Blatant Academic Fraud (2021)

153 points by jxmorris12 2y ago

nis0s 2y ago
> Proclaiming that your work is a “promising first step” in your introduction, despite being fully aware that nobody will ever build on it. Science produces discrete units which can be used in different ways, if not in their exact form from a preceding research. I am not sure it’s reasonable to say that existing ideas, even if not cited, are not inspirational (to the researchers themselves). Peer-review isn’t perfect, but I think that all accepted papers have something academically or scientifically relevant, even if there’s no guarantee that the paper will generate hundreds of subsequent citations. I think improving your subsequent work is more important, which includes mentioning why you think some previous work may not be as relevant anymore. This last step is often missing from many research papers. I think the author is right that it doesn’t quite make sense to publish anything you know isn’t quite correct. But I can think of several papers in different fields which someone may think are “not quite correct”, but the goal of such papers, I think, is to demonstrate the power of low probability scenarios, or edge cases. Edge cases are important because they break expected behavior, and are often the root cause of system fragility, system evolution, or poor generalization in other systems.
gumbojuice 2y ago
For one of my first papers during PhD, I had a blatant bug in my implementation. Only realized while doing further research building on that previous paper. I think in any field it's natural to start out naive with an idea that may just be a few steps away from solving something important. Somewhere in the middle only to realize you're not, and then scramble with the ethical dilemma around your work, is it "good enough" or not. I was there anyway.
- boxed 2y ago
  There shouldn't be an ethical dilemma. If you find out your previous paper was wrong, you can publish a new paper saying so. That's 2 papers for the price of one(ish). Everyone wins :P
Peteragain 2y ago
I agree with the analysis completely, but the solution is depressing. I keep thinking that publications on arxiv might be a better source of knowledge given the motivation for publishing there is not a contribution to career progression. Keyword search over arxiv papers? But perhaps we should bring back the idea of anonymous publishing:-0
smolder 2y ago
It's unfortunate that it's more work and less reward to shut down fraud that to perpetrate it. I think its more of a problem than ever before, even though grifting has always been a thing.
GuestFAUniverse 2y ago
I am a co-author on a paper I never asked for, but my supervisor insisted, because the petty idea upgrading his desperate try to the point of "considerable at all" came from me. It was a chair which normally prouded itself of only publishing in the most highly regarded journals of its field (internally graded A, B, C). They had a few years without viable paper. Desperate to publish. From my POV this was a D. The paper is worthless crap, hastily put together within two weeks. It should have been obvious to the reviewers. I feel ashamed that my name is on it. I wish I could retract it. So, yes please: make it hard to impossible for paper mills and kill the whole publish or perish approach.
- deleted 2y ago
  [deleted]
- cgcrob 2y ago
  If it’s any consolation I split up with an ex partner after she wanted to put me as a co-author on a pseudoscience bullshit paper that she was working on to try and hit her quota. Her entire field, in the social sciences, is inventing a wild idea and using meta analysis to give it credibility. Then flying to conferences and submitting expenses. I contributed nothing other than a statistical framework which was discarded when it broke their predefined conclusion.
  - anoncow 2y ago
    When research is just means to an end... I think as children if we are taught what earning a living means, people who only want to make ends meet would try to do it using other less damaging methods. For e.g., sales and marketing are not bad places for such people. When it comes to research people should know that perhaps money will not be great. It is because we aren't aware of the full picture as children, we follow our passions (or we follow cool passions) and then realise that money is also important and then resort to unethical means to get that money. Let's be transparent about hard fields with children so that when they enter such fields they know what they are getting into.
    - 2y ago
ein0p 2y ago
What's funny is that if you're well-read in your field, all such bullshit is plainly apparent. You know what the good baseline is, for example, and you know why the author didn't choose it. You know the deficiencies of the benchmarks and see how they were exploited to juice the results. You know their approach is infeasible IRL and can clearly articulate why. Etc, etc. You folks aren't fooling anyone but fools. It's sort of like e.g. a lazy employee thinks their manager doesn't know they're slacking off like mad. As a former manager I can tell you - that is most certainly not the case, and there's no way to hide.
- 2cynykyl 2y ago
  This is a very important comment. And to flip the perspective, name recognition in our given fields is not just a vanity metric for getting more funding. It is earned by publishing quality work. I can trust people I know who do things correctly.
- Tyr42 2y ago
  This was the major upside to taking a seminar class where we read papers and discuss them. The prof tried to highlight some of these points. Not just seeing the graphs that are there but also the graphs they could have added but didn't.
wucke13 2y ago
Tangentially relevant: Gernots list of benchmarking crimes. https://gernot-heiser.org/benchmarking-crimes.html https://gernot-heiser.org/benchmarking-crimes.html
- bjackman 2y ago
  Ha, I thought this would be a really useful resource but I think the people the author is complaining about do much better than most benchmarking I see in the industry Almost all the benchmarking results I see is just a percentage difference between two algebraic means, no statistical analysis whatsoever. Very common interaction: QA folks say "your change degraded some of our metrics and improved some others". I know they are full of shit because it's impossible that my change improved any perf metrics. I ask for statistical details, they don't have any, this meeting was a waste of time, it will be next time too. The fact that I get these reactions suggests that everyone else just lets each other get away with it.
  - porridgeraisin 2y ago
    Yep. The most recent example that's stuck in my head is actually much worse: they didn't even take the mean! One sample! https://github.com/denoland/pm-benchmark https://github.com/denoland/pm-benchmark Check the run bench shell script (there's not much else in the repo anyways)
    - genewitch 2y ago
      Hey, that's perfectly valid for arguing with your friend about which one to deploy on our server, all things equal. I do this sort of thing to see what tools are faster all the time. ripgrep, ag(silver searcher), grep, MongoDB was one we were arguing about for a while recently.
      - burntsushi 2y ago
        Which one won? :) (I'm the author of ripgrep.)
proto-n 2y ago
"For the first time, researchers reading conference proceedings will be forced to wonder: does this work truly merit my attention? Or is its publication simply the result of fraud? [...] But the mere possibility that any given paper was published through fraud forces people to engage more skeptically with all published work." Well... spending a few weeks reproducing a shiny conference paper that simply doesn't work and is easily beaten by any classical baseline will do that to you in the first few months of your PhD imo. I've become so skeptic over the years that I assume almost all papers to be lies until proven otherwise. "This surfaces the fundamental tension between good science and career progression buried deep at the heart of academia. Most researchers are to some extent “career researchers”, motivated by the power and prestige that rewards those who excel in the academic system, rather than idealistic pursuit of scientific truth." For the first years of my PhD I simply refused to parttake in the subtle kinds of frauud listed in the second paragraph of the post. As a result, I barely had any publications worth mentioning. Mostly papers shared with others, where I couldn't stop the paper from happening by the time I realized that there is too little substance for me to be comfortable with it. As a result, my publication history looks sad and my carreer looks nothing like I wished it would. Now, a few years later, I've become much better at research and can now get my papers to the point where I'm comfortable submitting them with a straight face. I've also came to terms with overselling something that does have substance, just not as much as I wish it had.
- huijzer 2y ago
  > I've become so skeptic over the years that I assume almost all papers to be lies until proven otherwise. I couldn't agree more. I have read a lot of psychology papers during my PhD and I think there is very little signal in the papers. Many empirical papers for example use basically the same "gold standard" analysis, which is fundamentally flawed in many ways. One problem for example is that if you would use another statistical model, then the conclusions would often be wildly different. Another being that the signal is often so weak that you can't use it to predict much (to be useful). If you try to select individuals for example, the only thing you can tell is that the group on average is less neurotic. But for individuals there is no better chance of picking the right one than average. The point of a good paper is to take these sketchy analyses and write a beautiful story around it with convincing speculation. It sounds absurd but take a random quantitative psychology paper and check which percentage of the claims made in the discussion are actually based on the actual data from the paper. But the worst part about this is that these problems exist for literally decades. Nobody cares. The funding agencies grade people not on correctness but on the number of citations. As a result, you see that many subcultures exist who's sole existence is about promoting the importance of their subculture. It is quite common in academia to cite someone in the introduction just to "prove" that some idea is worth pursuing. But does it work? Doesn't matter. Just keep writing papers. So I'm not saying that all research is bad. I'm saying that indeed most papers are not very useful or correct. Many researchers try, but the incentives are extremely crooked.
bjackman 2y ago
> Submitting a paper to a conference because it’s got a decent shot at acceptance and you don’t want the time you spent on it go to waste, even though you’ve since realized that the core ideas aren’t quite correct. I don't see a problem with this? If papers are the vehicle for conference entries why shouldn't authors submit it just because it's wrong? Conferences are for discussion. So go there and discuss it... "My paper says XYZ, but since I wrote it I realised ABC" - sounds like a good talk to me? (Naivety check: I am not an academic)
- tgsovlerkhgsel 2y ago
  The issue is omitting the second ("ABC") part. Mainly because "XYZ looks like a promising way to cure cancer" would be considered a more interesting result than "XYZ looks like a promising way to cure cancer, but isn't due to ABC", and thus the former will be accepted at more prestigious conferences/journals than the latter.
- lgeorget 2y ago
  In other fields than computer science, that would be more the case I think because conferences are not given as much importance. Journals are what matter and since these publications take more time and are usually more selective (for the well-known journals at least), you tend to have better science in them. Computer science have few journals and the standard of publication is the conference.
- sideshowb 2y ago
  Yes. As the saying goes, If we knew what we were doing it wouldn't be research. Finished papers often have flaws, if you try to write something perfect you may never finish it. They're called limitations and you list them in conclusions and suggest addressing in future work. (Experience check: I is one)
- light_hue_1 2y ago
  That's not what papers are for. But I can see how not being an academic would make you think that. What you're describing are workshops with what we would call non-archival proceedings. Places where you write whatever you want and then talk about it. Publications, conference or journal, are supposed to be what are called archival. They are a record of what we've discovered and want to share with the world. They are supposed to be sent into the world after we carefully complete a line of work. Publications are not supposed to spam the system with half-baked junk. Sadly, that's what a lot of people are doing these days.
orleyhuxwell 2y ago
[dead]
userbinator 2y ago
(2021) Together, we can force the community to reckon with its own shortcomings, and develop stronger, better, and more scientific norms. It is a harsh treatment, to be sure – a chemotherapy regimen that risks destroying us entirely. But this is our best shot at destroying the cancer that has infected our community. And now, nearly 4 years later, research funding gets DOGE'd...
- pbronez 2y ago
  Yup. and MAHA'd. The question is: what comes next? Once the current administration executes their democratic mandate to destroy institutions, how should we rebuild those capabilities? In this case, how could the federal government ensure that academic funding flows to researchers and institutions that are doing genuinely high quality research? How can we remove funding from low-quality and fraudulent actors?
bigbacaloa 2y ago
[dead]
tonyg 2y ago
> Undermining the credibility of computer science research is the best possible outcome for the field, since the institution in its current form does not deserve the credibility that it has. Horseshit. This might be true for AI research (and even there that's an awfully broad brush you're using, mate), but it's certainly not true for other areas of computer science.
- SirHumphrey 2y ago
  Maybe your lab is different (if you work in a research setting), but most of the researchers will readily admit that most of their research output is at least somewhat bull**. It's something that is trained in to people from high school research projects onwards - people judging your results usually do not have time or ability to research you work and even if they have they usually have much more importing things to do than check your mediocre results. As a society, we have far too much trust in science, however any time this argument is brought up, we focus on conspiracy theorists who struggle with 100+ years old theories as if the visage of the public trust in science will change their mind, ignoring that any member of the public, who accidentally discovers the Jenga tower the science is built on (but hidden) will become much more likely to believe those charlatans in the future.
  - tensor 2y ago
    Excuse me? Based on what? In my time in academia exactly zero researchers would claim that their work is somewhat bullshit. As a society, there is laughably little support for science, instead the majority of policy and business decisions are based on fairy tales and snake oil. We need more trust in science.
    - genewitch 2y ago
      As of this comment scroll location, there are at least three people that admitted to co-authiring a paper they knew was BS. So you don't know anyone, but I can counter with those three anecdotes. 3>1 Qed I am right, statistically.
- 2y ago
deleted 2y ago
[deleted]
vladms 2y ago
Sometimes it is good to make parallels with other things to check if proposals make sense. How would sound "go and try to steal every car out there, so that car manufacturers improve their car security" ? For me it sounds counter-productive. I have a feeling that lately (tens of years) many people try to focus on the negatives, rather than the positives. Should we focus on the 3 amazing papers this year, cited by hundreds, that resulted in clear progress or should we complain that 100 papers are useless? Let's focus on 100 because "someone is wrong on the internet". I did a PhD (so might have more experience) but papers are meant for dissemination. For me everybody that wants to have them "perfect"/"useful" papers imagines a system that does lots of work for them. The system could be improved, but if anything (just throwing an idea) maybe researchers should try to do research in the industry to prove themselves. Then come back after 10 year in academia (maybe with savings) so that they are more independent of "career progression". A lot of research was done (historically, >100 years ago) by rich people, not constrained by a career.
- anjc 2y ago
  > Should we focus on the 3 amazing papers this year, cited by hundreds, that resulted in clear progress or should we complain that 100 papers are useless? Agree, and it seems that this is how fields naturally evolve anyway.
bowsamic 2y ago
It's not an exaggeration to say I literally don't know a single person who doesn't engage in this kind of academic fraud to some degree
lqet 2y ago
> * A group of colluding authors writes and submits papers to the conference. > * The colluders share, amongst themselves, the titles of each other's papers, violating the tenet of blind reviewing and creating a significant undisclosed conflict of interest. > * The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers. Is it that common that conference reviewers also submit papers to the conference? Wouldn't that alone already be a conflict of interest? (After all, you then have an interest in dismissing as many papers as possible to increase the likelihood of your own paper being accepted). And how do you create "duplicate accounts"? The conferences I have submitted to, and reviewed for, all had an invitation-like process for potential reviewers.
- proto-n 2y ago
  Not only common, it's become a requirement to review papers if you submit one yourself. Yeah it's not ideal for multiple reasons (what you said + prompt engineering grad students dismissing proper papers without having the slighest idea about the field), but the amount of submissions is so incredibly huge that it's impossible to do it any other way.
  - lqet 2y ago
    Then I guess I should be grateful that my academic niche is so small.
- michaelt 2y ago
  Many bodies that fund academic work will happily pay for you to fly to a conference and stay at a hotel if you're presenting a paper at the conference - but they'll be a lot less willing if you aren't presenting anything. So a decent % of attendees will be presenting papers. And finding reviewers who know their stuff, who'll work for free, and who'll review thoroughly in a short timescale isn't easy.
- twic 2y ago
  Even if this reviewers weren't allowed to be submitters, if there is more than one conference, or the conference runs for more than one year, the same mechanism can be used.
ploynog 2y ago
Double-blind review is a mirage that does not hold up. While I was in academia I reviewed a paper that turned out to be a blatant case of plagiarism. It was a clear Level 1 copy according to the IEEE plagiarism levels (Uncredited Verbatim Copy of more than 50% of a single paper). I submitted all of these findings with the original paper and what parts were copied (essentially all of it) as my review. A few days later I got an email from the author (some professor) who wanted to discuss this with me, claiming that the paper was written by some of his students who were not credited as authors. They were unexperienced, made a mistake, yaddah yaddah yaddah. I forwarded the mail to the editors and never heard from this case again. I don't expect that anything happened, the corrective actions for a level-1 violation are pretty harsh and would have been hard to miss. The fact that this person was able to obtain my name and contact info shattered any trust I had in the "blind" part of the double-blind review process. The other two reviewers had recommended to accept the paper without revisions, by the way.
- deleted 2y ago
  [deleted]
- 0_____0 2y ago
  This seems like an issue of administration rather than an issue with the idea of a double-blind review. If you conduct a review that isn't properly blinded, and doesn't have an observable effect, can it really be called a double blind review?
  - velcrovan 2y ago
    Maybe more that a non-idealistic model of the real world, and common direct experience, show that incentives strongly favor an administrative approach that compromises the double blind.
    - 0_____0 2y ago
      Unless there's a better way to do it, I this shows a need for better structures for governance and auditing of review boards... Information and science care not for our human folly, it's up to us to seek and execute them properly.
anjc 2y ago
Overly pessimistic, and doesn't acknowledge that heads of steam only build behind promising findings, while the deficient (or 'fraudulent') work die on the vine, published or not. In other words the system tends to work. Secondly, there are many ingredients required to successfully publish, communicate science, foster collaboration, etc., beyond technical brilliance. I'm sure we all know many technically brilliant people whose career never advanced because they lacked in some necessary area. People shouldn't be discouraged from improving in all areas because OP's delicate genius is offended by their technical ability. Speaking of discouragement, it's a shame and a disgrace that you publicly called your colleague's work bullshit, including a first author that isn't yourself.
- bluefirebrand 2y ago
  > Overly pessimistic, and doesn't acknowledge that heads of steam only build behind promising findings, while the deficient (or 'fraudulent') work die on the vine, published or not. This might be true in hard sciences where a "head of steam" can only build based on real, replicatable results But it's very common that public policy is proposed and adopted based on findings from soft sciences like psychology and sociology If policy is adopted based on a research paper, I would count that as a "head of steam" being built. And if that paper is fraudulent, then we are adopting well-intentioned policy on false pretenses
  - anjc 2y ago
    I did just mean AI and Computer Science per OP. By "head of steam" I mean to say that much research is built on it, think the likes of "Attention is All you Need". There isn't quite an equivalent of this in public policy in my experience. Conversely, computer science/AI doesn't have an equivalent of the rigor that public policy research tends to go through. CS has e.g., benchmark datasets, typical evaluation metrics, but these are more like norms rather than requirements, whereas in public policy, instruments for validations are far more rigorously tested and enforced. Depending on the area. I agree that outright fraud would be detrimental, but I think OP overblows this issue completely and should apologise to his co-authors.
jillesvangurp 2y ago
The academic world is essentially the world's oldest social network. The way academic publishing works is through a convoluted reputation system of academics endorsing each other's works via various publications, giving each other a 'like' by referring work, debating work in public at conferences, hiring each other's students, protege's, etc. Fraud here basically means faking reputation. There are many ways to do this. And it's common because doing scientific work goes hand in hand with very generous funding. And money corrupts things. So attempts to fake reputation, plagiarize work, artificially boost relevance through low reputable referencing, bribes, etc. are as old as scientific publishing is. There are a few interesting dynamics here that counter this: high quality publications will want to defend their reputation. E.g. Nature retracting an article tends to be scandalous. They do it to preserve their reputation. And it tends to be bad for the reputation of affected authors. Their reputation is based on them having very high standards and a long history of important people publishing important things that changed the world. Every time they publish something, that's the reputation that is at stake. So, they are strict. And they should be. The problem is all the second and third rate researchers that make up most of the scientific community. We don't all get to have Einstein level reputations. And things are fiercely competitive at the bottom. And if you have no reputation, sacrificing it is a small price to pay. Also the prestigious publications are guarded by an elitist, highly political, in-crowd. We're talking big money here. And money corrupts. So, this works both ways. With AI thrown in the mix, the academic world has to up its game. And the tools it is going to have to use are essentially the same used in other social networks. Bluesky, Twitter, etc. have exactly the same problem as scientific publishers; but at a much larger scale. They want to surface reputable stuff and filter out all the AI generated misinformation and it's an arms race. One solution is using more AI or trying other clever tricks. A simpler solution is tying reputations to digital signatures. Scientific work is not anonymous. You literally stake your reputation by tying your (good) name to a publication and going on the record by "publishing" something. Digital signatures add some strength to that that AIs can't fake or forge. Either you said it and signed it; or you didn't. Easy to verify. And either you are reputable, by having your signature associated with a lot of reputable publications, or you are not. Also easy to verify. If disreputable stuff gets flagged, you simply scrutinize all the authors and publications involved and let them sort out their reputations by taking appropriate actions (firing people, withdrawing articles, publicly apologizing, etc.). They'll all be eager to restore their reputations so that should be uncontroversial. Or they don't and lose their reputation. Digital signatures are a severely underused tool currently. We've had access to those for half a century or so. The challenge isn't technical but institutional. Lots of disreputable people and institutions are currently making a lot of money by operating in the shadows. The tools are there to fix this. But people don't seem to necessarily want to.
nicwilson 2y ago
Hmmm, I wonder if you could turn this into a sport and have like one paper per year per group of total BS, and shame on the reviewers/conference/journal if they don't catch it, and kudos to the submitters the more blatant it is. Come to think of it, is there a "Journal of Academic Fraud"?
michaelt 2y ago
> And we must ensure that explicit suggestions to modify one’s science in the service of one’s career – “you need to do X to be published”, “you need to publish Y to graduate”, “you need to avoid criticizing Z to get hired” – carry social penalties as severe as a suggestion of plagiarism or fraud. One of the pernicious things in this area is that, even as we teach young researchers how to avoid making mistakes and engage sceptically with the work of others and that scientific fraud is a nontrivial issue, we also tell them how to commit fraud themselves and that their competition is doing it. "Watch out for P-hacking, that's where the researcher uses a form of analysis that has a small chance of a false positive, and analyses loads of subsets of your dataset until a false positive arises and just publishes that one" "Watch out for over-fitting to benchmarks, like a car taking the speed crown by sacrificing the ability to corner" "Watch out for incomplete descriptions of test setups, like testing on a 'continent-scale map' but not mentioning how detailed a map it was" "Watch out for citations where the cited paper doesn't say what is claimed, some people will copy-and-paste citations without reading the source paper" "Watch out for papers using complicated notation, fancy equations and jargon to make you feel this looks like a 'proper' paper" "Watch out for deceptive choice of accurate numbers, like a study with a 25% completion rate including the drop-outs in the number of participants" "Watch out for simulations with inaccurate noise models, if the noise is gaussian in the simulation but a random walk in reality, great simulated results won't transfer to reality" I've made no suggestion at all that you should modify your science or commit fraud - but I've also just trained you in how to do it.
- proto-n 2y ago
  It's really not that hard to come up with ways to commit fraud if you want to. On the other hand, it's very easy to make such mistakes if you don't know to avoid them. This characterization is very misguided IMO.
  - michaelt 2y ago
    Ah, perhaps I wasn't clear about what I'm trying to say. I don't think we should stop training researchers in common mistakes and fraudulent methods to watch out for. I'm just saying: I don't believe anyone actually tells budding researchers that they should commit fraud. Instead I think the process is probably more like this: Year 1: Statistics/research training. Here are a load of subtle mistakes to watch out for and avoid. Scientific fraud happens sometimes. Don't do it, it's very dishonest. Year 2: Starting research. Gee a lot of these papers I'm reading are hard to reproduce, or unclear. Maybe fraud is widespread - or maybe they're just smarter or better equipped than me. Year 3: "You really ought to have published some papers by now, the average student in your position has 3 papers. If you don't want to flunk out you really need to start showing some progress"
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
ngriffiths 2y ago
I had two experiences at the polar opposites of the spectrum - one research team I worked on had very high standards and was comfortable being patient for material that had value. The other involved an approach that obviously stood no chance to be useful to anyone. Some differences: - The first one was in a space with more low hanging fruit - The first one was after large effect sizes, not the kind where you can massage the statistical model - The second one was a topic with far higher public interest - The second one was primarily an analytic project, whereas the first one was primarily experimental I feel like bad science lives in the middle of a spectrum - you have young fields/subfields with boring but impressive experimental breakthroughs, and on the other end you have highly political questions that have been argued to death without resolution. Bad science is about borrowing some of the strategies used in politics because all the important experiments have already been done.
throw4847285 2y ago
Unfortunately, I think this article is not pessimistic enough. If it were true that academic fraud is important for establishing the boundaries of respectable scholarship in a field, then behavioral economics wouldn't exist anymore. And evolutionary psychology would be a fraction of its current size. Trendiness trumps all notions of academic rigor, and as long as a field "feels like" it's on the cutting edge it can go pretty far before collapsing in on itself.
- daveguy 2y ago
  I'm wondering how someone knows the size of the field of evolutionary psychology and what it should be instead of what it is so well that they feel no evidence is necessary. Are you in the field yourself? Or do you have questions about evolution in general? Or none of the above?
  - tovej 2y ago
    Evolutionary psychology is highly speculative and is widely known to be full of questionable results. So much so that there's a wikipedia page for how bad it is: https://en.m.wikipedia.org/wiki/Criticism_of_evolutionary_psychology https://en.m.wikipedia.org/wiki/Criticism_of_evolutionary_ps...
    - throw4847285 2y ago
      The SEP page is a slightly more rigorous take, though unlike Wikipedia it is clearly primarily authored by one person, a philosopher of biology who I don't know much about. Still, I think it's a fair and detailed overview of the field, from a methodological perspective. https://plato.stanford.edu/entries/evolutionary-psychology/ https://plato.stanford.edu/entries/evolutionary-psychology/
  - throw4847285 2y ago
    Just cut out everybody who is studying "sexual selection" and I think the field is way healthier. Weird that so many evo psych scholars of a certain persuasion are fixated on that. I have a theory, but it veers into armchair non-evolutionary psychology, and it's not very nice.
jimbokun 2y ago
> Most researchers are to some extent “career researchers”, motivated by the power and prestige that rewards those who excel in the academic system, rather than idealistic pursuit of scientific truth. This is the funny part. There is little to no power and prestige to be had in the academic system. To a first approximation no one outside academia cares. I was just working as a staff programmer and taking grad courses with my tuition benefit, and found myself getting caught up in the mentality of needing a PhD to really be successful and valuable. Then got a job in industry making far more money and realized how academia is a small self contained world with status hierarchies irrelevant outside that small world.
- vonneumannstan 2y ago
  >This is the funny part. There is little to no power and prestige to be had in the academic system. To a first approximation no one outside academia cares. They have power over their students and relative power over other Professors. That's plenty enough incentive for most. There can also be fame and fortune for the most famous among them. See Francesa Gino, Dan Ariely, etc.
- BeetleB 2y ago
  In the social sciences, there is a lot of prestige to be had. Do some ground breaking work, engage with the public via bestselling books, and then get invited by the president to work on policy. Even if they don't work for the administration, there are plenty of other bodies that will value them and pay large sums of money (or let them have large influence). Very common amongst economists, and more and more common amongst disciplines like psychology. Even in technical fields, if you can manage to become a big name, you can do consulting work and get paid quite well. > Then got a job in industry making far more money This is not a healthy way to look at it. The average mechanical engineer isn't making tons of money in industry. A ME professor at a top university likely makes more. A biology major with just a BS degree will make less than the average biology associate professor. But more importantly, there's a simpler reason why money is a poor metric to measure: You can always get more money in finance or medicine than as a mechanical engineer. Does it make sense to denigrate a whole profession just because one can make more money elsewhere?
jarbus 2y ago
I don't have top-tier publications, and I haven't gotten any awards. I've seen people get awards for bullshit and farming prestigous publications. I only have one citation for work that is 100% mine. But every time I read something like this, I feel proud that I don't bullshit, and at least try to do real science. I truly believe in all my work and every sentence I write. That being said, the insane emphasis on venue is what's pushing me out of academia. I can't compete with people like this.
- tmaly 2y ago
  I am just thinking about what will happen when these papers are used to train LLMs

153 points by jxmorris12 2y ago

> Proclaiming that your work is a “promising first step” in your introduction, despite being fully aware that nobody will ever build on it. Science produces discrete units which can be used in different ways, if not in their exact form from a preceding research. I am not sure it’s reasonable to say that existing ideas, even if not cited, are not inspirational (to the researchers themselves). Peer-review isn’t perfect, but I think that all accepted papers have something academically or scientifically relevant, even if there’s no guarantee that the paper will generate hundreds of subsequent citations. I think improving your subsequent work is more important, which includes mentioning why you think some previous work may not be as relevant anymore. This last step is often missing from many research papers. I think the author is right that it doesn’t quite make sense to publish anything you know isn’t quite correct. But I can think of several papers in different fields which someone may think are “not quite correct”, but the goal of such papers, I think, is to demonstrate the power of low probability scenarios, or edge cases. Edge cases are important because they break expected behavior, and are often the root cause of system fragility, system evolution, or poor generalization in other systems.

gumbojuice 2y ago

For one of my first papers during PhD, I had a blatant bug in my implementation. Only realized while doing further research building on that previous paper. I think in any field it's natural to start out naive with an idea that may just be a few steps away from solving something important. Somewhere in the middle only to realize you're not, and then scramble with the ethical dilemma around your work, is it "good enough" or not. I was there anyway.

boxed 2y ago
There shouldn't be an ethical dilemma. If you find out your previous paper was wrong, you can publish a new paper saying so. That's 2 papers for the price of one(ish). Everyone wins :P

Peteragain 2y ago

I agree with the analysis completely, but the solution is depressing. I keep thinking that publications on arxiv might be a better source of knowledge given the motivation for publishing there is not a contribution to career progression. Keyword search over arxiv papers? But perhaps we should bring back the idea of anonymous publishing:-0

smolder 2y ago

It's unfortunate that it's more work and less reward to shut down fraud that to perpetrate it. I think its more of a problem than ever before, even though grifting has always been a thing.

GuestFAUniverse 2y ago

I am a co-author on a paper I never asked for, but my supervisor insisted, because the petty idea upgrading his desperate try to the point of "considerable at all" came from me. It was a chair which normally prouded itself of only publishing in the most highly regarded journals of its field (internally graded A, B, C). They had a few years without viable paper. Desperate to publish. From my POV this was a D. The paper is worthless crap, hastily put together within two weeks. It should have been obvious to the reviewers. I feel ashamed that my name is on it. I wish I could retract it. So, yes please: make it hard to impossible for paper mills and kill the whole publish or perish approach.

deleted 2y ago
[deleted]
cgcrob 2y ago
If it’s any consolation I split up with an ex partner after she wanted to put me as a co-author on a pseudoscience bullshit paper that she was working on to try and hit her quota. Her entire field, in the social sciences, is inventing a wild idea and using meta analysis to give it credibility. Then flying to conferences and submitting expenses. I contributed nothing other than a statistical framework which was discarded when it broke their predefined conclusion.
- anoncow 2y ago
  When research is just means to an end... I think as children if we are taught what earning a living means, people who only want to make ends meet would try to do it using other less damaging methods. For e.g., sales and marketing are not bad places for such people. When it comes to research people should know that perhaps money will not be great. It is because we aren't aware of the full picture as children, we follow our passions (or we follow cool passions) and then realise that money is also important and then resort to unethical means to get that money. Let's be transparent about hard fields with children so that when they enter such fields they know what they are getting into.
  - 2y ago

ein0p 2y ago

What's funny is that if you're well-read in your field, all such bullshit is plainly apparent. You know what the good baseline is, for example, and you know why the author didn't choose it. You know the deficiencies of the benchmarks and see how they were exploited to juice the results. You know their approach is infeasible IRL and can clearly articulate why. Etc, etc. You folks aren't fooling anyone but fools. It's sort of like e.g. a lazy employee thinks their manager doesn't know they're slacking off like mad. As a former manager I can tell you - that is most certainly not the case, and there's no way to hide.

2cynykyl 2y ago
This is a very important comment. And to flip the perspective, name recognition in our given fields is not just a vanity metric for getting more funding. It is earned by publishing quality work. I can trust people I know who do things correctly.
Tyr42 2y ago
This was the major upside to taking a seminar class where we read papers and discuss them. The prof tried to highlight some of these points. Not just seeing the graphs that are there but also the graphs they could have added but didn't.

wucke13 2y ago

Tangentially relevant: Gernots list of benchmarking crimes. https://gernot-heiser.org/benchmarking-crimes.html https://gernot-heiser.org/benchmarking-crimes.html

bjackman 2y ago
Ha, I thought this would be a really useful resource but I think the people the author is complaining about do much better than most benchmarking I see in the industry Almost all the benchmarking results I see is just a percentage difference between two algebraic means, no statistical analysis whatsoever. Very common interaction: QA folks say "your change degraded some of our metrics and improved some others". I know they are full of shit because it's impossible that my change improved any perf metrics. I ask for statistical details, they don't have any, this meeting was a waste of time, it will be next time too. The fact that I get these reactions suggests that everyone else just lets each other get away with it.
- porridgeraisin 2y ago
  Yep. The most recent example that's stuck in my head is actually much worse: they didn't even take the mean! One sample! https://github.com/denoland/pm-benchmark https://github.com/denoland/pm-benchmark Check the run bench shell script (there's not much else in the repo anyways)
  - genewitch 2y ago
    Hey, that's perfectly valid for arguing with your friend about which one to deploy on our server, all things equal. I do this sort of thing to see what tools are faster all the time. ripgrep, ag(silver searcher), grep, MongoDB was one we were arguing about for a while recently.
    - burntsushi 2y ago
      Which one won? :) (I'm the author of ripgrep.)

proto-n 2y ago

"For the first time, researchers reading conference proceedings will be forced to wonder: does this work truly merit my attention? Or is its publication simply the result of fraud? [...] But the mere possibility that any given paper was published through fraud forces people to engage more skeptically with all published work." Well... spending a few weeks reproducing a shiny conference paper that simply doesn't work and is easily beaten by any classical baseline will do that to you in the first few months of your PhD imo. I've become so skeptic over the years that I assume almost all papers to be lies until proven otherwise. "This surfaces the fundamental tension between good science and career progression buried deep at the heart of academia. Most researchers are to some extent “career researchers”, motivated by the power and prestige that rewards those who excel in the academic system, rather than idealistic pursuit of scientific truth." For the first years of my PhD I simply refused to parttake in the subtle kinds of frauud listed in the second paragraph of the post. As a result, I barely had any publications worth mentioning. Mostly papers shared with others, where I couldn't stop the paper from happening by the time I realized that there is too little substance for me to be comfortable with it. As a result, my publication history looks sad and my carreer looks nothing like I wished it would. Now, a few years later, I've become much better at research and can now get my papers to the point where I'm comfortable submitting them with a straight face. I've also came to terms with overselling something that does have substance, just not as much as I wish it had.

huijzer 2y ago
> I've become so skeptic over the years that I assume almost all papers to be lies until proven otherwise. I couldn't agree more. I have read a lot of psychology papers during my PhD and I think there is very little signal in the papers. Many empirical papers for example use basically the same "gold standard" analysis, which is fundamentally flawed in many ways. One problem for example is that if you would use another statistical model, then the conclusions would often be wildly different. Another being that the signal is often so weak that you can't use it to predict much (to be useful). If you try to select individuals for example, the only thing you can tell is that the group on average is less neurotic. But for individuals there is no better chance of picking the right one than average. The point of a good paper is to take these sketchy analyses and write a beautiful story around it with convincing speculation. It sounds absurd but take a random quantitative psychology paper and check which percentage of the claims made in the discussion are actually based on the actual data from the paper. But the worst part about this is that these problems exist for literally decades. Nobody cares. The funding agencies grade people not on correctness but on the number of citations. As a result, you see that many subcultures exist who's sole existence is about promoting the importance of their subculture. It is quite common in academia to cite someone in the introduction just to "prove" that some idea is worth pursuing. But does it work? Doesn't matter. Just keep writing papers. So I'm not saying that all research is bad. I'm saying that indeed most papers are not very useful or correct. Many researchers try, but the incentives are extremely crooked.

bjackman 2y ago

> Submitting a paper to a conference because it’s got a decent shot at acceptance and you don’t want the time you spent on it go to waste, even though you’ve since realized that the core ideas aren’t quite correct. I don't see a problem with this? If papers are the vehicle for conference entries why shouldn't authors submit it just because it's wrong? Conferences are for discussion. So go there and discuss it... "My paper says XYZ, but since I wrote it I realised ABC" - sounds like a good talk to me? (Naivety check: I am not an academic)

tgsovlerkhgsel 2y ago
The issue is omitting the second ("ABC") part. Mainly because "XYZ looks like a promising way to cure cancer" would be considered a more interesting result than "XYZ looks like a promising way to cure cancer, but isn't due to ABC", and thus the former will be accepted at more prestigious conferences/journals than the latter.
lgeorget 2y ago
In other fields than computer science, that would be more the case I think because conferences are not given as much importance. Journals are what matter and since these publications take more time and are usually more selective (for the well-known journals at least), you tend to have better science in them. Computer science have few journals and the standard of publication is the conference.
sideshowb 2y ago
Yes. As the saying goes, If we knew what we were doing it wouldn't be research. Finished papers often have flaws, if you try to write something perfect you may never finish it. They're called limitations and you list them in conclusions and suggest addressing in future work. (Experience check: I is one)
light_hue_1 2y ago
That's not what papers are for. But I can see how not being an academic would make you think that. What you're describing are workshops with what we would call non-archival proceedings. Places where you write whatever you want and then talk about it. Publications, conference or journal, are supposed to be what are called archival. They are a record of what we've discovered and want to share with the world. They are supposed to be sent into the world after we carefully complete a line of work. Publications are not supposed to spam the system with half-baked junk. Sadly, that's what a lot of people are doing these days.

orleyhuxwell 2y ago

[dead]

userbinator 2y ago

(2021) Together, we can force the community to reckon with its own shortcomings, and develop stronger, better, and more scientific norms. It is a harsh treatment, to be sure – a chemotherapy regimen that risks destroying us entirely. But this is our best shot at destroying the cancer that has infected our community. And now, nearly 4 years later, research funding gets DOGE'd...

pbronez 2y ago
Yup. and MAHA'd. The question is: what comes next? Once the current administration executes their democratic mandate to destroy institutions, how should we rebuild those capabilities? In this case, how could the federal government ensure that academic funding flows to researchers and institutions that are doing genuinely high quality research? How can we remove funding from low-quality and fraudulent actors?

bigbacaloa 2y ago

[dead]

tonyg 2y ago

> Undermining the credibility of computer science research is the best possible outcome for the field, since the institution in its current form does not deserve the credibility that it has. Horseshit. This might be true for AI research (and even there that's an awfully broad brush you're using, mate), but it's certainly not true for other areas of computer science.

SirHumphrey 2y ago
Maybe your lab is different (if you work in a research setting), but most of the researchers will readily admit that most of their research output is at least somewhat bull**. It's something that is trained in to people from high school research projects onwards - people judging your results usually do not have time or ability to research you work and even if they have they usually have much more importing things to do than check your mediocre results. As a society, we have far too much trust in science, however any time this argument is brought up, we focus on conspiracy theorists who struggle with 100+ years old theories as if the visage of the public trust in science will change their mind, ignoring that any member of the public, who accidentally discovers the Jenga tower the science is built on (but hidden) will become much more likely to believe those charlatans in the future.
- tensor 2y ago
  Excuse me? Based on what? In my time in academia exactly zero researchers would claim that their work is somewhat bullshit. As a society, there is laughably little support for science, instead the majority of policy and business decisions are based on fairy tales and snake oil. We need more trust in science.
  - genewitch 2y ago
    As of this comment scroll location, there are at least three people that admitted to co-authiring a paper they knew was BS. So you don't know anyone, but I can counter with those three anecdotes. 3>1 Qed I am right, statistically.
2y ago

deleted 2y ago

[deleted]

vladms 2y ago

Sometimes it is good to make parallels with other things to check if proposals make sense. How would sound "go and try to steal every car out there, so that car manufacturers improve their car security" ? For me it sounds counter-productive. I have a feeling that lately (tens of years) many people try to focus on the negatives, rather than the positives. Should we focus on the 3 amazing papers this year, cited by hundreds, that resulted in clear progress or should we complain that 100 papers are useless? Let's focus on 100 because "someone is wrong on the internet". I did a PhD (so might have more experience) but papers are meant for dissemination. For me everybody that wants to have them "perfect"/"useful" papers imagines a system that does lots of work for them. The system could be improved, but if anything (just throwing an idea) maybe researchers should try to do research in the industry to prove themselves. Then come back after 10 year in academia (maybe with savings) so that they are more independent of "career progression". A lot of research was done (historically, >100 years ago) by rich people, not constrained by a career.

anjc 2y ago
> Should we focus on the 3 amazing papers this year, cited by hundreds, that resulted in clear progress or should we complain that 100 papers are useless? Agree, and it seems that this is how fields naturally evolve anyway.

bowsamic 2y ago

It's not an exaggeration to say I literally don't know a single person who doesn't engage in this kind of academic fraud to some degree

lqet 2y ago

> * A group of colluding authors writes and submits papers to the conference. > * The colluders share, amongst themselves, the titles of each other's papers, violating the tenet of blind reviewing and creating a significant undisclosed conflict of interest. > * The colluders hide conflicts of interest, then bid to review these papers, sometimes from duplicate accounts, in an attempt to be assigned to these papers as reviewers. Is it that common that conference reviewers also submit papers to the conference? Wouldn't that alone already be a conflict of interest? (After all, you then have an interest in dismissing as many papers as possible to increase the likelihood of your own paper being accepted). And how do you create "duplicate accounts"? The conferences I have submitted to, and reviewed for, all had an invitation-like process for potential reviewers.

proto-n 2y ago
Not only common, it's become a requirement to review papers if you submit one yourself. Yeah it's not ideal for multiple reasons (what you said + prompt engineering grad students dismissing proper papers without having the slighest idea about the field), but the amount of submissions is so incredibly huge that it's impossible to do it any other way.
- lqet 2y ago
  Then I guess I should be grateful that my academic niche is so small.
michaelt 2y ago
Many bodies that fund academic work will happily pay for you to fly to a conference and stay at a hotel if you're presenting a paper at the conference - but they'll be a lot less willing if you aren't presenting anything. So a decent % of attendees will be presenting papers. And finding reviewers who know their stuff, who'll work for free, and who'll review thoroughly in a short timescale isn't easy.
twic 2y ago
Even if this reviewers weren't allowed to be submitters, if there is more than one conference, or the conference runs for more than one year, the same mechanism can be used.

ploynog 2y ago

Double-blind review is a mirage that does not hold up. While I was in academia I reviewed a paper that turned out to be a blatant case of plagiarism. It was a clear Level 1 copy according to the IEEE plagiarism levels (Uncredited Verbatim Copy of more than 50% of a single paper). I submitted all of these findings with the original paper and what parts were copied (essentially all of it) as my review. A few days later I got an email from the author (some professor) who wanted to discuss this with me, claiming that the paper was written by some of his students who were not credited as authors. They were unexperienced, made a mistake, yaddah yaddah yaddah. I forwarded the mail to the editors and never heard from this case again. I don't expect that anything happened, the corrective actions for a level-1 violation are pretty harsh and would have been hard to miss. The fact that this person was able to obtain my name and contact info shattered any trust I had in the "blind" part of the double-blind review process. The other two reviewers had recommended to accept the paper without revisions, by the way.

deleted 2y ago
[deleted]
0_____0 2y ago
This seems like an issue of administration rather than an issue with the idea of a double-blind review. If you conduct a review that isn't properly blinded, and doesn't have an observable effect, can it really be called a double blind review?
- velcrovan 2y ago
  Maybe more that a non-idealistic model of the real world, and common direct experience, show that incentives strongly favor an administrative approach that compromises the double blind.
  - 0_____0 2y ago
    Unless there's a better way to do it, I this shows a need for better structures for governance and auditing of review boards... Information and science care not for our human folly, it's up to us to seek and execute them properly.

anjc 2y ago

Overly pessimistic, and doesn't acknowledge that heads of steam only build behind promising findings, while the deficient (or 'fraudulent') work die on the vine, published or not. In other words the system tends to work. Secondly, there are many ingredients required to successfully publish, communicate science, foster collaboration, etc., beyond technical brilliance. I'm sure we all know many technically brilliant people whose career never advanced because they lacked in some necessary area. People shouldn't be discouraged from improving in all areas because OP's delicate genius is offended by their technical ability. Speaking of discouragement, it's a shame and a disgrace that you publicly called your colleague's work bullshit, including a first author that isn't yourself.

bluefirebrand 2y ago
> Overly pessimistic, and doesn't acknowledge that heads of steam only build behind promising findings, while the deficient (or 'fraudulent') work die on the vine, published or not. This might be true in hard sciences where a "head of steam" can only build based on real, replicatable results But it's very common that public policy is proposed and adopted based on findings from soft sciences like psychology and sociology If policy is adopted based on a research paper, I would count that as a "head of steam" being built. And if that paper is fraudulent, then we are adopting well-intentioned policy on false pretenses
- anjc 2y ago
  I did just mean AI and Computer Science per OP. By "head of steam" I mean to say that much research is built on it, think the likes of "Attention is All you Need". There isn't quite an equivalent of this in public policy in my experience. Conversely, computer science/AI doesn't have an equivalent of the rigor that public policy research tends to go through. CS has e.g., benchmark datasets, typical evaluation metrics, but these are more like norms rather than requirements, whereas in public policy, instruments for validations are far more rigorously tested and enforced. Depending on the area. I agree that outright fraud would be detrimental, but I think OP overblows this issue completely and should apologise to his co-authors.

jillesvangurp 2y ago

The academic world is essentially the world's oldest social network. The way academic publishing works is through a convoluted reputation system of academics endorsing each other's works via various publications, giving each other a 'like' by referring work, debating work in public at conferences, hiring each other's students, protege's, etc. Fraud here basically means faking reputation. There are many ways to do this. And it's common because doing scientific work goes hand in hand with very generous funding. And money corrupts things. So attempts to fake reputation, plagiarize work, artificially boost relevance through low reputable referencing, bribes, etc. are as old as scientific publishing is. There are a few interesting dynamics here that counter this: high quality publications will want to defend their reputation. E.g. Nature retracting an article tends to be scandalous. They do it to preserve their reputation. And it tends to be bad for the reputation of affected authors. Their reputation is based on them having very high standards and a long history of important people publishing important things that changed the world. Every time they publish something, that's the reputation that is at stake. So, they are strict. And they should be. The problem is all the second and third rate researchers that make up most of the scientific community. We don't all get to have Einstein level reputations. And things are fiercely competitive at the bottom. And if you have no reputation, sacrificing it is a small price to pay. Also the prestigious publications are guarded by an elitist, highly political, in-crowd. We're talking big money here. And money corrupts. So, this works both ways. With AI thrown in the mix, the academic world has to up its game. And the tools it is going to have to use are essentially the same used in other social networks. Bluesky, Twitter, etc. have exactly the same problem as scientific publishers; but at a much larger scale. They want to surface reputable stuff and filter out all the AI generated misinformation and it's an arms race. One solution is using more AI or trying other clever tricks. A simpler solution is tying reputations to digital signatures. Scientific work is not anonymous. You literally stake your reputation by tying your (good) name to a publication and going on the record by "publishing" something. Digital signatures add some strength to that that AIs can't fake or forge. Either you said it and signed it; or you didn't. Easy to verify. And either you are reputable, by having your signature associated with a lot of reputable publications, or you are not. Also easy to verify. If disreputable stuff gets flagged, you simply scrutinize all the authors and publications involved and let them sort out their reputations by taking appropriate actions (firing people, withdrawing articles, publicly apologizing, etc.). They'll all be eager to restore their reputations so that should be uncontroversial. Or they don't and lose their reputation. Digital signatures are a severely underused tool currently. We've had access to those for half a century or so. The challenge isn't technical but institutional. Lots of disreputable people and institutions are currently making a lot of money by operating in the shadows. The tools are there to fix this. But people don't seem to necessarily want to.

nicwilson 2y ago

Hmmm, I wonder if you could turn this into a sport and have like one paper per year per group of total BS, and shame on the reviewers/conference/journal if they don't catch it, and kudos to the submitters the more blatant it is. Come to think of it, is there a "Journal of Academic Fraud"?

michaelt 2y ago

> And we must ensure that explicit suggestions to modify one’s science in the service of one’s career – “you need to do X to be published”, “you need to publish Y to graduate”, “you need to avoid criticizing Z to get hired” – carry social penalties as severe as a suggestion of plagiarism or fraud. One of the pernicious things in this area is that, even as we teach young researchers how to avoid making mistakes and engage sceptically with the work of others and that scientific fraud is a nontrivial issue, we also tell them how to commit fraud themselves and that their competition is doing it. "Watch out for P-hacking, that's where the researcher uses a form of analysis that has a small chance of a false positive, and analyses loads of subsets of your dataset until a false positive arises and just publishes that one" "Watch out for over-fitting to benchmarks, like a car taking the speed crown by sacrificing the ability to corner" "Watch out for incomplete descriptions of test setups, like testing on a 'continent-scale map' but not mentioning how detailed a map it was" "Watch out for citations where the cited paper doesn't say what is claimed, some people will copy-and-paste citations without reading the source paper" "Watch out for papers using complicated notation, fancy equations and jargon to make you feel this looks like a 'proper' paper" "Watch out for deceptive choice of accurate numbers, like a study with a 25% completion rate including the drop-outs in the number of participants" "Watch out for simulations with inaccurate noise models, if the noise is gaussian in the simulation but a random walk in reality, great simulated results won't transfer to reality" I've made no suggestion at all that you should modify your science or commit fraud - but I've also just trained you in how to do it.

proto-n 2y ago
It's really not that hard to come up with ways to commit fraud if you want to. On the other hand, it's very easy to make such mistakes if you don't know to avoid them. This characterization is very misguided IMO.
- michaelt 2y ago
  Ah, perhaps I wasn't clear about what I'm trying to say. I don't think we should stop training researchers in common mistakes and fraudulent methods to watch out for. I'm just saying: I don't believe anyone actually tells budding researchers that they should commit fraud. Instead I think the process is probably more like this: Year 1: Statistics/research training. Here are a load of subtle mistakes to watch out for and avoid. Scientific fraud happens sometimes. Don't do it, it's very dishonest. Year 2: Starting research. Gee a lot of these papers I'm reading are hard to reproduce, or unclear. Maybe fraud is widespread - or maybe they're just smarter or better equipped than me. Year 3: "You really ought to have published some papers by now, the average student in your position has 3 papers. If you don't want to flunk out you really need to start showing some progress"

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

ngriffiths 2y ago

I had two experiences at the polar opposites of the spectrum - one research team I worked on had very high standards and was comfortable being patient for material that had value. The other involved an approach that obviously stood no chance to be useful to anyone. Some differences: - The first one was in a space with more low hanging fruit - The first one was after large effect sizes, not the kind where you can massage the statistical model - The second one was a topic with far higher public interest - The second one was primarily an analytic project, whereas the first one was primarily experimental I feel like bad science lives in the middle of a spectrum - you have young fields/subfields with boring but impressive experimental breakthroughs, and on the other end you have highly political questions that have been argued to death without resolution. Bad science is about borrowing some of the strategies used in politics because all the important experiments have already been done.

throw4847285 2y ago

Unfortunately, I think this article is not pessimistic enough. If it were true that academic fraud is important for establishing the boundaries of respectable scholarship in a field, then behavioral economics wouldn't exist anymore. And evolutionary psychology would be a fraction of its current size. Trendiness trumps all notions of academic rigor, and as long as a field "feels like" it's on the cutting edge it can go pretty far before collapsing in on itself.

daveguy 2y ago
I'm wondering how someone knows the size of the field of evolutionary psychology and what it should be instead of what it is so well that they feel no evidence is necessary. Are you in the field yourself? Or do you have questions about evolution in general? Or none of the above?
- tovej 2y ago
  Evolutionary psychology is highly speculative and is widely known to be full of questionable results. So much so that there's a wikipedia page for how bad it is: https://en.m.wikipedia.org/wiki/Criticism_of_evolutionary_psychology https://en.m.wikipedia.org/wiki/Criticism_of_evolutionary_ps...
  - throw4847285 2y ago
    The SEP page is a slightly more rigorous take, though unlike Wikipedia it is clearly primarily authored by one person, a philosopher of biology who I don't know much about. Still, I think it's a fair and detailed overview of the field, from a methodological perspective. https://plato.stanford.edu/entries/evolutionary-psychology/ https://plato.stanford.edu/entries/evolutionary-psychology/
- throw4847285 2y ago
  Just cut out everybody who is studying "sexual selection" and I think the field is way healthier. Weird that so many evo psych scholars of a certain persuasion are fixated on that. I have a theory, but it veers into armchair non-evolutionary psychology, and it's not very nice.

jimbokun 2y ago

> Most researchers are to some extent “career researchers”, motivated by the power and prestige that rewards those who excel in the academic system, rather than idealistic pursuit of scientific truth. This is the funny part. There is little to no power and prestige to be had in the academic system. To a first approximation no one outside academia cares. I was just working as a staff programmer and taking grad courses with my tuition benefit, and found myself getting caught up in the mentality of needing a PhD to really be successful and valuable. Then got a job in industry making far more money and realized how academia is a small self contained world with status hierarchies irrelevant outside that small world.

vonneumannstan 2y ago
>This is the funny part. There is little to no power and prestige to be had in the academic system. To a first approximation no one outside academia cares. They have power over their students and relative power over other Professors. That's plenty enough incentive for most. There can also be fame and fortune for the most famous among them. See Francesa Gino, Dan Ariely, etc.
BeetleB 2y ago
In the social sciences, there is a lot of prestige to be had. Do some ground breaking work, engage with the public via bestselling books, and then get invited by the president to work on policy. Even if they don't work for the administration, there are plenty of other bodies that will value them and pay large sums of money (or let them have large influence). Very common amongst economists, and more and more common amongst disciplines like psychology. Even in technical fields, if you can manage to become a big name, you can do consulting work and get paid quite well. > Then got a job in industry making far more money This is not a healthy way to look at it. The average mechanical engineer isn't making tons of money in industry. A ME professor at a top university likely makes more. A biology major with just a BS degree will make less than the average biology associate professor. But more importantly, there's a simpler reason why money is a poor metric to measure: You can always get more money in finance or medicine than as a mechanical engineer. Does it make sense to denigrate a whole profession just because one can make more money elsewhere?

jarbus 2y ago

I don't have top-tier publications, and I haven't gotten any awards. I've seen people get awards for bullshit and farming prestigous publications. I only have one citation for work that is 100% mine. But every time I read something like this, I feel proud that I don't bullshit, and at least try to do real science. I truly believe in all my work and every sentence I write. That being said, the insane emphasis on venue is what's pushing me out of academia. I can't compete with people like this.

tmaly 2y ago
I am just thinking about what will happen when these papers are used to train LLMs