10 ms·
Academic success is either a crapshoot or a scam
- mjfl 9y agoI would say that social science is an exception rather than the rule when it comes to reliance on p-values for their results. Many, or most scientific fields don't. For example DNA nanotechnology gets published in Nature a lot by creating concrete proof-of-concepts where the "result" is a picture on an atomic force microscope or something like that. Other examples abound, crystallography, microscopy. So saying that academic success is a crapshoot or a scam in general because of p-hacking is not correct. Within social science it is probably correct.
- goldenkey 9y agoI have to agree. The social sciences are one place where nonsense flies free and often with wings. But even in the technical sciences, many papers don't list p-values or anything remoteley quantitative. I think we need to be careful of taking the part for the whole.
- TeMPOraL 9y agoJudging by the replication crisis discussions over past two years, I'd add quite a bit of medicine to that list too. And I don't mean just psychology, which is social science with a medical degree - but also drug research. All of this, though, shows how hard those disciplines are. Relative to that, hard sciences are easy, because they study precisely defined things pretty closely entangled with laws of physics, and you can't easily bullshit mother nature. But as you move away from physics, complexity rapidly increases, the questions themselves become fuzzier, and the answers less clear. So IMO, the probability that academic success is crapshoot or scam increases with (Kolmogorov) complexity of the subject domain.
- davrosthedalek 9y agoIn the case of at least medicine, there is another degree of hardness involved: sample size. While in physics, for example, sample size is limited by cost, space or, in few cases, frequency of occurrence, in medicine, a lot of studies are limited in size because of ethics. Many studies, especially drug trials, have an risk for at least one group. It is unethical to put more people at risk then absolutely needed, so studies are normally designed so that signal-to-noise is the minimal viable.
- ekianjo 9y agonot jusT ethics. Cost is a huge factor as well for clinical trials.
- whatshisface 9y agoCost is a factor for sample size in any field: but physicists don't publish if they can't accumulate a sufficient sample. As a result physics experiments where sufficient sample sizes can't be obtained are, simply, never performed - and as a result physics as a field grows in the direction of high-certianty. Physicsts absolutely could be performing experiments with poorer statistics; it'd be cheaper, more advanced things would be done sooner, and we'd be having a replication crisis. If the social sciences were willing to go a century without discovering anything exciting, they could achieve whatever level of certianty they wanted on their current budget. (At the expense of new discoveries.)
- magicalist 9y ago> but physicists don't publish if they can't accumulate a sufficient sample What? I would say most published physics is speculative models based on under-sampled phenomena (and that's ignoring the whole explicitly theoretical branches...). You could argue the practitioners tend to be more realistic about the chance of their models being valid, but this comment still ends up a bit masturbatory.
- mathattack 9y agoHigh certainty but that comes with second order issues: many ideas are untested, and the ideas that are turn into papers with 100s of authors. This stifles independent thinking.
- whatshisface 9y agoHere's a modern experimental paper. Scroll down to the authors. You'll know it when you see it. https://cds.cern.ch/record/2282544/files/arXiv:1709.05260.pdf https://cds.cern.ch/record/2282544/files/arXiv:1709.05260.pd... Your statement about huge numbers of authors (as well as the great number of theoretical papers that have to wait a very long time before being confirmed or denied) is true.
- Retric 9y agoIt's not quite that medicine is more complex, it's that Medicine is practical and accepts not understanding the details. If drug X works then that's enough we will use it even if we don't understand exactly why it works. The downside is we end up building simplified models well Y is correlated with bad things so we should treat Y as a bad thing. Often this actually works really well, but also opens the floodgates for all sorts of issues. That's not to say a less practical approach that simply looked for the details would be more effective. But, it would look very different.
- denzil_correa 9y ago> It's not quite that medicine is more complex, it's that Medicine is practical and accepts not understanding the details. If drug X works then that's enough we will use it even if we don't understand exactly why it works. I'm always baffled by this. I can understand why we should move these drugs into practice. But, if the way you conduct research stops at X works - you should have a good look at the way you conduct research. I can completely understand short term gains but ignoring long term gains is very unscientific and immature.
- zdragnar 9y agoI'm afflicted by three chronic conditions which have no known causes. All of them are treatable; there are guesses as to why the treatments work, there isn't really a good way to establish a sound reason for them working. You can't just take a person apart to study them while they're alive, and you can't easily reconfigure them to account for variables either.
- sliverstorm 9y agoThey'd love to understand why X works, but a whole lot of people could be treated by X while they spend all that time and money figuring it out. Besides, there's plenty of Y where they were sure it was going to work, it fit all the models and theories, it did the right thing in mice, and then in a trial Y fails catastrophically.
- clairity 9y ago
- gboudrias 9y ago> And I don't mean just psychology, which is social science with a medical degree Psychologists don't have a medical degree, psychiatrists do and they could hardly be called social scientists. Hard sciences also don't have a very high replication rate, they have the same incentives which the article talks about, and the same publishing bias problem.
- xapata 9y agoPart of the replication issue is that even if the relationship exists, it's quite likely a small study will find nothing.
- JorgeGT 9y agoI agree, I publish in the engineering field and in the field most "proofs" are validation by simple comparison against experimental data, DNS numerical simulation, analytical expressions, etc. In some areas there isn't even validation at all, it's just "we put this interesting thing in the expensive and exclusive machine, and these are the results you guys". For instance testing different fuel mixtures in a rapid compression machine, testing prototypes in a wind tunnel/flow rig/anechoic chamber, etc.
- skummetmaelk 9y agoThat's actually pretty good. There is a movement towards publishing minimal results, such as those experiments. This will hopefully get rid of the need to publish only successful experiments.
- pesenti 9y agoIt would be interesting to see the correlation between how susceptible a domain is to p-hacking and the number of papers required to be successful. I know that in Mathematics it’s ok to publish a good paper every other year.
- hacknat 9y agoI think your view is a little biased by your work. Much of biological research still relies on statistical methods. The Higgs boson, most of patho-biology, drug research, are just a few fields/discoveries I know for sure rely heavily on statistical models.
- learnstats2 9y agoNo, this is attractive but plain wrong. If you look at scientific research in Nature, there are very few papers that don't mention p-values. Successfully building your biological proof-of-concept is lovely and remarkable but depends on a whole bunch of research done with p-values.
- Cyph0n 9y agoNo, he is not wrong. There are many fields where p-values are irrelevant. For instance, in the fields of electrical engineering and computer science, you can "prove" your results by either building/fabricating a prototype or writing a piece of software.
- avian 9y agoTrue. Often authors don't even share the said piece of software or detailed plans for the prototype.
- Godel_unicode 9y agoI wonder how true that is in data science, if you're dealing with a non-deterministic algorithm. You're going to have cases for which your magic NN doesn't recognize the cat.
- learnstats2 9y agoBack to this thread to note this method of "proving" results by existence is working great for chip manufacturers right now.
- bo1024 9y agoBasically, you are talking about "engineering" instead of "science". Not that I'm trying to start a long discussion about definitions or mutual exclusivity, but these kind of "results" are clearly about building something new and showing it works, not proposing a hypothesis about how the natural world works and testing that hypothesis.
- nonbel 9y ago>"proposing a hypothesis about how the natural world works and testing that hypothesis" The studies under question here don't really do that. Physicists often test their hypothesis (eg this satellite should undergo x seconds of time dilation), but in fields like biomedicine or psychology that is extremely rare. Instead they usually test a "null hypothesis" that there is no difference (eg the population mean of the treatment and control groups is the same) when their hypothesis predicts some vague but non-zero difference.
- bo1024 9y agoThe difference between testing a null hypothesis vs testing a new, proposed hypothesis does not affect my point about scientific method vs engineering.
- nonbel 9y agoSure, but I'd say there are three things going on here though. 1) Scientific method (what you originally described) 2) NHST (where some default hypothesis is tested) 3) Engineering (where you perform some kind of feat to demonstrate capability) The problems seem to be with people doing #2 rather than the others.
- bjourne 9y agoThe problem exists in computer science too. For example, a researcher might publish an article about a novel algorithm, claiming to improve Java VM performance by X%. Except there is only pseudo code given, so replicating the result is nigh impossible. But if you succeed in implementing the algorithm, more often than not you discover that it only improves performance in the researchers specific setup and is net negative for normal workloads.
- collyw 9y agoAs someone who has worked in biology labs doing databases for them, it's about the only statistical test I could name name as it is talked about a lot.
- jondubois 9y agoIt's the same with everything else. For example, with complex software projects, the ones that make the most incredible claims about their capabilities are much more likely to get hyped up and get blogged/tweeted about... And it's often at the expense of real quality.
- goldenkey 9y agoReal[ity] criticism definitely takes a hit when a craft/hobby becomes popularized such that juicy reviews aggrandize. A vicious cycle forms, such that the projects with the most incredible claims get debunked by popular people -- which causes more people to check out the project and use it. Rinse and repeat. One of the least bad things is that for open-source projects, usually even if a project is built badly before it attains it's five minutes of fame, newly minted contributors often turn the ship around if the original architects have a shred of humility. It's not all bad...
- gaius 9y agoMethodology merchants like “Uncle Bob” have built careers and even companies on that strategy
- goldenkey 9y agoOne has to wonder whether any of the qualitative social sciences really even have a right to soundblast about concepts that have no provable metric...no way to be tested, and cause no difference in the world whether they are right or wrong. Is it time for the social sciences to throw in the ringer and reform their teaching and relevant arguments into a more mathematical/axiomatic foothold? I for one think so.
- danieltillett 9y agoThis is the neutron bomb solution. I have to say I am sympathetic.
- theoh 9y agoAargh. Life doesn't proceed by proof. Not all that counts can be counted. William James (paraphrased): "James acknowledges that in our scientific age, there is something dubious about the voluntaristic view that, in some circumstances, we can legitimately choose to believe in the absence of any objective justification. However, he claims we naturally do so all the time, our moral and political ideas being obvious examples. When you believe that your mother loves you or in the sincerity of your best friend, you have no conclusively objective evidence. In addition, you will never be able to secure such evidence. Yet it often seems unreasonable to refuse to commit to believing such matters; if we did so, the pragmatic consequences would be a more impoverished social life. Indeed, in some cases, believing and acting on that belief can help increase the chances of the belief being true." http://www.iep.utm.edu/james-o/#SH4a http://www.iep.utm.edu/james-o/#SH4a
- zAy0LfpBZLC8mAC 9y ago> Life doesn't proceed by proof. Please distinguish proof and evidence. > Not all that counts can be counted. How do you know that? > However, he claims we naturally do so all the time, our moral and political ideas being obvious examples. Yeah, and so incredibly successful, too, how people believe that black people are inferior or homosexuality is an abomination, without any interest in objective evidence. > When you believe that your mother loves you or in the sincerity of your best friend, you have no conclusively objective evidence. Yeah, it's absolutely impossible to know whether your mother or your best friend is objectively treating you well or terribly, so what can you do? You just have to assume that they love you! > In addition, you will never be able to secure such evidence. WTF? Next you tell me it's absolutely impossible to know whether animals are aggressive and you will never be able to secure evidence? > Yet it often seems unreasonable to refuse to commit to believing such matters; if we did so, the pragmatic consequences would be a more impoverished social life. WTF?! Yeah, believing and acting as if someone loves you without any evidence is so socially enriching there is even a term for it: It's called stalking.
- knolan 9y ago> In my field of anthropology, the minimum acceptable number of pubs per year for a researcher with aspirations for tenure and promotion is about three. This means that, each year, I must discover three important new things about the world. This is typically not the case. Academics will have a number of students working for them and many academics push them incredibly hard while doing very little of the actual research themselves. Instead they spend their time chasing funding. Projects are often built on prior work so many new papers explore new areas of a previous discovery or measure said discovery in a new way. So you’re not discovering three new things every year. Some academics will even be bold enough to publish the same data in multiple journals with minor changes in analysis. With academic research it’s often a case of two steps forward one step back. You spend years working towards deep expertise in a field through postgraduate studies and then a postdoc or two. Then when you become academic staff you suddenly become an inexperienced manager and teacher. So you pass the work to inexperienced students who spend years relearning everything and often suffering from over aggressive supervision or complete abandonment. The funding bodies, as is their right, are increasingly looking for commercial outcomes to research so the scope moves from Research to r&D with a little ‘r’. So we see research on battery technology making ludicrous claims or graphene in all the things rather than real progress. This stunts the potential of scientific discovery in favour of pushing revenue generating students through universities and chasing funding for short term gain.
- bachmeier 9y ago> This is typically not the case. What you're describing is research in fields with lots of grant funding. In some fields, you don't spend your time writing grants, and you don't hire grad students to do your research for you. In those fields, teaching is important because you do a lot of it. In grant-heavy fields, it wouldn't matter if the university shut down their undergraduate programs, because the emphasis would still be on bringing in grant dollars.
- kurthr 9y agoIt may be true that some University/College are interested in research and teaching more than the grant money that is brought in to the department, but there's definitely a bias there. Even more important, if you don't have several graduate students working for you during tenure (supplementing your research & teaching efforts), it's going to be hard to tenure there either. How many classes/labs can you teach without an RA/TA? Remember the big intro courses that new untenured profs get, and that teaching grads is a big part of the job! You basically have to prove you can't handle grad students in some horribly catastrophic way not to get them from the dean/provost. Perhaps in a field like philosophy or math at a very small school it's possible with amazing individual results, but it's uncommon in my experiences which include watching numerous friends at many different schools (>10) and fields (bio, chem, phys, geo, cs, math, philosophy) both public and private. Maybe you mean outside the US? I'd believe that. Europe hasn't gone as far down the path of non-tenured teaching faculty as the US, but due to money they seem to be starting to.
- Maro 9y agoOne possible, very sad long-term effect could be that the Sciences (results, theories) will be deemed less relevant, less trustworth, less worthy of attention. Eg. I started a Phd in Physics, but I never finished it, I got disillusioned with Academia and some specifics in today's Physics research (like String Theory, Inflation). 5 years later, people are still beating on those drums, and whenever I read about some crazy new model, I think two thoughts: (i) nonsense (ii) I'm so happy I'm not spending my time on this.
- fiatjaf 9y agoIt would be a very good outcome if society stopped believing anything that comes out of the mouth of a "scientist". It's just sophistry all the way down.
- zeth__ 9y agoThe problem in physics is that you hear about the highest high fliers of the discipline and assume most people will be something like them, if less inspired/gifted. Then you get there and find out most people are shitty, territorial and wages put you in the same income bracket as fast food workers.
- tensor 9y agoThe scary part is that if science is deemed not worthy of attention, then we enter a new dark age were no progress is made and no new understanding is gained. That's truly frightening.
- yipopov 9y agoI was once asked to edit a sociology paper for grammar, spelling and style, but ended up deleting large swaths of it containing claims made up of thin air and found nowhere in the data or the citations. Of course this was not well received and I expect every one of those passages were promptly restored, complete with their horrendous spelling and grammar. The claim? That the biggest challenge facing remote poverty- and alcoholism-stricken communities of indigenous peoples were not enough tablets to be able to "Netflix and chill". Because surely that will solve all the world's problems.
- tome 9y agoWould be good if you could substantiate your comment by linking us to the published version.
- deleted 9y ago[deleted]
- yipopov 9y agoIf you don't mind I'd rather not have my name linked with fraudulent academic articles, regardless of who exactly the guilty party is. I could have just said "What the linked article says is correct", but I figured this anecdote would add more value. But yeah, whether you want to believe it or not, the linked article is pretty accurate. Successful academics either get lucky or they are frauds.
- shard972 9y agoYea im going to need that paper.
- Hendrikto 9y ago> empirical social scientists with lots of pubs in prestigious journals are either very lucky, or they are p-hacking. That’s the money quote. This is mainly applicable to “empirical social science”, which many already know as a big scam.
- jgmjgm 9y agoAbsolutely. Been there, done that. I worked for a guy who is top in his social science field. Total p-hacking was the norm. It was an eye-opening experience for someone new to the field expecting things to be done more honestly following something like the scientific method. Once I found a major bug in some code that was the basis of a significant chunk of his major book and an important article. (Basically an optimization process that didn't work but gave some nice results he built a story around.) When I wrote him a report on what was wrong and why, he didn't respond. I was supposed to write a paper based on the same code and all he would say was along the lines of "it should work". Over time I realized that the implication was that I should "make" it work. Of course, he never said those words but over time the message was clear. After beating my head against a wall for a long time, I eventually walked away from the project. Technically bad for my career, but I honestly couldn't see the point in being at a top university essentially selling snake oil. That being said, there are fields of social science that are much better: demography, political science, economics and even psychology (which is very difficult at the best of times!) are trying to clean up their act. Basically, any field of study with consistent data available openly is probably pretty decent. There are other fields like business/management where the pay is generally good (i.e. motivation for a nice job with good perks is there), many of these journals don't require data to be released, and the samples being studied tend to be convenience samples or impossible to reproduce. I generally don't trust the work in this area. Sadly, policy makers are often willing to take questionable academic studies that support their preconceived notions. When these things don't work out, the general public becomes jaded that their taxes were wasted... it makes all social science research look bad.
- sytelus 9y agoThis article is about "empirical research" which means articles with titles like "Study suggests XYZ". I avoid these type of "research" articles like plague. There probably exists a "study" that proves and disproves virtually anything. For example, study suggests coffee prevents cancer and vice versa. Such pseudo (and often badly done) statistical voting type of work shouldn't even be called "research" and people doing these shouldn't be called "scientists" (the word "surveyors" seems more appropriate). There is now a cottage industry of pushing out absurd correlations that would be immediately be picked up by media because, you know, so called "scientists" did that.
- ChrisSD 9y agoWhile I understand your broad point, I don't think it's helpful to conflate the study itself with the media reports of said study.
- erikpukinskis 9y agoI would be interested in your point if you were revealing something concrete about the OP and their research, but you’re just making handwavy claims about broad correlations (“empirical research” = “study suggests xyz”) The fact that you’re making overly broad claims accusing people of making overly broad claims doesn’t instill confidence that you know what you’re saying. If you’re such a champion of science, why not state your claims minimally concrete terms. And ideally make falsifiable claims.
- deleted 9y ago[deleted]
- sytelus 9y agoOP's point is that its hard to get funds to conduct 9 studies per year with hope that 3 will result in paper as expected from the tenure track. I neither have intention to counter argument nor I have insights in to specific field that author is working on (which seems to be anthropology). My argument is exactly more broader as you have mentioned: Lot of research - at least as captured by media - involves simply doing some statistical survey and getting comfortable p-value to suggest some weird correlation. There is a lot of money spent in this type of "research" because it is expensive to conduct such studies. My objection purely on calling this "science" and calling people doing this as "scientists" because these two fairly reputable words are being used to lure general public to believe such "studies". I feel such works should be called "surveys" and people doing it should be called "surveyors". The real scientific work should involve taking the survey only as a hint and then uncover actual mechanism that falsifiably proves or disproves the correlation with hopefully more predictions. Simple surveys don't have these properties and hence such studies shouldn't be labeled as work of scientists.
- fiatjaf 9y agoIf everybody is required to publish tons of stuff, you're just creating tons of stupid stuff no one will read, and that crap is going to cover the really important stuff.
- fiatjaf 9y agoIf you're going to reward only high-quality studies you'll lose the larger part of scientists, because there are very little of them who can produce anything high-quality. What you're proposing is mass unemployment.
- haZard_OS 9y agoPerhaps but such changes, if they occur at all, would likely be gradual and so shifts in the structure of academia would cushion the fall. In any case, if we can't expect and even demand high-quality research from professional scientists, I fear we are doomed as a species.
- stordoff 9y agoIf the studies they are producing aren't high-quality, is there any value to them? Do we, as a whole (because some individuals will lose out, as you say), benefit by rewarding low standards of work? I'd also said that many may be _capable_ of high-quality work, but merely have limited incentives to produce it currently.
- tbarbugli 9y agoYears ago after spending uncountable beers listening to the frustration of several PhDs friends, I had the idea of a journal for research failures/dead-ends. The idea was to look at unfruitful research as something useful for the community and give some credit to time and resources spent doing something that did not work (and should not repeated by others). I still wonder if that would create the wrong incentive though and if that could be feasible at all (who likes to talk about money and time wasted?) EDIT: fixed wrong sentence
- chrisseaton 9y ago> I had the idea of a journal for research failures/dead-ends A bit like the Journal of Negative Results in Biomedicine, the Journal of Negative Results — Ecology and Evolutionary Biology and the psychology Journal of Articles in Support of the Null Hypothesis? > several phd undergraduate friends People doing PhDs who are simultaneously undergraduates?
- hannob 9y ago> I had the idea of a journal for research failures/dead-ends. The problem is the following: If you seriously want to fix science you don't need one such journal. It would have to be the norm. It's simply not plausible that the majority of empirical research leads to positive effects. Even if you publish a few negative results (or "failures") - that doesn't even begin to address the problem of publication bias seriously. As long as the majority of scientific publications are not "failures" we should assume that the majority are false positives due to p-hacking or publication bias.
- whack 9y agoOne of the things I find surprising about the way we do science today, is the idea that "data is king" and that it "speaks for itself". In reality, data devoid of context, is absolutely meaningless. If the goal of scientific researchers and journals is to promote good science, it's absolutely shocking that practices such as pre-registration of hypothesis and methodologies, are still completely optional. https://wp.me/p3SyBR-8t https://wp.me/p3SyBR-8t
- make3 9y agoI'd be surprised if anyone is really arguing that the context of data is not important.
- whack 9y agoAnd yet, practices like pre-registration of hypotheses are still not widespread. If they did, it would eliminate so much of the cherry-picking of data that occurs. https://xkcd.com/882/ https://xkcd.com/882/
- Blazespinnaker 9y agoI thought quality of journal and number of citations (from quality journals) was more important than pubs per year.
- lewis500 9y agoAll of my research so far has been either theory/simulation or descriptive (e.g., this is how much this project did cost). After going to a lecture on someone's empirical research two weeks ago, I found myself asking why I have always been kind of turned off by statistical projects. I came up with two answers: One: at this state in my career I need to publish a lot. There is kind of a speed limit on how fast you can publish empirical work, but with theory the upper bound on your publication rate is how hard you work and how creative you are. Two: I feel more honest about my theoretical work. My models and simulations are so obviously contrived that there is no danger of anyone mistaking the map for the territory. By contrast, if you come up with some effect of size X in an empirical paper it might be reported in the media as some kind of golden truth like the charge of an electron. I think if I spent like 12 or 15 months on an empirical project, I also would be so desperate to get a result out of it as to fudge the result. Even among intuitive relationships, the chance you can honestly and provably identify an interesting relationship is really low. For this reason, I read theory and description from all kinds of researchers, but I tend to only be interested in the empirical work from the best-of-the-best. In my field, I think some of the best work is from Rebecca Diamond at Stanford, and one of her most recent papers had a negative result. I also really discount job market papers: these people spent years on a single project and their whole future is riding on finding something cool, so the incentive to fudge is just extraordinary. It's kind of like trusting a judge to try a case in which his son is the defendant. No matter what he told me about how fair he'll be, I wouldn't believe him.
- paultopia 9y agoThis rings really true to me. I'm switching to more empirical-ish work now, after tenure when I don't have a knife at my throat.
- averagewall 9y agoA bit off the topic but I always wondered why social scientists need funding or even a university for their experiments. I mean the kind where they put 20 people in a room and show them some pictures and ask them some questions. What resources do you need?: Time - do it as a hobby so it's limited to people who are passionate about discovering something, not just trying to rack up points. Access to human subjects - get people off the street. Universities can make access to student subjects easier but they're not representative of the world's people anyway so it's not that valuable. Money to pay the human subjects - save up? People who're passionate about cars, for example, somehow find the money to build modified cars and don't need funding. Ability - With the oversupply of researchers, maybe just leave it to those who can do it under their own steam? Hopefully undergrad degrees taught them enough to get started. This article says that university researchers aren't doing it with skill anyway but with luck or cheating. Motivation - if nobody can be bothered investigating some effect, then maybe it's not really that interesting anyway and there's not much value in studying it? What am I missing?
- abecedarius 9y agoSee http://slatestarcodex.com/2017/08/29/my-irb-nightmare/ http://slatestarcodex.com/2017/08/29/my-irb-nightmare/ for one relevant story. On the other hand, outside of institutions there are cases like https://www.gwern.net/zeo/Vitamin-D https://www.gwern.net/zeo/Vitamin-D and I hope there'll be more.
- shard972 9y agoMoney to pay their own salaries so they can do it full time? Not trying to be a complete asshole but seems like the most logical conclusion for someone with a SS degree which can't help you get a job outside of things like this.
- jgmjgm 9y agoDepends what you study. If you're an economist you need access to data like compustat or other. If you're a political scientist in the US much of the data is available but you also need to spend a while cleaning it (although there are great resources now too.) You're really talking about small scale interviews or qualitative work. That's an aspect of social science but it's not all of it.