5 ms·

> “paper slicing” I'm glad we agree that paper slicing is also plaguing academia. When I was a grad student, it bugged me to no end that those guys at Stanfo

by rhelz 3y ago

> “paper slicing” I'm glad we agree that paper slicing is also plaguing academia. When I was a grad student, it bugged me to no end that those guys at Stanford were out-publishing me--because they got enough results for 1.5 papers, and then made (3 choose 2) papers out of them. And yeah, if you think I wasn't tempted to follow suit, think again. I eventually left academia because I didn't want to cheat, or compete with cheaters, when we were being graded on the curve of "publish or perish". > fraud is rampant and driving the explosion in publication rates is deeply tempting to folks who don’t work directly in a scientific field. So, you are an assistant prof who was passed over for tenure, or you didn't get that Harvard appointment--it went to Francesca Gino, whose vita looks sooooo much better than yours, because she's doing TED talks to promote her book called "Why It Pays To Break The Rules In Work And Life". She's making $1 million a year consulting, while you are trying to get funding for your research, which isn't as flashy but at least is real science... ... if you are graded on the curve, how will you look against someone who cheated? It's the prisoner's dilemma. > How is anyone going to replicate a result if it’s not published? In my proposal, it works the same that way can they review a paper if it's not published. You write up a paper, detailing the claims you are making and the experiments and methods you used to justify your claims, and you submit it for publication. Then, 3 or 4 anonymous reviewers decide whether it's promising enough to go to the next step: hire another research group to replicate the results. If the results replicate, then and only then are they published. Yes, it's more expensive, so to finance it I propose that when the grant is being written, the principle investigators should also estimate what it would cost to hire reviewers and to fund experiments at another lab to confirm the results. When the grant is approved, that money is put in escrow until the research is done and submitted for publication, and disbursed to the reviewers and replicators to compensate them for pausing their own research and evaluating someone else's. If it replicates, it's published. If it doesn't replicate, you don't get to write up books on how to cheat your way to the top while you are cheating your way to the top. > This is literally the reason that scientific publication exists ... 100% agree, its great to share new results and all---but if a "result" doesn't replicate, its not a result. What, exactly, are Gino's peers supposed to learn from her fraudulent papers? If I'm trying to decide what to research, how do I know which lines are actually promising or not? Do I just go with researchers at a name-brand university (like Gino at Harvard?) These days papers can be written by machine as fast a a machine gun fires bullets. There's got to be some way of separating the signs from the noise. > brownie-point credit for publications... Publication is very important; one of my professors explained it this way: if you don't publish your results--i.e. you don't convince your peers that what you did is worthy of publishing--its like you didn't do anything at all. Then whole idea of research is to contribute to the edifice of science, and publishing is the vehicle by which that contribution is made. It's how you deliver the contribution to everybody else. And peer-review is how it is determined whether you actually made a contribution. So the solution can't be to just stop caring about how many papers are published. > Taken literally, your “no publication without replication” proposal would inhibit scientific replication entirely. I hope my explanation above addresses this concern....

matthewdgreen 3y ago
"Publication" is the process of writing up works and distributing them to other researchers. Having fast and efficient channels for publishing our results is essential to the progress of science. I strongly object to any proposal that would interrupt this essential process. If you (somehow) prevented me from distributing my preliminary results to other scientists, or from reading their preliminary results, then I would actively work around your proposal. Not because I hate you, but because science fundamentally requires access to these new ideas. Moreover, any researchers who waited for replication would fall behind and get scooped by the better-connected Ted-talk giving folks, who would obviously keep exchanging results. However, it's obvious that when you say publication you don't mean it in the literal sense of communicating scientific results. Instead what you're trying to reform is academic credit. We all know that scientists [and the bureaucrats who fund them] love to find ways to measure research productivity, and they've chosen to use publications (in specific venues) as a proxy for that contribution. And, following Goodhart's law, any measure that becomes a target ceases to be a good measure. So the purpose of your proposal really has nothing to do with scientific publication for its first-order effects (i.e., distributing results to colleagues) but rather, you just want to reform the way that credit is allocated to scientists for purposes of funding, promotion, and general adulation (that's a joke.) My suggestion is: if you want to reform the way credit is allocated, why not focus explicitly on that goal? There's no need to put in place complicated new systems that will make it harder for anyone to publish (and that real scientists will instantly work around.) Instead, just go to the NSF and NEH and various university administrations and convince them to stop hiring/funding/firing researchers using the metrics they currently use? I think overall you'll find that this is still a tough sell, and a massive uphill battle. But it has the advantages of (1) addressing the actual problem you're concerned with, (2) not preventing researchers from doing their job, (3) has fewer overall points of failure, and (4) isn't subject to being worked around by funders/administrators who simply adopt new measures after you "reform" the publication system (e.g., switching to citation counts for arXiv works, or surveying other researchers for their opinion.)
- rhelz 3y ago
  > If you (somehow) prevented me from distributing my preliminary results Dude, nobody can prevent you from putting your paper on arxiv. But arxiv is a vanity press-I'm sorry to put it in such negative terms, but that's the God's honest truth of it. We have free speech; you can get a vanity press to print whatever you want, as fast as you want it to. But that is not doing science. Alas, arxiv is needed, because the rest of the system is so broken. But imagine if you could submit your paper to any journal, and be guaranteed that it would be peer reviewed within 2 weeks. We could do that if we didn't have to rely on volunteer labor, but paid qualified people to do the job thoroughly and on time. > Having fast and efficient channels for publishing our results Have you ever submitted a paper to a journal? If so, I'm sure you were as frustrated as I was that my paper was just sitting on reviewers desks for a year before they got around to giving it a cursory glance. If we actually payed reviewers, we could specify that the reviews must be done on a certain time schedule. My proposal would greatly accelerate the rate at which non-fraudulent, scientific results get published and communicated to other researchers. > I strongly object to any proposal that would interrupt this essential process. Its not an essential process to science to pick the fastest and cheapest vanity press. What essential for science is getting repeatable results. That's science. > Having fast and efficient channels for publishing Ever faster publishing of ever more vanity projects is not science, nor does it help science. Quite the opposite. > it's obvious that when you say publication you don't mean it in the literal sense of communicating scientific results. No, that's exactly what I mean. But putting something up on arxiv isn't "communicating scientific results." Until it has been peer reviewed and shown to be replicable, it just isn't a scientific result. > [change metric, etc] I don't want to change the metric. We don't have a bad metric--we have frauds claiming they have met the metric when they haven't. The problem isn't with the metric, its that the metric isn't actually being enforced like it should be. > any measure that becomes a target ceases to be a good measure. Peer reviews and replications are not a measure of how good your science is. The measure for good vs poor science are things like how useful your results are, how general they are, whether they unify previously disconnected areas of knowledge...things like that. Whether you are generating repeatable results or not isn't the difference between poor science and good science. It's the the difference between doing science and not doing science. You can't get rid of peer review, and the demand for repeatable results and still be doing science. Science is the the process of getting peer reviewed and repeatable results. Peer review and repeatability isn't how you judge the results of the race--they are the bare minimum requirements to enter the race to begin with. > your proposal really has nothing to do with scientific publication for its first-order effects (i.e., distributing results to colleagues) Distributing what results? If your "results" can't pass peer review, or they can't be replicated, the are not scientific results. If you skip those steps, you are are skipping doing science. You are not a scientist, you are just cosplaying a scientist. In order to deliver scientific results, quickly or slowly, you actually have to have produced scientific results. > if you want to reform the way credit is allocated, why not focus explicitly on that goal? Well, I'm not trying to reform the way credit is allocated. I'm trying to stop people from getting credit for fraudulent results. Before credit is allocated, something credible must have been produced!! And until your paper passes peer review, it is not credible, and until it has been replicated, its not science. > that real scientists will instantly work around. Even calling them "real scientists" betrays a deep conceptual error. There are not "real scientists" and "other kinds of scientists." There are scientists, and there are non-scientists. The distinction isn't between "real scientists" who take every shortcut and cheat as much as they can get away with, and "poor schlubby scientists" who don't have the guts to cheat. Scientists (not "real scientists", just scientists) insist on peer review and reproducible results. You can't "work around" the most basic criteria for something be scientific and still be a scientist doing science. > just .... convince them to stop hiring/funding/firing researchers using the metrics they currently use So...I'm supposed go to the national science foundation, and every body which funds scientific research, every institution which purports to hire scientists---and somehow convince them to stop actually doing science? sigh its not your fault man. The problem has been going on for so long now, that there are generations of cosplayer-professors, who have been graduating cosplayer-ph.d.s. Imagine people going to Star Trek conventions, dressing up, geeking out---but after few generations they forget that they are cosplaying, and think they are actually on a starship.... Seems ludicrous, but that's kind of what arxiv has inadvertently done. It doesn't help that have people who cosplay being "science" journalists or "science" popularizers, who trawl arxiv for juicy headlines, and happily write up an article about "research" which hasn't been peer reviewed or replicated. It just encourages more race-to-the bottom, by encouraging "researchers" to post exaggerated claims.

by rhelz 3y ago

matthewdgreen 3y ago

"Publication" is the process of writing up works and distributing them to other researchers. Having fast and efficient channels for publishing our results is essential to the progress of science. I strongly object to any proposal that would interrupt this essential process. If you (somehow) prevented me from distributing my preliminary results to other scientists, or from reading their preliminary results, then I would actively work around your proposal. Not because I hate you, but because science fundamentally requires access to these new ideas. Moreover, any researchers who waited for replication would fall behind and get scooped by the better-connected Ted-talk giving folks, who would obviously keep exchanging results. However, it's obvious that when you say publication you don't mean it in the literal sense of communicating scientific results. Instead what you're trying to reform is academic credit. We all know that scientists [and the bureaucrats who fund them] love to find ways to measure research productivity, and they've chosen to use publications (in specific venues) as a proxy for that contribution. And, following Goodhart's law, any measure that becomes a target ceases to be a good measure. So the purpose of your proposal really has nothing to do with scientific publication for its first-order effects (i.e., distributing results to colleagues) but rather, you just want to reform the way that credit is allocated to scientists for purposes of funding, promotion, and general adulation (that's a joke.) My suggestion is: if you want to reform the way credit is allocated, why not focus explicitly on that goal? There's no need to put in place complicated new systems that will make it harder for anyone to publish (and that real scientists will instantly work around.) Instead, just go to the NSF and NEH and various university administrations and convince them to stop hiring/funding/firing researchers using the metrics they currently use? I think overall you'll find that this is still a tough sell, and a massive uphill battle. But it has the advantages of (1) addressing the actual problem you're concerned with, (2) not preventing researchers from doing their job, (3) has fewer overall points of failure, and (4) isn't subject to being worked around by funders/administrators who simply adopt new measures after you "reform" the publication system (e.g., switching to citation counts for arXiv works, or surveying other researchers for their opinion.)

rhelz 3y ago
> If you (somehow) prevented me from distributing my preliminary results Dude, nobody can prevent you from putting your paper on arxiv. But arxiv is a vanity press-I'm sorry to put it in such negative terms, but that's the God's honest truth of it. We have free speech; you can get a vanity press to print whatever you want, as fast as you want it to. But that is not doing science. Alas, arxiv is needed, because the rest of the system is so broken. But imagine if you could submit your paper to any journal, and be guaranteed that it would be peer reviewed within 2 weeks. We could do that if we didn't have to rely on volunteer labor, but paid qualified people to do the job thoroughly and on time. > Having fast and efficient channels for publishing our results Have you ever submitted a paper to a journal? If so, I'm sure you were as frustrated as I was that my paper was just sitting on reviewers desks for a year before they got around to giving it a cursory glance. If we actually payed reviewers, we could specify that the reviews must be done on a certain time schedule. My proposal would greatly accelerate the rate at which non-fraudulent, scientific results get published and communicated to other researchers. > I strongly object to any proposal that would interrupt this essential process. Its not an essential process to science to pick the fastest and cheapest vanity press. What essential for science is getting repeatable results. That's science. > Having fast and efficient channels for publishing Ever faster publishing of ever more vanity projects is not science, nor does it help science. Quite the opposite. > it's obvious that when you say publication you don't mean it in the literal sense of communicating scientific results. No, that's exactly what I mean. But putting something up on arxiv isn't "communicating scientific results." Until it has been peer reviewed and shown to be replicable, it just isn't a scientific result. > [change metric, etc] I don't want to change the metric. We don't have a bad metric--we have frauds claiming they have met the metric when they haven't. The problem isn't with the metric, its that the metric isn't actually being enforced like it should be. > any measure that becomes a target ceases to be a good measure. Peer reviews and replications are not a measure of how good your science is. The measure for good vs poor science are things like how useful your results are, how general they are, whether they unify previously disconnected areas of knowledge...things like that. Whether you are generating repeatable results or not isn't the difference between poor science and good science. It's the the difference between doing science and not doing science. You can't get rid of peer review, and the demand for repeatable results and still be doing science. Science is the the process of getting peer reviewed and repeatable results. Peer review and repeatability isn't how you judge the results of the race--they are the bare minimum requirements to enter the race to begin with. > your proposal really has nothing to do with scientific publication for its first-order effects (i.e., distributing results to colleagues) Distributing what results? If your "results" can't pass peer review, or they can't be replicated, the are not scientific results. If you skip those steps, you are are skipping doing science. You are not a scientist, you are just cosplaying a scientist. In order to deliver scientific results, quickly or slowly, you actually have to have produced scientific results. > if you want to reform the way credit is allocated, why not focus explicitly on that goal? Well, I'm not trying to reform the way credit is allocated. I'm trying to stop people from getting credit for fraudulent results. Before credit is allocated, something credible must have been produced!! And until your paper passes peer review, it is not credible, and until it has been replicated, its not science. > that real scientists will instantly work around. Even calling them "real scientists" betrays a deep conceptual error. There are not "real scientists" and "other kinds of scientists." There are scientists, and there are non-scientists. The distinction isn't between "real scientists" who take every shortcut and cheat as much as they can get away with, and "poor schlubby scientists" who don't have the guts to cheat. Scientists (not "real scientists", just scientists) insist on peer review and reproducible results. You can't "work around" the most basic criteria for something be scientific and still be a scientist doing science. > just .... convince them to stop hiring/funding/firing researchers using the metrics they currently use So...I'm supposed go to the national science foundation, and every body which funds scientific research, every institution which purports to hire scientists---and somehow convince them to stop actually doing science? sigh its not your fault man. The problem has been going on for so long now, that there are generations of cosplayer-professors, who have been graduating cosplayer-ph.d.s. Imagine people going to Star Trek conventions, dressing up, geeking out---but after few generations they forget that they are cosplaying, and think they are actually on a starship.... Seems ludicrous, but that's kind of what arxiv has inadvertently done. It doesn't help that have people who cosplay being "science" journalists or "science" popularizers, who trawl arxiv for juicy headlines, and happily write up an article about "research" which hasn't been peer reviewed or replicated. It just encourages more race-to-the bottom, by encouraging "researchers" to post exaggerated claims.