4 ms·

You have the causality wrong. Prestige comes to journals by doing a good (or at least, better than peers) job of being a journal, which is providing a necessary

by ska 3y ago

You have the causality wrong. Prestige comes to journals by doing a good (or at least, better than peers) job of being a journal, which is providing a necessary function to the academic research process. If you want to improve on that system, you have to improve on those functions, or reduce the reliance on them by providing something better. Put it another way, if you can design a system with a better ROC curve for classifying research, with a better TP rate for good papers, and have it cost less in real terms that current academic papers, then you are on to something. If all you've got is "papers should be free" or "it's too hard to access publishing from the outside" what you have are complaints, not solutions.

staunton 3y ago
Like I said, it's not clear to me what exactly established journals have been doing "better" historically than a preprint server could. You say they are better at "being a journal" -- OK. They are established, well connected (to science communities, industry, journalism, funding agencies, etc.) and have been maintaining a reputation for a long time, usually much longer than preprint servers exist. That's basically prestige, which isn't nothing (I didn't claim prestige is nothing). However, this doesn't demonstrate journals do anything relevant to the advancement of science fundamentally better right now. What I "have" is that 1. It's not obvious that a journal is fundamentally better at organizing unpaid voluntary reviewers compared to a preprint server. 2. Scientific publishing has insanely high profit margins. How come? My theory is that they are selling prestige first and foremost, i.e., a luxury good, (to scientists, universities and funding agencies simultaneously) and purchasing decisions there are made by people who are spending public money, not their own. Both of these points (luxury good, public spending) seem like strong contributors to high margins. The public is paying for the research and for access to articles, while journals nowadays on first glance seem to only provide a little bit of web hosting, a little bit of reviewer coordination and a little bit of manuscript editing. 3. It's not obvious that the submission and peer review systems we have now (in journals) is worth the time and effort. The role of peer review is misrepresented in journalism and the expectations are not met. If one could separate publication ("preprint") on one side, and, on the other side, review and "being featured by important outlets or institutions", authors could save a lot of time time (that could be used for more research). Others would have access to interesting results earlier and be able to build on top of them. Next, in a separate process some institution could select important works, scrutinize and review them, perhaps paying experts to do so, and perhaps replicate where appropriate. The issue with this is that academics need the prestige provided by journals for career advancement, universities need the prestige to justify their spending to funding agencies and politicians, and funding agencies likewise need the prestige to justify their spending to politicians. The "replication crisis" and the like indicate that this prestige is overvalued. The hope is, economically speaking, that the market for "academic prestige" can either be disrupted, or the price the public has to pay can be lowered "through competition". It's interesting what that might look like. Preprint servers, open data and more direct science communication seem like steps in the right direction.
- rhelz 3y ago
  Your concerns are valid, and I think that my proposal of paying reviewers and replicators addresses them all. > It's not obvious that a journal is fundamentally better at organizing unpaid voluntary reviewers compared to a preprint server. So, let's not rely on unpaid, voluntary labor. Pay them. > they are selling prestige first and foremost, Yes. So give them a better business model--if they can make money reviewing papers, they won't have to create artificial value by creating artificial scarcity. > The role of peer review is misrepresented in journalism and the expectations are not met. If you pay somebody, you can specify the expectations you think should be met. If they don't meet those expectations, they don't get paid. > this prestige is overvalued. The prestige is not overvalued--it is just too easily obtainable by fraud. Something has got to be done. > Preprint servers, open data and more direct science communication seem like steps in the right direction. They are vanity presses. And they don't even do what you think they are doing. Today, the problem isn't too little information, it's too much misinformation. LLMs can chug out papers by the millions. Is a search engine going to help you cut through that and find what you are looking for? What if 2 million papers which match your search criteria? You gunna read through them all, trying to find the 5 papers which were actually written by a real scientist? Are you even going to see them? Is the search engine going to do a better job than peer review of presenting you the papers you actually want to read?
  - staunton 3y ago
    > my proposal of paying reviewers and replicators addresses [all concerns] Your other comment didn't say who might be paying reviewers. Journals clearly won't (why should they, they have grrat profits in the current system and will fight tooth and nail to delay any changes whatever). Universities and even funding agencies cannot (conflict of interest). > Is the search engine going to do a better job than peer review of presenting you the papers you actually want to read? I do actually expect to see that happen.
    - rhelz 3y ago
      > Your other comment didn't say who might be paying reviewers. In the parent comment to this thread, I talk about this. My proposal is that when a researcher writes up the grant to get their research funded, they should estimate how much it would cost to pay reviewers and replicators, and include those figures in the cost. If the researcher gets the grant approved, then the funding agency will put the money for peer review/replication into escrow. When the research is finished and the investigators have written up a paper to describe their methods and results, the money in escrow is disbursed to the reviewers and replicators. If reviewers agree its good, and if it replicates, then the paper is published. If not, well we just dodged a bullet. > I do actually expect to see that happen. Are search engines getting better or worse for you? It was a lot easier getting the right paper from a search engine 10 years ago. Now, you just get half a page of irrelevant ads, and another half page of links boosted by payola. Just imagine what it will be like when there are literally MILLIONS of bad papers for each good paper. Then Billions. There is no finite limit to the amount of bullshit that LLMs can--and therefore will--output.

by ska 3y ago

staunton 3y ago

Like I said, it's not clear to me what exactly established journals have been doing "better" historically than a preprint server could. You say they are better at "being a journal" -- OK. They are established, well connected (to science communities, industry, journalism, funding agencies, etc.) and have been maintaining a reputation for a long time, usually much longer than preprint servers exist. That's basically prestige, which isn't nothing (I didn't claim prestige is nothing). However, this doesn't demonstrate journals do anything relevant to the advancement of science fundamentally better right now. What I "have" is that 1. It's not obvious that a journal is fundamentally better at organizing unpaid voluntary reviewers compared to a preprint server. 2. Scientific publishing has insanely high profit margins. How come? My theory is that they are selling prestige first and foremost, i.e., a luxury good, (to scientists, universities and funding agencies simultaneously) and purchasing decisions there are made by people who are spending public money, not their own. Both of these points (luxury good, public spending) seem like strong contributors to high margins. The public is paying for the research and for access to articles, while journals nowadays on first glance seem to only provide a little bit of web hosting, a little bit of reviewer coordination and a little bit of manuscript editing. 3. It's not obvious that the submission and peer review systems we have now (in journals) is worth the time and effort. The role of peer review is misrepresented in journalism and the expectations are not met. If one could separate publication ("preprint") on one side, and, on the other side, review and "being featured by important outlets or institutions", authors could save a lot of time time (that could be used for more research). Others would have access to interesting results earlier and be able to build on top of them. Next, in a separate process some institution could select important works, scrutinize and review them, perhaps paying experts to do so, and perhaps replicate where appropriate. The issue with this is that academics need the prestige provided by journals for career advancement, universities need the prestige to justify their spending to funding agencies and politicians, and funding agencies likewise need the prestige to justify their spending to politicians. The "replication crisis" and the like indicate that this prestige is overvalued. The hope is, economically speaking, that the market for "academic prestige" can either be disrupted, or the price the public has to pay can be lowered "through competition". It's interesting what that might look like. Preprint servers, open data and more direct science communication seem like steps in the right direction.

rhelz 3y ago
Your concerns are valid, and I think that my proposal of paying reviewers and replicators addresses them all. > It's not obvious that a journal is fundamentally better at organizing unpaid voluntary reviewers compared to a preprint server. So, let's not rely on unpaid, voluntary labor. Pay them. > they are selling prestige first and foremost, Yes. So give them a better business model--if they can make money reviewing papers, they won't have to create artificial value by creating artificial scarcity. > The role of peer review is misrepresented in journalism and the expectations are not met. If you pay somebody, you can specify the expectations you think should be met. If they don't meet those expectations, they don't get paid. > this prestige is overvalued. The prestige is not overvalued--it is just too easily obtainable by fraud. Something has got to be done. > Preprint servers, open data and more direct science communication seem like steps in the right direction. They are vanity presses. And they don't even do what you think they are doing. Today, the problem isn't too little information, it's too much misinformation. LLMs can chug out papers by the millions. Is a search engine going to help you cut through that and find what you are looking for? What if 2 million papers which match your search criteria? You gunna read through them all, trying to find the 5 papers which were actually written by a real scientist? Are you even going to see them? Is the search engine going to do a better job than peer review of presenting you the papers you actually want to read?
- staunton 3y ago
  > my proposal of paying reviewers and replicators addresses [all concerns] Your other comment didn't say who might be paying reviewers. Journals clearly won't (why should they, they have grrat profits in the current system and will fight tooth and nail to delay any changes whatever). Universities and even funding agencies cannot (conflict of interest). > Is the search engine going to do a better job than peer review of presenting you the papers you actually want to read? I do actually expect to see that happen.
  - rhelz 3y ago
    > Your other comment didn't say who might be paying reviewers. In the parent comment to this thread, I talk about this. My proposal is that when a researcher writes up the grant to get their research funded, they should estimate how much it would cost to pay reviewers and replicators, and include those figures in the cost. If the researcher gets the grant approved, then the funding agency will put the money for peer review/replication into escrow. When the research is finished and the investigators have written up a paper to describe their methods and results, the money in escrow is disbursed to the reviewers and replicators. If reviewers agree its good, and if it replicates, then the paper is published. If not, well we just dodged a bullet. > I do actually expect to see that happen. Are search engines getting better or worse for you? It was a lot easier getting the right paper from a search engine 10 years ago. Now, you just get half a page of irrelevant ads, and another half page of links boosted by payola. Just imagine what it will be like when there are literally MILLIONS of bad papers for each good paper. Then Billions. There is no finite limit to the amount of bullshit that LLMs can--and therefore will--output.