6 ms·

The same thing is trying to happen in scientific publication that's been happening in every other published medium: scarcity is gone. So why not embrace the abu

The same thing is trying to happen in scientific publication that's been happening in every other published medium: scarcity is gone. So why not embrace the abundance? Well if there's any contrary case to be made, it's that the explosion in quantity inevitably reduces the average quality. In the arts, low quality corresponds to that ineffable quality of shittiness. In news, it's fakeness. In science, it's irreproducibility. So what do you do? All the former built-in scarcity-based ways of filtering for quality went out the window along with the scarcity itself. You can no longer use the limited space on physical sheets of papyrus (or vinyl LP records, or celluloid) as an excuse to exclude things. So you need new ways of sorting out good quality work from the (suddenly vast) quantities of mediocre work. Seems like you're doing pretty well if you're lucky enough to have a whole new class of people step in to help with the filtering part. Unless of course you believe a scientific discovery is fundamentally the work of a person, an ego, rather than a fact that existed long before that particular person happened to discover it. Democratization ends elitism; unfortunately elites sometimes really do consist of the best of a given thing. Especially newer elites. Every elite starts out as a meritocracy and ends up some kind of weird legacy-based cabal/cartel that deserves to be overthrown (that's the one you're probably accustomed to thinking of when you hear the term "elitism").

petschge 7y ago
(I agree with the parent poster, but this seems like the right place to add the following.) One thing that people who claim "the scarcity is gone, let's just publish everything" do miss is that there still is a scarcity: The time and attention of researchers. Let me expand a bit on that from my own field of studying reconnection in astrophysical plasma. As you can tell from the description it is not a well defined closed topic. Instead there is tons of more or less adjacent fields. The study of solar flares (which might be triggered by reconnection), the study of coronal heating (the heat might be due to magnetic field getting converted to heat by reconnection), astronomical observations of AGN jets (reconnection might be what produces energetic particles there that we see in the SDE of those sources), observations of pulsar wind nebulae (the energetic particles there might have been accelerated at the termination shock. or due to reconnection), observations of giant pulses in (some) pulsars (that might be due to reconnection at the Y point of the current sheet just outside the light cylinder) and so on. On top of the related physical topics I need to keep on top of development methods in the simulation method I use (particle in cell codes) and in several related simulation methods (either because improvements there might make them viable to study reconnection which wasn't possible before, or because the neat new trick in method X might also improve the characteristics in PiC codes). If I wanted to I could spend easily 100 hours per week just reading papers. But staying current with the field is just 5 or maybe 10 percent of my job. So what I do is the following: Papers that fall close enough to my special topic I will read. All of them. I will actually print them out, annotate them, go over them with a fine toothed comb. For many other papers I will read the abstracts (10 to 50 each morning) to find the 1 or 2 paper that are worth reading. And this is where selection by the editors and limitations to publications come in. They rank important papers. I am much more likely to read "a novel approach to plasma simulation" if it was accepted into the Astrophysical Journal than if it just appears on ArXiv. Because ApJ does not like pure code papers. So if it made the cut 2 or 3 experts in the field deemed it worth the time of the community. Now that doesn't mean that all ArXiv papers are bad, that we should publish less or anything like that. When I talk to a colleague and ask about a technical detail I am SOOOO happy when they say "it's in the Arxiv paper ID 1906.bla". And on the other hand there is the notion of "it was in Nature but it still might be right". Bottom line is: Do not discount the sorting by topic (numerical vs observational vs theoretical), impact ("here is one data point" vs "here is a completely new approach") and quality ("I'm not too convinced, but maybe it gives somebody an idea" vs "holy smokes how did we all miss that!") that journals provide. Any better, future alternative needs this sorting and ranking. Just dumping it onto the internet is not the solution.
- dorchadas 7y ago
  You're right, it does need some type of ranking. I think the endorsement could be something like that. Instead of having journals endorse it, let specific academics do it. So if Dr. John Doe really likes an article, he can endorse it personally, and put his personal reputation up to bat for it. And then have an algorithm that sorts based on endorsements, or how trusted the endorsers are ('Do you agree with this endorsement', or how many endorsements they themselves have). Let the authors easily input all the topics they want as tags, making it easy to search as well. And, of course, public comments about what would need to change to get an endorsement, as well as allowing readers to look at all previous versions of an article to see what changed. Basically, let the academics take care of endorsing and topic tagging themselves.
  - petschge 7y ago
    Reviewing every paper by just one to three qualified people is already hard and putting a lot of strain on the system. Trying to solve that by "just have everybody read everything and endorse what they like" is going to break it. Or rather most paper will never be read nor endorsed by a professional. All it would do it turn science into even more of a who like whom.
- baxtr 7y ago
  Great insight, thanks! It sounds like AI could play a part here in order to achieve the required efficiency, e.g. sorting out the right papers for you.
  - petschge 7y ago
    All I can say is that all the feeds that currently promise to solve that problem (Nasa ADS, Google Scholar, Researchgate and the feeds of several journals) are all pretty bad. I get several suggestions for medical/biophysical papers every day (probably due to "plasma" and "particle in _C_ell").
- isodude 7y ago
  The same is true within IT, you need to read a lot to make sure you're in the loop. I think it correlates to map structures, if every point in a map is connected to every other point there will be a large surface for each point in the beginning but as the amount of points grow the surface (of possible attention in this case) will shrink. How much area of attention is needed to perform the best in a creative research, to produce the best work in a given time? Maybe the answer to that question will yield more papers with better quality. My personal belief is that we need to dream more and do less work to make sure the work that is done is of more quality such that no filtration is needed and we can truly publish everything.
deleted 7y ago
[deleted]
fortdriad 7y ago
This is a complex topic and I think Gelman (the linked poster) is either misinterpreting and/or confused by Kaufman and Glǎveanu (the article he's discussing). Just for some context, I agree with Gelman and KG both in their main arguments. There's different issues colliding in the open science movement. One is what you're referring to, the fact that scarcity is gone. Combined with the overcrowding and hypercompetitiveness of science, you not only have what you are referring to, a decrease in verdicality, you also have a decrease in signal-to-noise in general. So the nonsense increases, but so do the traditional signals to quality. So nonsense appears in high-profile journals, and very quality work appears in low-profile journals or even as "unpublished" pieces. The other problem though, what my colleagues refer to as the "science police", is an increasing tendency for certain groups to argue that a certain set of practices are not only good, but necessary for "good" science, and by implication, everything that does not is "bad" science, in a black-and-white kind of way. For one thing, not all problems are with replication. Nonsense can replicate well, and very important legitimate phenomena can be difficult to replicate. If something is really not replicable at all, that's a problem, but replicability per se is only one part of scientific progress, and it comes in degrees with various causes. It's also much more difficult to determine what is replicable sometimes than it might seem on the surface. Replicate what? What's important to replicate? How? Sometimes this is clear, but other times it is not. Also, when you really delve into it, there's not really a good rationale for what, exactly, are the important ingredients for open science, or why. For example, is it really necessary to have preregistered studies? What's to keep someone from preregistering but then silently declining to publish null results? Or to "preregister" something they've already collected? If an important unanalyzed pre-existing dataset becomes available, is that "tainted" because it wasn't preregistered? Is it important to preregister, or just to make the data openly available? Is it better to use modeling to identify anomalies in studies, or to rely on preregistration? These issues aren't always clear. I think there's a sense sometimes that the open science movement is not only trying to dismantle a broken system run by an established elite, but to replace it dogmatically with a new system run by a new elite, with its own imperfect rules. Already I've seen misuses of open science guidelines used to bully and discredit legitimate work (for example, by suggesting that someone is hiding something by not sharing data, when the data contains protected healthcare information and would be accessible to them anyway if they would just go through proper channels). This is tricky to discuss, as you might imagine, so it comes out in pieces like KG's piece. Gelman is asking "why not publish everything", which is responding (I think) to something different from what KG are responding to. Maybe I'm misreading KG, but I think they might also argue "why not publish everything"; they just have a different group they're addressing when they would say that.

petschge 7y ago

(I agree with the parent poster, but this seems like the right place to add the following.) One thing that people who claim "the scarcity is gone, let's just publish everything" do miss is that there still is a scarcity: The time and attention of researchers. Let me expand a bit on that from my own field of studying reconnection in astrophysical plasma. As you can tell from the description it is not a well defined closed topic. Instead there is tons of more or less adjacent fields. The study of solar flares (which might be triggered by reconnection), the study of coronal heating (the heat might be due to magnetic field getting converted to heat by reconnection), astronomical observations of AGN jets (reconnection might be what produces energetic particles there that we see in the SDE of those sources), observations of pulsar wind nebulae (the energetic particles there might have been accelerated at the termination shock. or due to reconnection), observations of giant pulses in (some) pulsars (that might be due to reconnection at the Y point of the current sheet just outside the light cylinder) and so on. On top of the related physical topics I need to keep on top of development methods in the simulation method I use (particle in cell codes) and in several related simulation methods (either because improvements there might make them viable to study reconnection which wasn't possible before, or because the neat new trick in method X might also improve the characteristics in PiC codes). If I wanted to I could spend easily 100 hours per week just reading papers. But staying current with the field is just 5 or maybe 10 percent of my job. So what I do is the following: Papers that fall close enough to my special topic I will read. All of them. I will actually print them out, annotate them, go over them with a fine toothed comb. For many other papers I will read the abstracts (10 to 50 each morning) to find the 1 or 2 paper that are worth reading. And this is where selection by the editors and limitations to publications come in. They rank important papers. I am much more likely to read "a novel approach to plasma simulation" if it was accepted into the Astrophysical Journal than if it just appears on ArXiv. Because ApJ does not like pure code papers. So if it made the cut 2 or 3 experts in the field deemed it worth the time of the community. Now that doesn't mean that all ArXiv papers are bad, that we should publish less or anything like that. When I talk to a colleague and ask about a technical detail I am SOOOO happy when they say "it's in the Arxiv paper ID 1906.bla". And on the other hand there is the notion of "it was in Nature but it still might be right". Bottom line is: Do not discount the sorting by topic (numerical vs observational vs theoretical), impact ("here is one data point" vs "here is a completely new approach") and quality ("I'm not too convinced, but maybe it gives somebody an idea" vs "holy smokes how did we all miss that!") that journals provide. Any better, future alternative needs this sorting and ranking. Just dumping it onto the internet is not the solution.

dorchadas 7y ago
You're right, it does need some type of ranking. I think the endorsement could be something like that. Instead of having journals endorse it, let specific academics do it. So if Dr. John Doe really likes an article, he can endorse it personally, and put his personal reputation up to bat for it. And then have an algorithm that sorts based on endorsements, or how trusted the endorsers are ('Do you agree with this endorsement', or how many endorsements they themselves have). Let the authors easily input all the topics they want as tags, making it easy to search as well. And, of course, public comments about what would need to change to get an endorsement, as well as allowing readers to look at all previous versions of an article to see what changed. Basically, let the academics take care of endorsing and topic tagging themselves.
- petschge 7y ago
  Reviewing every paper by just one to three qualified people is already hard and putting a lot of strain on the system. Trying to solve that by "just have everybody read everything and endorse what they like" is going to break it. Or rather most paper will never be read nor endorsed by a professional. All it would do it turn science into even more of a who like whom.
baxtr 7y ago
Great insight, thanks! It sounds like AI could play a part here in order to achieve the required efficiency, e.g. sorting out the right papers for you.
- petschge 7y ago
  All I can say is that all the feeds that currently promise to solve that problem (Nasa ADS, Google Scholar, Researchgate and the feeds of several journals) are all pretty bad. I get several suggestions for medical/biophysical papers every day (probably due to "plasma" and "particle in _C_ell").
isodude 7y ago
The same is true within IT, you need to read a lot to make sure you're in the loop. I think it correlates to map structures, if every point in a map is connected to every other point there will be a large surface for each point in the beginning but as the amount of points grow the surface (of possible attention in this case) will shrink. How much area of attention is needed to perform the best in a creative research, to produce the best work in a given time? Maybe the answer to that question will yield more papers with better quality. My personal belief is that we need to dream more and do less work to make sure the work that is done is of more quality such that no filtration is needed and we can truly publish everything.

deleted 7y ago

[deleted]

fortdriad 7y ago

This is a complex topic and I think Gelman (the linked poster) is either misinterpreting and/or confused by Kaufman and Glǎveanu (the article he's discussing). Just for some context, I agree with Gelman and KG both in their main arguments. There's different issues colliding in the open science movement. One is what you're referring to, the fact that scarcity is gone. Combined with the overcrowding and hypercompetitiveness of science, you not only have what you are referring to, a decrease in verdicality, you also have a decrease in signal-to-noise in general. So the nonsense increases, but so do the traditional signals to quality. So nonsense appears in high-profile journals, and very quality work appears in low-profile journals or even as "unpublished" pieces. The other problem though, what my colleagues refer to as the "science police", is an increasing tendency for certain groups to argue that a certain set of practices are not only good, but necessary for "good" science, and by implication, everything that does not is "bad" science, in a black-and-white kind of way. For one thing, not all problems are with replication. Nonsense can replicate well, and very important legitimate phenomena can be difficult to replicate. If something is really not replicable at all, that's a problem, but replicability per se is only one part of scientific progress, and it comes in degrees with various causes. It's also much more difficult to determine what is replicable sometimes than it might seem on the surface. Replicate what? What's important to replicate? How? Sometimes this is clear, but other times it is not. Also, when you really delve into it, there's not really a good rationale for what, exactly, are the important ingredients for open science, or why. For example, is it really necessary to have preregistered studies? What's to keep someone from preregistering but then silently declining to publish null results? Or to "preregister" something they've already collected? If an important unanalyzed pre-existing dataset becomes available, is that "tainted" because it wasn't preregistered? Is it important to preregister, or just to make the data openly available? Is it better to use modeling to identify anomalies in studies, or to rely on preregistration? These issues aren't always clear. I think there's a sense sometimes that the open science movement is not only trying to dismantle a broken system run by an established elite, but to replace it dogmatically with a new system run by a new elite, with its own imperfect rules. Already I've seen misuses of open science guidelines used to bully and discredit legitimate work (for example, by suggesting that someone is hiding something by not sharing data, when the data contains protected healthcare information and would be accessible to them anyway if they would just go through proper channels). This is tricky to discuss, as you might imagine, so it comes out in pieces like KG's piece. Gelman is asking "why not publish everything", which is responding (I think) to something different from what KG are responding to. Maybe I'm misreading KG, but I think they might also argue "why not publish everything"; they just have a different group they're addressing when they would say that.