3 ms·

Solution is to publish data, not „papers“ first and assign it a replication score - how many times it was verified by independent research. The paper can foll

Solution is to publish data, not „papers“ first and assign it a replication score - how many times it was verified by independent research. The paper can follow with the explanation, but citations will no longer be important - what will matter is the contribution to the replication score (will also work as an incentive to confirm other‘s results).

m_mueller 5y ago
I think it would be gamed just like the current system. Instead of citation rings you just get replication rings.
- ivan_gammel 5y ago
  If someone gets contradicting result the replication score of the entire ring can be nullified or in case of intentional manipulation with data negated.
  - m_mueller 5y ago
    But you have the same basic problem as now - you’d need some sort of science police to control it, which goes against the scientific process. Essentially it’s a problem of establishing trust in an untrusted system. Putting it that way actually makes it sound like a blockchain problem. Maybe there could be some incentive system to replicate work based on smart contracts, but I don’t know how you could ensure the replicating parties to be independent.
    - ivan_gammel 5y ago
      Scientific progress today heavily depends on financial support of society, so as a whole it cannot be completely decentralized and independent. People want to know how their money are spent and want to have guarantees that science will not create something awful. This means that policing of science is inevitable and important part of the system. It is not a question if we need science “police”, it is a question how it should look like. Today it is decentralized: someone maintains the list of media publishing in which will count for citation index, there are ethical committees and scientific boards, lawmakers regularly tell what can be done and what should not etc. How this will change if there will be a new system of incentives in place, we can only imagine: it can be a good or a bad thing, but as long as the system remains democratic, all problems should be easy to fix.
    - deleted 5y ago
      [deleted]
headmelted 5y ago
This seems like the right answer. Don’t (credible) journalists have an honour system of getting at least three sources for a story? Can’t we make researchers get at least two more confirmations from separate teams for something far more important?
- waheoo 5y ago
  There aren't many credible journalists left. Maybe like 5.
  - cto_of_antifa 5y ago
    True dat. NYT did 9/11
- PeterisP 5y ago
  A key function of scientific publication is to inform other researchers in the field about potentially interesting things as quickly as resonable. Getting "two more confirmations from separate teams" is a very high bar, as it's not about just asking a source, it's asking someone else to do all the same work again. Not only we don't require it before publication, we don't expect it to happen for the vast majority of publications ever. Important studies get replicated, but most don't get repeated ever. A partial explanation of the original article's observation is the (very many!) papers that don't have much citations and don't fail to replicate because nobody cared enough to put the work in to try. If publication would require two more confirmations from separate teams, that would mean (a) doing the work in triplicate, so you get three times less results for the same effort; (b) the process would take twice as long as I spend a year doing the experiment and then someone else can start and spend a year doing the same experiment, and only then it gets published; (c) there's a funding issue - I have somehow got funding to spend many months of multiple people on this, but who's paying the other independent teams to do that?; (d) it's not a given that there are two other teams capable of doing the exact same research, e.g. if you want to publish a study on the results of an innovative surgery procedure, it's plausible that there aren't (yet!) any other surgeons worldwide who are ready to perform that operation, that will come some time after the publication; (e) many types of science really can't get a separate confirmation - for example, we have only one Large Hadron Collider, you can't re-do archeological digs, event-specific on-site sociological data gathering can't really be repeated, etc; so you have to take the data at face value.
Bombthecat 5y ago
Somewhere in there i see a blockchain pitch.
- ivan_gammel 5y ago
  I personally love this comment as a quintessence of startup mentality.
- dorgo 5y ago
  Game theory might be better suited.
- alboaie 5y ago
  Having a scientific blockchain looks like a decent idea... but of course it will not suffice and will be gamed. The real causes of the mess are complexity of the world compared with our minds and tools and the lack of epistemologic undestanding as society,institutions, culture. Science can't be more than a nice and usefull collection of heuristics. Otherwise is just the Scientism religion lurking arround and pretenting to read God's mind. Metarationality concepts could offer an exit from the inevitable mess.
  - lasagnaphil 5y ago
    What do you mean by 'metarationality'? It's a term I've never seen before and I'm curious about it.
seoaeu 5y ago
Ok, I got 12, 18, 45. Does anyone want to verify my results? If so, I'll write up a paper describing what they mean... Hopefully it is clear that that data is useless without some written text explaining what it means. Given that for hundreds of years the accepted way of presenting that explanatory text was by writing papers, I don't see any reason to abandon that. Tweaking our strategies for replication (after a description of the experiment has been published!) and reputation don't seem to contradict that.

m_mueller 5y ago

I think it would be gamed just like the current system. Instead of citation rings you just get replication rings.

ivan_gammel 5y ago
If someone gets contradicting result the replication score of the entire ring can be nullified or in case of intentional manipulation with data negated.
- m_mueller 5y ago
  But you have the same basic problem as now - you’d need some sort of science police to control it, which goes against the scientific process. Essentially it’s a problem of establishing trust in an untrusted system. Putting it that way actually makes it sound like a blockchain problem. Maybe there could be some incentive system to replicate work based on smart contracts, but I don’t know how you could ensure the replicating parties to be independent.
  - ivan_gammel 5y ago
    Scientific progress today heavily depends on financial support of society, so as a whole it cannot be completely decentralized and independent. People want to know how their money are spent and want to have guarantees that science will not create something awful. This means that policing of science is inevitable and important part of the system. It is not a question if we need science “police”, it is a question how it should look like. Today it is decentralized: someone maintains the list of media publishing in which will count for citation index, there are ethical committees and scientific boards, lawmakers regularly tell what can be done and what should not etc. How this will change if there will be a new system of incentives in place, we can only imagine: it can be a good or a bad thing, but as long as the system remains democratic, all problems should be easy to fix.
  - deleted 5y ago
    [deleted]

headmelted 5y ago

This seems like the right answer. Don’t (credible) journalists have an honour system of getting at least three sources for a story? Can’t we make researchers get at least two more confirmations from separate teams for something far more important?

waheoo 5y ago
There aren't many credible journalists left. Maybe like 5.
- cto_of_antifa 5y ago
  True dat. NYT did 9/11
PeterisP 5y ago
A key function of scientific publication is to inform other researchers in the field about potentially interesting things as quickly as resonable. Getting "two more confirmations from separate teams" is a very high bar, as it's not about just asking a source, it's asking someone else to do all the same work again. Not only we don't require it before publication, we don't expect it to happen for the vast majority of publications ever. Important studies get replicated, but most don't get repeated ever. A partial explanation of the original article's observation is the (very many!) papers that don't have much citations and don't fail to replicate because nobody cared enough to put the work in to try. If publication would require two more confirmations from separate teams, that would mean (a) doing the work in triplicate, so you get three times less results for the same effort; (b) the process would take twice as long as I spend a year doing the experiment and then someone else can start and spend a year doing the same experiment, and only then it gets published; (c) there's a funding issue - I have somehow got funding to spend many months of multiple people on this, but who's paying the other independent teams to do that?; (d) it's not a given that there are two other teams capable of doing the exact same research, e.g. if you want to publish a study on the results of an innovative surgery procedure, it's plausible that there aren't (yet!) any other surgeons worldwide who are ready to perform that operation, that will come some time after the publication; (e) many types of science really can't get a separate confirmation - for example, we have only one Large Hadron Collider, you can't re-do archeological digs, event-specific on-site sociological data gathering can't really be repeated, etc; so you have to take the data at face value.

Bombthecat 5y ago

Somewhere in there i see a blockchain pitch.

ivan_gammel 5y ago
I personally love this comment as a quintessence of startup mentality.
dorgo 5y ago
Game theory might be better suited.
alboaie 5y ago
Having a scientific blockchain looks like a decent idea... but of course it will not suffice and will be gamed. The real causes of the mess are complexity of the world compared with our minds and tools and the lack of epistemologic undestanding as society,institutions, culture. Science can't be more than a nice and usefull collection of heuristics. Otherwise is just the Scientism religion lurking arround and pretenting to read God's mind. Metarationality concepts could offer an exit from the inevitable mess.
- lasagnaphil 5y ago
  What do you mean by 'metarationality'? It's a term I've never seen before and I'm curious about it.

seoaeu 5y ago

Ok, I got 12, 18, 45. Does anyone want to verify my results? If so, I'll write up a paper describing what they mean... Hopefully it is clear that that data is useless without some written text explaining what it means. Given that for hundreds of years the accepted way of presenting that explanatory text was by writing papers, I don't see any reason to abandon that. Tweaking our strategies for replication (after a description of the experiment has been published!) and reputation don't seem to contradict that.