4 ms·

Furthermore, it would be extremely useful if scientific research was transparant from inception to outcome. Scientific ‘papers’ are archaic forms of knowledge

by afpx 8y ago

Furthermore, it would be extremely useful if scientific research was transparant from inception to outcome. Scientific ‘papers’ are archaic forms of knowledge transfer suited for a time when physical paper was the only way. These days, science would serve us better if we could see everything involved in the process. I don’t want to see only the results, I want to see the whole notebook and all the hurdles along the way. Why can’t we follow the researchers and the accumulation of evidence in real time?

radarsat1 8y ago
Sounds nice, but as a scientist I can say that it would be a huge overhead to have to constantly prepare half finished work for public consumption throughout the entire process of a study. Scientific papers make sense because the point is to summarize your results, discuss what they may imply, e.g what theories they support or deny, and describe what you did in a reproducible way so that the work can be verified. The writing of a paper is an act of communication. It's important that it is an explicit thing that researchers are trained to do properly, not something that they just shit out at the end of a poorly done study that didn't work, or worse, in the middle of something where they can't draw any conclusions yet. E.g. journalists already pick up on wrong or hype-inducing interpretations of poorly written press releases of published papers. Imagine if they could also pick up on half-finished work. "Study being done could mean something amazing, maybe, if it works!" You think fake news is a problem now? Imagine the world you propose. Now, if researchers were encouraged more to describe in detail their process and everything that went right and wrong throughout a study, along with data and algorithms and everything, perhaps as appendices or in supplemental material like blogs or videos etc., and and I find this is what is happening lately, that would be fine, and a nice ideal to strive for. But realize that scientists are already required to not only perform the study but write about it, and convince every skeptic that they are right, go to conferences and get an article accepted by a journal which can take a year or more. And now add to this that they are required to prepare the data and software for public consumption, make videos and blog posts that describe everything, answer all questions that the public has. Think about all that overhead you are demanding that goes so far above and beyond doing the actual science. It's not small. And they are not paid extra for it, in fact their paycheck is probably half what they could make doing closed science for a for-profit company. Meanwhile their job as an academic is only to explore new ideas and convince their peers of their worth. Why should they go the extra mile, for free, for every member of the public who demands answers and transparency? Sorry, but it's too. much. work.
- afpx 8y ago
  Actually, when reading papers, I find the ‘communication’ parts to have the least value. It’s probably just me, but I try to skip over those parts because they tend to bias and frame the results. I want to understand the intent, strategy, methodology, math, models, data, outcomes, etc. but not editorializing. Plus, many good scientists aren’t necessarily experts at prose or communication, anyway. Why can’t the communication parts be provided as addendum? For people who work in private R&D, like me, we try to keep lab notebooks so that we have records of pretty much everything imagined or tried, warts and all. Sure, it can seem haphazard, unformatted, and not packaged for general consumption. But, it’s meant for collegues anyway, and it fits the purpose. As for your last paragraph, I totally empathize with your frustrations. But, those seem to be more structural issues with the economics and politics of science and the academies. I certainly would love to see change there, as well. I wish research could be funded in different ways. But, as a consumer of research, I can only really describe my particular use cases.
  - radarsat1 8y ago
    What I mean by "communication" is what you describe as "the intent, strategy, methodology, math, models, data, outcomes." What do you mean by editorializing? If you mean discussing implications on existing theories, I think this can be done badly or done well (like all parts of a paper.) But when doing research it's important to read follow-ups and rebuttals. We can't forget that science is a process, an on-going discussion. It's not a single result or paper.
    - afpx 8y ago
      I think many readers (science journalists, in particular) only read the abstract, introduction, and then immediately jump to the conclusions. The introduction section almost always frames the paper with opinions and biases. And, the conclusion section often focuses too heavily on how great the results are (and the author’s prowess) without identifying interesting issues or irregularities encountered in the methodology, data, or outcomes. I get that publishing is a huge part of a scientist’s career success and for self-promotion, but I see many papers that over emphasize those aspects. Again, I don’t fault them. One can’t easily get funding without a reputation. But, in my idealized (and probably unrealistic) world it would be different. I do want to stress that I’m not complaining. What I have access to is amazing. But, sometimes I wish I could see it all.
samontar 8y ago
Because they rightly fear that someone will scoop them. The Nash equilibrium is to wait till you’re done to publish.
jhauris 8y ago
A growing number of people seem to agree with this ([0],[1]). There are a couple of places ([2],[3]) where one can preregister a hypothesis and experiment before collecting data. There are also some journals that are starting to encourage preregistration ([4],[5]). It would be nice if this phenomenon spread beyond the few fields it's started in. I don't know about needing to see all the intermediate data, that seems extreme if the methodology is properly described and the raw data is available. 0: https://www.bitss.org/2014/06/13/preregistration-controversy/ https://www.bitss.org/2014/06/13/preregistration-controversy... 1: https://www.apa.org/science/about/psa/2015/08/pre-registration.aspx https://www.apa.org/science/about/psa/2015/08/pre-registrati... 2: https://aspredicted.org/ https://aspredicted.org/ 3: https://osf.io/ https://osf.io/ 4: https://www.psychologicalscience.org/publications/psychological_science/preregistration https://www.psychologicalscience.org/publications/psychologi... 5: https://www.journals.elsevier.com/cortex/news/registered-reports-a-new-article-format-from-cortex/ https://www.journals.elsevier.com/cortex/news/registered-rep...
- radarsat1 8y ago
  > preregister a hypothesis and experiment before collecting data This reminds me of an old website, I think it was called Half Baked, where people could (I think anonymously) post just any ridiculous idea for an invention. It was fun to read. Unfortunately I can't find a link. But man, when I think about it in terms of science and in the more modern context of preprints, it sounds like Flag Planting, Extreme Edition. Even the ML community, who encourage preprint publication more than any other field I know of, have had some pretty heated exchanges about the validity of using that medium for flag planting, and that is when the "flag planting" at least consists of a short paper properly describing an idea, with some rudimentary empirical result and analysis. So I dunno, it's an intriguing idea, but not without controversy I think.

by afpx 8y ago

radarsat1 8y ago

Sounds nice, but as a scientist I can say that it would be a huge overhead to have to constantly prepare half finished work for public consumption throughout the entire process of a study. Scientific papers make sense because the point is to summarize your results, discuss what they may imply, e.g what theories they support or deny, and describe what you did in a reproducible way so that the work can be verified. The writing of a paper is an act of communication. It's important that it is an explicit thing that researchers are trained to do properly, not something that they just shit out at the end of a poorly done study that didn't work, or worse, in the middle of something where they can't draw any conclusions yet. E.g. journalists already pick up on wrong or hype-inducing interpretations of poorly written press releases of published papers. Imagine if they could also pick up on half-finished work. "Study being done could mean something amazing, maybe, if it works!" You think fake news is a problem now? Imagine the world you propose. Now, if researchers were encouraged more to describe in detail their process and everything that went right and wrong throughout a study, along with data and algorithms and everything, perhaps as appendices or in supplemental material like blogs or videos etc., and and I find this is what is happening lately, that would be fine, and a nice ideal to strive for. But realize that scientists are already required to not only perform the study but write about it, and convince every skeptic that they are right, go to conferences and get an article accepted by a journal which can take a year or more. And now add to this that they are required to prepare the data and software for public consumption, make videos and blog posts that describe everything, answer all questions that the public has. Think about all that overhead you are demanding that goes so far above and beyond doing the actual science. It's not small. And they are not paid extra for it, in fact their paycheck is probably half what they could make doing closed science for a for-profit company. Meanwhile their job as an academic is only to explore new ideas and convince their peers of their worth. Why should they go the extra mile, for free, for every member of the public who demands answers and transparency? Sorry, but it's too. much. work.

afpx 8y ago
Actually, when reading papers, I find the ‘communication’ parts to have the least value. It’s probably just me, but I try to skip over those parts because they tend to bias and frame the results. I want to understand the intent, strategy, methodology, math, models, data, outcomes, etc. but not editorializing. Plus, many good scientists aren’t necessarily experts at prose or communication, anyway. Why can’t the communication parts be provided as addendum? For people who work in private R&D, like me, we try to keep lab notebooks so that we have records of pretty much everything imagined or tried, warts and all. Sure, it can seem haphazard, unformatted, and not packaged for general consumption. But, it’s meant for collegues anyway, and it fits the purpose. As for your last paragraph, I totally empathize with your frustrations. But, those seem to be more structural issues with the economics and politics of science and the academies. I certainly would love to see change there, as well. I wish research could be funded in different ways. But, as a consumer of research, I can only really describe my particular use cases.
- radarsat1 8y ago
  What I mean by "communication" is what you describe as "the intent, strategy, methodology, math, models, data, outcomes." What do you mean by editorializing? If you mean discussing implications on existing theories, I think this can be done badly or done well (like all parts of a paper.) But when doing research it's important to read follow-ups and rebuttals. We can't forget that science is a process, an on-going discussion. It's not a single result or paper.
  - afpx 8y ago
    I think many readers (science journalists, in particular) only read the abstract, introduction, and then immediately jump to the conclusions. The introduction section almost always frames the paper with opinions and biases. And, the conclusion section often focuses too heavily on how great the results are (and the author’s prowess) without identifying interesting issues or irregularities encountered in the methodology, data, or outcomes. I get that publishing is a huge part of a scientist’s career success and for self-promotion, but I see many papers that over emphasize those aspects. Again, I don’t fault them. One can’t easily get funding without a reputation. But, in my idealized (and probably unrealistic) world it would be different. I do want to stress that I’m not complaining. What I have access to is amazing. But, sometimes I wish I could see it all.

samontar 8y ago

Because they rightly fear that someone will scoop them. The Nash equilibrium is to wait till you’re done to publish.

jhauris 8y ago

A growing number of people seem to agree with this ([0],[1]). There are a couple of places ([2],[3]) where one can preregister a hypothesis and experiment before collecting data. There are also some journals that are starting to encourage preregistration ([4],[5]). It would be nice if this phenomenon spread beyond the few fields it's started in. I don't know about needing to see all the intermediate data, that seems extreme if the methodology is properly described and the raw data is available. 0: https://www.bitss.org/2014/06/13/preregistration-controversy/ https://www.bitss.org/2014/06/13/preregistration-controversy... 1: https://www.apa.org/science/about/psa/2015/08/pre-registration.aspx https://www.apa.org/science/about/psa/2015/08/pre-registrati... 2: https://aspredicted.org/ https://aspredicted.org/ 3: https://osf.io/ https://osf.io/ 4: https://www.psychologicalscience.org/publications/psychological_science/preregistration https://www.psychologicalscience.org/publications/psychologi... 5: https://www.journals.elsevier.com/cortex/news/registered-reports-a-new-article-format-from-cortex/ https://www.journals.elsevier.com/cortex/news/registered-rep...

radarsat1 8y ago
> preregister a hypothesis and experiment before collecting data This reminds me of an old website, I think it was called Half Baked, where people could (I think anonymously) post just any ridiculous idea for an invention. It was fun to read. Unfortunately I can't find a link. But man, when I think about it in terms of science and in the more modern context of preprints, it sounds like Flag Planting, Extreme Edition. Even the ML community, who encourage preprint publication more than any other field I know of, have had some pretty heated exchanges about the validity of using that medium for flag planting, and that is when the "flag planting" at least consists of a short paper properly describing an idea, with some rudimentary empirical result and analysis. So I dunno, it's an intriguing idea, but not without controversy I think.