26 ms·

I wonder whether a 'GitHub' for Science wouldn't help things along, just as it's helped the open source software community. Scientist could have repos with ful

I wonder whether a 'GitHub' for Science wouldn't help things along, just as it's helped the open source software community. Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper. The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work. Just a thought.

foxbarrington 11y ago
Would be even better if they opened regardless of the results. It's valuable to know what doesn't pan out / what not to try.
- fineman 11y ago
  With an analogy to open-source, my failed projects (like we're talking about here) fall largely into two distinct categories. Those that I never started, or were totally half-baked, or turned out to trivial - the uninteresting or incomplete. 90%. And 5% turn out to be illustrative learning moments. One of my first OOP projects trying to model the real world in hierarchical classes. It was a good non-contrived example of a need for multiple orthogonal inheritance methods as any one can easily be forced into contortions in modeling even simple structures. I tried to write a project with my own trig implementations, just because. That'd at least be funny... But the rest are a few lines of drunk-coding, or something I found a library to do and never developed, or in some other way a waste of time to look at. So I don't want to hold scientists to standards I wouldn't want. Nor do I want to miss the value in the noise. So maybe we should have a honor-system somewhat like this - when you first describe to project to anyone looking for funding, or discussed it with more than a handful of peers, or ... then you make a note in your lab book. "Research direction: see if flatworms are really flat because Y - hope it'll give insight into Z". And your internal audit system submits all of these to a pool of peers who pick by interest in the area. If it's chosen, an auditor checks if the idea was ever really developed, dropping it if not, and summarizing whatever is there (or letting you write a paper) if there are results. It sounds like a lot of work, but considering that everyone is asking for grants and being audited anyways, and this would replace vast swaths of less-pointed simple bookkeeping audit with peer-review of your methods, etc. In other words, likely higher quality results for the funding bodies in less time, and with the additional benefit of being able to publish interesting failure results. Thus achieving a balance between researcher privacy and ability to try without public mockery on failure, funding efficiency, and scientific benefit.
mkstowegnv 11y ago
There is a lot of activity along these lines - see https://en.wikipedia.org/wiki/Open_science https://en.wikipedia.org/wiki/Open_science and especially follow the "see also" links.
beloch 11y ago
Very little of what is actually needed to reproduce an experiment goes into academic papers. What you'd really want is a more formal, digital version of the "lab book". Experimentalists are taught, from their earliest days as an undergrad, to keep notes on what they're doing. The idea is to put everything useful in there. Got a new piece of gear with a long startup procedure? Put it in your book. Designing an experiment? Put the design in there. Ordered parts? Make a note of it. Running an experiment and finally found a way to do things that works? Write down the procedure. Got some results from your experiment? Print them out and paste them in your book. Saw something odd? Write it down. The result is a linear log that tracks a lot of stuff, both useful and useless. With experience, you learn to make use of tables of contents and margin notes, but searching through them for something dimly recollected can sometimes be time consuming. Lab books are where the gory details of how to actually do things is stored. The procedures, diagrams, and details in them are simply not conveyed in papers because there's no space for them, and many are viewed to be "common knowledge", although this kind of knowledge is probably only common to a small number of labs in the world. The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary. They're written with yourself as the intended audience and can sometimes be impenetrable for anyone else. If you're working in a commercial lab, as opposed to research in academia, these books might be considered the property of your employer. In this situation you might try to make them more friendly to other human beings, but that often isn't a priority. Putting things into a digital form would, in general, take a lot longer. Even with good proficiency in LaTeX, complex equations are quicker to just write on paper. Hand-drawn diagrams are also very common in lab books, and the software/hardware to draw easily in a digital form is not yet ubiquitous. Finally, digital storage occasionally dies. A lab book is a precious possession whose loss can cause months or years of difficulty. Physical hard-copy is easier to store long-term. Some people have undergraduate lab books from half a century ago sitting on their office bookshelves. There is nothing that is technologically insurmountable to moving to a digital lab book format. Microsoft's surface tablets make digital drawing pretty painless. Cloud storage could be maintained long-term. Etc. The benefits of digital lab books could be considerable. They'd be easier to search and distribute. You could include copies in a digital appendix to an experiment (currently, most journals discourage large attachments because they don't like to pay for storage). To my knowledge, nobody has actually put everything that's needed together into a single cohesive program. You'd want a note-taking program with full LaTeX and freehand drawing support. You'd need to be able to paste in tables, graphs, and data files easily. It would need to have a long-term archiving solution and a really good interface. It would need to be as fast or faster than writing on paper for pretty much everything. It would also probably need to be open source, as scientists would probably not trust proprietary software that might make their notes difficult to access at some point in the future. Finally, the hardware to run it on would need to be ubiquitous.
LyndsySimon 11y ago
> Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper. > The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work. Funny you should say that... I work at the Center for Open Science, and our founder was quoted in the article. One of the major use cases for our primary product (the Open Science Framework) is to facilitate preregistration. Here's the relevant section of our "Getting Started" page, detailing how preregistration works: https://osf.io/getting-started/#registrations https://osf.io/getting-started/#registrations You can find out more about COS at http://cos.io http://cos.io.

foxbarrington 11y ago

Would be even better if they opened regardless of the results. It's valuable to know what doesn't pan out / what not to try.

fineman 11y ago
With an analogy to open-source, my failed projects (like we're talking about here) fall largely into two distinct categories. Those that I never started, or were totally half-baked, or turned out to trivial - the uninteresting or incomplete. 90%. And 5% turn out to be illustrative learning moments. One of my first OOP projects trying to model the real world in hierarchical classes. It was a good non-contrived example of a need for multiple orthogonal inheritance methods as any one can easily be forced into contortions in modeling even simple structures. I tried to write a project with my own trig implementations, just because. That'd at least be funny... But the rest are a few lines of drunk-coding, or something I found a library to do and never developed, or in some other way a waste of time to look at. So I don't want to hold scientists to standards I wouldn't want. Nor do I want to miss the value in the noise. So maybe we should have a honor-system somewhat like this - when you first describe to project to anyone looking for funding, or discussed it with more than a handful of peers, or ... then you make a note in your lab book. "Research direction: see if flatworms are really flat because Y - hope it'll give insight into Z". And your internal audit system submits all of these to a pool of peers who pick by interest in the area. If it's chosen, an auditor checks if the idea was ever really developed, dropping it if not, and summarizing whatever is there (or letting you write a paper) if there are results. It sounds like a lot of work, but considering that everyone is asking for grants and being audited anyways, and this would replace vast swaths of less-pointed simple bookkeeping audit with peer-review of your methods, etc. In other words, likely higher quality results for the funding bodies in less time, and with the additional benefit of being able to publish interesting failure results. Thus achieving a balance between researcher privacy and ability to try without public mockery on failure, funding efficiency, and scientific benefit.

mkstowegnv 11y ago

There is a lot of activity along these lines - see https://en.wikipedia.org/wiki/Open_science https://en.wikipedia.org/wiki/Open_science and especially follow the "see also" links.

beloch 11y ago

Very little of what is actually needed to reproduce an experiment goes into academic papers. What you'd really want is a more formal, digital version of the "lab book". Experimentalists are taught, from their earliest days as an undergrad, to keep notes on what they're doing. The idea is to put everything useful in there. Got a new piece of gear with a long startup procedure? Put it in your book. Designing an experiment? Put the design in there. Ordered parts? Make a note of it. Running an experiment and finally found a way to do things that works? Write down the procedure. Got some results from your experiment? Print them out and paste them in your book. Saw something odd? Write it down. The result is a linear log that tracks a lot of stuff, both useful and useless. With experience, you learn to make use of tables of contents and margin notes, but searching through them for something dimly recollected can sometimes be time consuming. Lab books are where the gory details of how to actually do things is stored. The procedures, diagrams, and details in them are simply not conveyed in papers because there's no space for them, and many are viewed to be "common knowledge", although this kind of knowledge is probably only common to a small number of labs in the world. The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary. They're written with yourself as the intended audience and can sometimes be impenetrable for anyone else. If you're working in a commercial lab, as opposed to research in academia, these books might be considered the property of your employer. In this situation you might try to make them more friendly to other human beings, but that often isn't a priority. Putting things into a digital form would, in general, take a lot longer. Even with good proficiency in LaTeX, complex equations are quicker to just write on paper. Hand-drawn diagrams are also very common in lab books, and the software/hardware to draw easily in a digital form is not yet ubiquitous. Finally, digital storage occasionally dies. A lab book is a precious possession whose loss can cause months or years of difficulty. Physical hard-copy is easier to store long-term. Some people have undergraduate lab books from half a century ago sitting on their office bookshelves. There is nothing that is technologically insurmountable to moving to a digital lab book format. Microsoft's surface tablets make digital drawing pretty painless. Cloud storage could be maintained long-term. Etc. The benefits of digital lab books could be considerable. They'd be easier to search and distribute. You could include copies in a digital appendix to an experiment (currently, most journals discourage large attachments because they don't like to pay for storage). To my knowledge, nobody has actually put everything that's needed together into a single cohesive program. You'd want a note-taking program with full LaTeX and freehand drawing support. You'd need to be able to paste in tables, graphs, and data files easily. It would need to have a long-term archiving solution and a really good interface. It would need to be as fast or faster than writing on paper for pretty much everything. It would also probably need to be open source, as scientists would probably not trust proprietary software that might make their notes difficult to access at some point in the future. Finally, the hardware to run it on would need to be ubiquitous.

LyndsySimon 11y ago

> Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper. > The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work. Funny you should say that... I work at the Center for Open Science, and our founder was quoted in the article. One of the major use cases for our primary product (the Open Science Framework) is to facilitate preregistration. Here's the relevant section of our "Getting Started" page, detailing how preregistration works: https://osf.io/getting-started/#registrations https://osf.io/getting-started/#registrations You can find out more about COS at http://cos.io http://cos.io.