5 ms·

I have a slightly another idea. Any research activity of a scientist should be recorded by him/her in his/her own repository. Any activity, like reading a paper

by ordu 8y ago

I have a slightly another idea. Any research activity of a scientist should be recorded by him/her in his/her own repository. Any activity, like reading a paper, or moments like "I have an idea", or "lets try a small scale pilot experiment with fellow scientists of mine as a subjects", or "pilot experiment was unsuccessful, but can be modified...". Any publication should be forked from the main repository, selecting from any relevant data from parent, and in that fork strict procedures of scientific research are recorded, processed, and finally end up with an article. In this case you can track how an idea was formed, how it was modified to match empirical data. You can track when methods of data processing was chosen (before gathering of data or after), and so on. Such a repository can give enough information to decide was the research is a fraud or no.

eXpl0it3r 8y ago
It would also heavily detract from actually writing the paper. It's easy in hindsight to think about certain steps it took to get where you end up, but if you are at the start, there are simply too many intentional and unintentional steps to record everything. I much rather have people focus on the actual science than documenting their steps ever few minutes.
- ordu 8y ago
  An article is a hindsight reflections on what was done. If you are doing research you need to make notes. I personally can't remember what I've read and where I got an idea. I need to write things down, to be able to refer to it later. I'm logging my thoughts on regular basis, just to be sure they are verbalized and formalized. The goal is not to document every step in a formal manner, the goal is to leave a trail that can be tracked by a curious researcher to see how article was formed. Current situation is bad, because articles do not capture all the information that needed to validate the research findings. p<0.05 is not enough for this. Including more information into the article is one of the ways to deal with that problem. But the other is to leave footprints, to create some historical data, that could be analized. Or not analized if no one needed it. It is like lazy-evaluation: just generate some data to allow your followers to reason about what precisely you've done. If they needed it, they will research your trail. You do not need to go through a difficult task of writing a high-quality texts about every your step. The only thing you need to change is to make such notes public. It is coupled with some potential ethical issues, like privacy of subjects for example. If I run experiment, I gathered some data, I couldn't just throw this data into public access. It is possible that there are other ethical issues, maybe something about privacy of researcher as a person. These issues can make logging research activities a pain.
joppy 8y ago
It’s common practice for scientists to keep logbooks for this exact reason. This is already done.
- ordu 8y ago
  As I understand that logbooks are not available to other scientists, and if group of highly qualified scientists are making a meta-research, combining results of a dozen of articles (and rejecting two more dozens from their study), they cannot use logbooks of articles to reason about articles.
  - joppy 8y ago
    It's true that more open-access logbooks might improve things. But even if you had the logbook, you still wouldn't have the custom equipment, experimental apparatus, experimental subjects, and oversight throughout the experiment which you would need to verify that the logbook itself has data worth a damn. At some point, you have to trust other professionals working in your field to do their work well. Or rather, decide who to trust.
    - ordu 8y ago
      Maybe you are right, but I have a chat with a person who did meta-research and as I understand they dig into articles, looking for methodology, and if they found methodology good enough they would incorporate that article into their meta-research. It impies trust in good intentions of others. But not the blind trust that anything was done by a scientist was done good. If logbook can be used to verify that data processing methods was picked before the first experimental subject came to a lab, then it is already a good sign. It could be much harder to do some p-hacking like [this](https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...).
sn41 8y ago
I am not sure about this... are you talking about experimental research? I am a theoretician. I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues. The advantage of theoretical papers is that the reasoning is a proof/argument developed in the paper itself. With theory, there's less chance for a fraudulent paper, even though the results may have gaps.
- naasking 8y ago
  > I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues. I think this stigma against making mistakes is a big problem, as is the ego-investment. We're all stumbling around in the dark, and we shouldn't be afraid to admit that.
- ordu 8y ago
  > I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues. Thats the point. If you published your preliminary silly idea and your colleagues didn't find immediately that it is silly, then your not sillier than they are. :) > the reasoning is a proof/argument developed in the paper itself Even when I studied math I found sometimes that just to have proof is not enough to understand. The only way to understand sometimes is to learn how the proof was devised. What the hell author thought to get an idea of some trick in the proof? Some tricks looks as a magic, you see they are true from a logical standpoint, but they make no sense. You cannot understand how they lead to a QED. You can verify proof by logic -- not a problem, but you unable to understand. And if you are unable to understand, then at least you are unable to reproduce this proof at the exam, the only way is to learn it by heart. By I never trained my memory to learn by heart, it is really hard for me and annoying, and moreover I do not believe in learning by heart: to master a subject you need to understand it, not to learn it by heart. My math experience taught me that if I cannot follow the thoughts of other mathematician by just reading his proof, it means that he is smarter than me, so it doesn't matter then how stupid he behaved while searching for his proof, he outsmarted me already. But yes, I was talking about experimental research.
twic 8y ago
It's a bit rich for this to be proposed in a software-centric forum when our industry is merrily rebasing its way to clean linear histories that record the story the author wants to tell rather than what actually happened.
asdff 8y ago
That is the idea behind the lab notebook (what was done) + journal (what was thought) system that people are supposed to be following. We only know about Gregor Mendel because of his meticulous notebooks, for instance. My boss taught me the easiest way to write a paper is to start writing it before your first experiment, references and all; it's a great way to find weaknesses in your approach before you go too far in one direction or another. The problem is people get lazy documenting everything, especially when you are trying to leave the lab as fast as possible at the end of the day :)

by ordu 8y ago

eXpl0it3r 8y ago

It would also heavily detract from actually writing the paper. It's easy in hindsight to think about certain steps it took to get where you end up, but if you are at the start, there are simply too many intentional and unintentional steps to record everything. I much rather have people focus on the actual science than documenting their steps ever few minutes.

ordu 8y ago
An article is a hindsight reflections on what was done. If you are doing research you need to make notes. I personally can't remember what I've read and where I got an idea. I need to write things down, to be able to refer to it later. I'm logging my thoughts on regular basis, just to be sure they are verbalized and formalized. The goal is not to document every step in a formal manner, the goal is to leave a trail that can be tracked by a curious researcher to see how article was formed. Current situation is bad, because articles do not capture all the information that needed to validate the research findings. p<0.05 is not enough for this. Including more information into the article is one of the ways to deal with that problem. But the other is to leave footprints, to create some historical data, that could be analized. Or not analized if no one needed it. It is like lazy-evaluation: just generate some data to allow your followers to reason about what precisely you've done. If they needed it, they will research your trail. You do not need to go through a difficult task of writing a high-quality texts about every your step. The only thing you need to change is to make such notes public. It is coupled with some potential ethical issues, like privacy of subjects for example. If I run experiment, I gathered some data, I couldn't just throw this data into public access. It is possible that there are other ethical issues, maybe something about privacy of researcher as a person. These issues can make logging research activities a pain.

joppy 8y ago

It’s common practice for scientists to keep logbooks for this exact reason. This is already done.

ordu 8y ago
As I understand that logbooks are not available to other scientists, and if group of highly qualified scientists are making a meta-research, combining results of a dozen of articles (and rejecting two more dozens from their study), they cannot use logbooks of articles to reason about articles.
- joppy 8y ago
  It's true that more open-access logbooks might improve things. But even if you had the logbook, you still wouldn't have the custom equipment, experimental apparatus, experimental subjects, and oversight throughout the experiment which you would need to verify that the logbook itself has data worth a damn. At some point, you have to trust other professionals working in your field to do their work well. Or rather, decide who to trust.
  - ordu 8y ago
    Maybe you are right, but I have a chat with a person who did meta-research and as I understand they dig into articles, looking for methodology, and if they found methodology good enough they would incorporate that article into their meta-research. It impies trust in good intentions of others. But not the blind trust that anything was done by a scientist was done good. If logbook can be used to verify that data processing methods was picked before the first experimental subject came to a lab, then it is already a good sign. It could be much harder to do some p-hacking like [this](https://io9.gizmodo.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800 https://io9.gizmodo.com/i-fooled-millions-into-thinking-choc...).

sn41 8y ago

I am not sure about this... are you talking about experimental research? I am a theoretician. I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues. The advantage of theoretical papers is that the reasoning is a proof/argument developed in the paper itself. With theory, there's less chance for a fraudulent paper, even though the results may have gaps.

naasking 8y ago
> I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues. I think this stigma against making mistakes is a big problem, as is the ego-investment. We're all stumbling around in the dark, and we shouldn't be afraid to admit that.
ordu 8y ago
> I go through several idiotic ideas before hitting on the right one. I'd be embarrassed to admit my silly, initial attempts before my colleagues. Thats the point. If you published your preliminary silly idea and your colleagues didn't find immediately that it is silly, then your not sillier than they are. :) > the reasoning is a proof/argument developed in the paper itself Even when I studied math I found sometimes that just to have proof is not enough to understand. The only way to understand sometimes is to learn how the proof was devised. What the hell author thought to get an idea of some trick in the proof? Some tricks looks as a magic, you see they are true from a logical standpoint, but they make no sense. You cannot understand how they lead to a QED. You can verify proof by logic -- not a problem, but you unable to understand. And if you are unable to understand, then at least you are unable to reproduce this proof at the exam, the only way is to learn it by heart. By I never trained my memory to learn by heart, it is really hard for me and annoying, and moreover I do not believe in learning by heart: to master a subject you need to understand it, not to learn it by heart. My math experience taught me that if I cannot follow the thoughts of other mathematician by just reading his proof, it means that he is smarter than me, so it doesn't matter then how stupid he behaved while searching for his proof, he outsmarted me already. But yes, I was talking about experimental research.

twic 8y ago

It's a bit rich for this to be proposed in a software-centric forum when our industry is merrily rebasing its way to clean linear histories that record the story the author wants to tell rather than what actually happened.

asdff 8y ago

That is the idea behind the lab notebook (what was done) + journal (what was thought) system that people are supposed to be following. We only know about Gregor Mendel because of his meticulous notebooks, for instance. My boss taught me the easiest way to write a paper is to start writing it before your first experiment, references and all; it's a great way to find weaknesses in your approach before you go too far in one direction or another. The problem is people get lazy documenting everything, especially when you are trying to leave the lab as fast as possible at the end of the day :)