3 ms·

It being extremely difficult or impossible to design a convincing study doesn't make the other ones convincing.

Timon3 3y ago
I am not saying that any study is or is not convincing. I am literally just asking: how could you define a test for this? I am not stating that any individual study is good or bad or anything in between. I want to know: if I design a study, how do I design it in a way that makes it a good study in regards to this question?
- bee_rider 3y ago
  I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. So, if posed with that question, I would not have a good response. Despite this, I would still believe the dropped piece of paper experiment was not very good.
  - Timon3 3y ago
    > I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. I am allowed to ask, right? It doesn't seem crazy to me that when someone criticizes a study, someone else will ask "how could you solve the problem you describe?". I would like to know if someone has any idea on how to falsify it, because if someone were to find one, it might convince me of the criticism. > Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. I don't think that is a good comparison. A better one would be if there is an experiment, and then someone comments on the results by saying "well, it could have been dark matter particles interacting in some way". It might as well be true, but it's not definitive if they don't explain how to measure and isolate that. Someone else asking for how they would design the experiment to take this into account seems like a completely reasonable ask to me.
    - joebob42 3y ago
      I don't think you're asking an unfair question. Its now been answered, by many people, to the effect of "this study isn't convincing, but we don't see a simple way to design a convincing one".
      - Timon3 3y ago
        I suppose it has. The answer is (hopefully understandably) disappointing, since it can be used to discard any scientific efforts involving subjects, but it's what I expected.
- ang_cire 3y ago
  You'd need the participants not to know they are part of a study. They'd need to be in a situation where they believe that the statement of their true beliefs is critical. This isn't impossible (many studies do use primary sources which were not created for the purposes of a study, and where there was some incentive based on real-world circumstances for them to be truthful), but it is much more difficult and leaves gaps where insufficient primary sources materials addressing the questions of the study are available. Psychologists are well aware that the current model for many psychology studies, which consists of either asking an extremely small sample of participants to sit in a computer lab at a university for 20 minutes answering questions, or sending out very loosely-targeted emails soliciting participation, are both very inefficient. But they gotta get published, cause that's how you put food on the table.

I am not saying that any study is or is not convincing. I am literally just asking: how could you define a test for this? I am not stating that any individual study is good or bad or anything in between. I want to know: if I design a study, how do I design it in a way that makes it a good study in regards to this question?

bee_rider 3y ago
I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. So, if posed with that question, I would not have a good response. Despite this, I would still believe the dropped piece of paper experiment was not very good.
- Timon3 3y ago
  > I don’t think it is reasonable to expect for a person on in a sort of informal online conversation like this to be able to define a bulletproof experiment for the topic. I am allowed to ask, right? It doesn't seem crazy to me that when someone criticizes a study, someone else will ask "how could you solve the problem you describe?". I would like to know if someone has any idea on how to falsify it, because if someone were to find one, it might convince me of the criticism. > Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist. I don't think that is a good comparison. A better one would be if there is an experiment, and then someone comments on the results by saying "well, it could have been dark matter particles interacting in some way". It might as well be true, but it's not definitive if they don't explain how to measure and isolate that. Someone else asking for how they would design the experiment to take this into account seems like a completely reasonable ask to me.
  - joebob42 3y ago
    I don't think you're asking an unfair question. Its now been answered, by many people, to the effect of "this study isn't convincing, but we don't see a simple way to design a convincing one".
    - Timon3 3y ago
      I suppose it has. The answer is (hopefully understandably) disappointing, since it can be used to discard any scientific efforts involving subjects, but it's what I expected.
ang_cire 3y ago
You'd need the participants not to know they are part of a study. They'd need to be in a situation where they believe that the statement of their true beliefs is critical. This isn't impossible (many studies do use primary sources which were not created for the purposes of a study, and where there was some incentive based on real-world circumstances for them to be truthful), but it is much more difficult and leaves gaps where insufficient primary sources materials addressing the questions of the study are available. Psychologists are well aware that the current model for many psychology studies, which consists of either asking an extremely small sample of participants to sit in a computer lab at a university for 20 minutes answering questions, or sending out very loosely-targeted emails soliciting participation, are both very inefficient. But they gotta get published, cause that's how you put food on the table.