3 ms·

"I personally think it's fair to compare social science research with monks researching a religious text. They can work hard for years without ever coming up wi

"I personally think it's fair to compare social science research with monks researching a religious text. They can work hard for years without ever coming up with a new idea." Based on my own experiences, I really don't think this applies to other social sciences very well at all. I'm not particularly familiar with psych papers, which may well be mostly based on text-on-text, but in all the social science disciplines I am familiar with there's a great deal of focus on fieldwork and diving into the messy nitty-gritty of the actual world. Is psych uniquely constrained from doing this somehow (patient ethics?), or is it a particular culture in the psych departments causing this?

huijzer 2y ago
> but in all the social science disciplines I am familiar with there's a great deal of focus on fieldwork and diving into the messy nitty-gritty of the actual world. Can you give examples? What I've seen is that questionnaires with high "validity" and "reliability" are put out to participants and then statistical tests are conducted to re-confirm significant results. However, these significant results typically do not mean that the results are practically useful. Maybe on average the results are significant, but the effect is too small to really be able to pick out individuals.
- kombookcha 2y ago
  Certainly. I'm an anthropologist by training, where extensive fieldwork is the gold standard. Second to this comes a great deal of methodological nerd stuff (IE how to make sure you get good data from all that fieldwork). 'Theory' is mostly subsumed into this, and essentially comes in the form of securing the epistemic quality of the method. I'm passingly familiar with sociology, area studies, cultural studies and linguistics - all of which seem to me, like they also value real world observations in their respective areas much more highly than it sounds like psych does. From your description of these questionaires, it sounds like a methodology issue to me. If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. If your chosen method for collecting the data is already distorting the field, then you're injecting a major risk of just confirming your own bias. I appreciate that psych is trying to get at a different type of knowledge, but from where I'm at this sounds like an "if all you have is a hammer, everything looks like a nail" type of problem. I'm very much not trained to do statistical analysis though, which I believe is part of the toolkit for psych phds - so maybe it actually is the best tool for what they're trying to do.
  - disgruntledphd2 2y ago
    > If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. Generally, you'd run a bunch of interviews to develop a question bank, and then run a bunch of pilots to figure out which set of questions work best. I do agree that there's a big problem in that you base your new measure on how it performs on the old measures, which is problematic if both measures are survey based. This is a real problem in a bunch of psychology research. That being said, the actual problem is the culture. You are required to spout out theory if you want to be taken seriously. I only put a theory chapter into my PhD because I was forced to. Like, I think theory can be useful, but it's putting the cart before the horse, given that we're mostly in the stamp collecting phase of psychology as a science.
    - kombookcha 2y ago
      That's a very interesting perspective, thank you. If you don't mind me asking, do you have a sense of why this feels necessary in psych - is it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? How did you go about it, when they made you put in a theory chapter? What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory?
      - disgruntledphd2 2y ago
        > What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory? It would get rejected. I actually replicated a really interesting negative finding (around optimism and health) multiple times with a good design, and couldn't get it published as I didn't have an explanation for why this happened (i do now, but have been in the private sector for well over a decade at this point). > it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? Always seemed like physics envy to me at least.

huijzer 2y ago

> but in all the social science disciplines I am familiar with there's a great deal of focus on fieldwork and diving into the messy nitty-gritty of the actual world. Can you give examples? What I've seen is that questionnaires with high "validity" and "reliability" are put out to participants and then statistical tests are conducted to re-confirm significant results. However, these significant results typically do not mean that the results are practically useful. Maybe on average the results are significant, but the effect is too small to really be able to pick out individuals.

kombookcha 2y ago
Certainly. I'm an anthropologist by training, where extensive fieldwork is the gold standard. Second to this comes a great deal of methodological nerd stuff (IE how to make sure you get good data from all that fieldwork). 'Theory' is mostly subsumed into this, and essentially comes in the form of securing the epistemic quality of the method. I'm passingly familiar with sociology, area studies, cultural studies and linguistics - all of which seem to me, like they also value real world observations in their respective areas much more highly than it sounds like psych does. From your description of these questionaires, it sounds like a methodology issue to me. If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. If your chosen method for collecting the data is already distorting the field, then you're injecting a major risk of just confirming your own bias. I appreciate that psych is trying to get at a different type of knowledge, but from where I'm at this sounds like an "if all you have is a hammer, everything looks like a nail" type of problem. I'm very much not trained to do statistical analysis though, which I believe is part of the toolkit for psych phds - so maybe it actually is the best tool for what they're trying to do.
- disgruntledphd2 2y ago
  > If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. Generally, you'd run a bunch of interviews to develop a question bank, and then run a bunch of pilots to figure out which set of questions work best. I do agree that there's a big problem in that you base your new measure on how it performs on the old measures, which is problematic if both measures are survey based. This is a real problem in a bunch of psychology research. That being said, the actual problem is the culture. You are required to spout out theory if you want to be taken seriously. I only put a theory chapter into my PhD because I was forced to. Like, I think theory can be useful, but it's putting the cart before the horse, given that we're mostly in the stamp collecting phase of psychology as a science.
  - kombookcha 2y ago
    That's a very interesting perspective, thank you. If you don't mind me asking, do you have a sense of why this feels necessary in psych - is it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? How did you go about it, when they made you put in a theory chapter? What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory?
    - disgruntledphd2 2y ago
      > What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory? It would get rejected. I actually replicated a really interesting negative finding (around optimism and health) multiple times with a good design, and couldn't get it published as I didn't have an explanation for why this happened (i do now, but have been in the private sector for well over a decade at this point). > it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? Always seemed like physics envy to me at least.