4 ms·

Certainly. I'm an anthropologist by training, where extensive fieldwork is the gold standard. Second to this comes a great deal of methodological nerd stuff (IE

Certainly. I'm an anthropologist by training, where extensive fieldwork is the gold standard. Second to this comes a great deal of methodological nerd stuff (IE how to make sure you get good data from all that fieldwork). 'Theory' is mostly subsumed into this, and essentially comes in the form of securing the epistemic quality of the method. I'm passingly familiar with sociology, area studies, cultural studies and linguistics - all of which seem to me, like they also value real world observations in their respective areas much more highly than it sounds like psych does. From your description of these questionaires, it sounds like a methodology issue to me. If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. If your chosen method for collecting the data is already distorting the field, then you're injecting a major risk of just confirming your own bias. I appreciate that psych is trying to get at a different type of knowledge, but from where I'm at this sounds like an "if all you have is a hammer, everything looks like a nail" type of problem. I'm very much not trained to do statistical analysis though, which I believe is part of the toolkit for psych phds - so maybe it actually is the best tool for what they're trying to do.

disgruntledphd2 2y ago
> If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. Generally, you'd run a bunch of interviews to develop a question bank, and then run a bunch of pilots to figure out which set of questions work best. I do agree that there's a big problem in that you base your new measure on how it performs on the old measures, which is problematic if both measures are survey based. This is a real problem in a bunch of psychology research. That being said, the actual problem is the culture. You are required to spout out theory if you want to be taken seriously. I only put a theory chapter into my PhD because I was forced to. Like, I think theory can be useful, but it's putting the cart before the horse, given that we're mostly in the stamp collecting phase of psychology as a science.
- kombookcha 2y ago
  That's a very interesting perspective, thank you. If you don't mind me asking, do you have a sense of why this feels necessary in psych - is it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? How did you go about it, when they made you put in a theory chapter? What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory?
  - disgruntledphd2 2y ago
    > What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory? It would get rejected. I actually replicated a really interesting negative finding (around optimism and health) multiple times with a good design, and couldn't get it published as I didn't have an explanation for why this happened (i do now, but have been in the private sector for well over a decade at this point). > it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? Always seemed like physics envy to me at least.
    - kombookcha 2y ago
      That sounds tough to work in. Do people ever cross-pollinate between theoretical positions and use eachothers data without the competing theory attached, or are they completely siloed off?
      - disgruntledphd2 2y ago
        I dunno really, it honestly seemed like the theory was the point, which is so ass backwards that I just couldn't handle it. Note that this all happened before the replication crisis, but I 100% wasn't surprised, having been shocked and appalled by the statistical methods used in "top" papers. As Andrew Gelman notes, peer review doesn't help if all of your peers are wrong about the same things.

disgruntledphd2 2y ago

> If your baseline is a questionaire, then the questions you would think to ask are going to be informed by your preconceived notions of the issue, and you're forcing the raw data out there to conform to these formats you've set for it. Generally, you'd run a bunch of interviews to develop a question bank, and then run a bunch of pilots to figure out which set of questions work best. I do agree that there's a big problem in that you base your new measure on how it performs on the old measures, which is problematic if both measures are survey based. This is a real problem in a bunch of psychology research. That being said, the actual problem is the culture. You are required to spout out theory if you want to be taken seriously. I only put a theory chapter into my PhD because I was forced to. Like, I think theory can be useful, but it's putting the cart before the horse, given that we're mostly in the stamp collecting phase of psychology as a science.

kombookcha 2y ago
That's a very interesting perspective, thank you. If you don't mind me asking, do you have a sense of why this feels necessary in psych - is it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? How did you go about it, when they made you put in a theory chapter? What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory?
- disgruntledphd2 2y ago
  > What would happen if you turned in some psych work that was robust in its method, but without explicit references to theory? It would get rejected. I actually replicated a really interesting negative finding (around optimism and health) multiple times with a good design, and couldn't get it published as I didn't have an explanation for why this happened (i do now, but have been in the private sector for well over a decade at this point). > it deference to some preferred school of thought that needs to sign off on your work, or is it more that you need to signal that you know lots of theory? Always seemed like physics envy to me at least.
  - kombookcha 2y ago
    That sounds tough to work in. Do people ever cross-pollinate between theoretical positions and use eachothers data without the competing theory attached, or are they completely siloed off?
    - disgruntledphd2 2y ago
      I dunno really, it honestly seemed like the theory was the point, which is so ass backwards that I just couldn't handle it. Note that this all happened before the replication crisis, but I 100% wasn't surprised, having been shocked and appalled by the statistical methods used in "top" papers. As Andrew Gelman notes, peer review doesn't help if all of your peers are wrong about the same things.