4 ms·

My doctoral degree has required training in both Computer Science and Social Science, and regularly interact with both communities. Drawing on this, I feel that

My doctoral degree has required training in both Computer Science and Social Science, and regularly interact with both communities. Drawing on this, I feel that this comment is unnecessarily partisan and divisive. > 2) Mentioning the replication crisis in social science gets a lot of the social science academics observably upset and defensive. The replication crisis is widely discussed in social sciences. Nearly every major social science journal will have at least one, likely many, editorials and articles on the crisis tailored to their field. Maybe this is explicitely a Management Studies thing? The field is relatively small and insular, and so cannot be generalized to wider social science. And are your experiences drawon from the community as a whole, or the faculty in your department? > 3) I have enjoyed my time spent with CS academics a lot more, luckily have some interaction even now - the ones I have spoken to seem to lack the ego-driven blind spots the social scientists exhibit. The CS group seems to have a lot more fun with their research. Maybe this is just the local community of wherever you studied? I've met equal parts ego-driven and chill people in both CS and Social Science. A major concept in most social science fields is sampling—I'd urge you to consider how representative the people you are talking to actually are, and how your sample might be biased from the global population. My prior is that there would be little difference in personality, in aggregate, between the fields. > I.e. I have observed that even when a manager or leader wants to hire developers, social science damaged managers and HR will actively sabotage these efforts because they are still, fundamentally, driven by Taylorist, control fetish concepts (which they refuse to accept) I mean, that's one possibility. The other is that hiring people is difficult and expensive, and that credentials are a quick and useful, if flawed, method of sorting through the pile. And are most HR people really trained in Management? I honestly don't know, but I doubt that the typical Psychology or Sociology major, for instance, is going to know of, remember, or consider using Taylorist management theory. It definitely seems that you enjoyed your time in CS more than your time in Management. That's great! CS is an important field with lots of cool people. But generalizing this preference into a wider philosophy that privileges CS at the expense of (all?) social sciences is silly.

hndudette2 6y ago
Regarding the replication crisis, can I ask why we don't yet have a central body to which scientists report their intended experimental design, sample size and research hypothesis before the experiment begins? Then we can eliminate publication bias due to null results not being published which is a big part of the replication crisis. The meta-analyses only survey those studies that reported this ahead of time and the scope of possible publication bias can be quantified.
- pmyteh 6y ago
  There are registration databases, but there are issues. Who trawls the (now huge) database to find the registered studies which never reported? In any case, that could indicate a file drawer problem publication bias or simply that the funding disappeared or a researcher decided not to proceed. How do you move from very spotty preregistration to compulsory preregistration? It would probably take a coordinated push from all the major funders. And they haven't successfully moved to fully open access publication in over a decade of trying, despite the obvious financial gain to the non-publisher world of winning at that coordination game. How do you handle work on the margins of registerability? Not all scientific work is testing hypotheses, some is exploratory. Should a quantitative description be ruled out if it wasn't registered? If so, how are reasonable hypotheses for future work established? If not, you have to be extra vigilant to stop people smuggling phrases that imply confirmation into nominally exploratory work. And the other problem with the replication crisis is that even decently-planned, pre-registered, straightforward studies can not replicate. Sometimes for reasonable reasons (significance was actually by chance, some unanticipated confounder interfered) and sometimes for thoroughly blameworthy ones (rather than p-hacking you can just fudge the data, or make 'accidental' programming errors). So I'm all in favour of pre-registration, but it's not a magic bullet.
  - hndudette2 6y ago
    We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going. All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet. Who trawls the (now huge) database Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis. How do you move from very spotty preregistration to compulsory preregistration? Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned. How do you handle work on the margins of registerability? Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.

hndudette2 6y ago

Regarding the replication crisis, can I ask why we don't yet have a central body to which scientists report their intended experimental design, sample size and research hypothesis before the experiment begins? Then we can eliminate publication bias due to null results not being published which is a big part of the replication crisis. The meta-analyses only survey those studies that reported this ahead of time and the scope of possible publication bias can be quantified.

pmyteh 6y ago
There are registration databases, but there are issues. Who trawls the (now huge) database to find the registered studies which never reported? In any case, that could indicate a file drawer problem publication bias or simply that the funding disappeared or a researcher decided not to proceed. How do you move from very spotty preregistration to compulsory preregistration? It would probably take a coordinated push from all the major funders. And they haven't successfully moved to fully open access publication in over a decade of trying, despite the obvious financial gain to the non-publisher world of winning at that coordination game. How do you handle work on the margins of registerability? Not all scientific work is testing hypotheses, some is exploratory. Should a quantitative description be ruled out if it wasn't registered? If so, how are reasonable hypotheses for future work established? If not, you have to be extra vigilant to stop people smuggling phrases that imply confirmation into nominally exploratory work. And the other problem with the replication crisis is that even decently-planned, pre-registered, straightforward studies can not replicate. Sometimes for reasonable reasons (significance was actually by chance, some unanticipated confounder interfered) and sometimes for thoroughly blameworthy ones (rather than p-hacking you can just fudge the data, or make 'accidental' programming errors). So I'm all in favour of pre-registration, but it's not a magic bullet.
- hndudette2 6y ago
  We both probably agree that parapsychology hasn't demonstrated that psychic phenomenon is real, and yet their various meta-analyses purport to show that it is. The experimental design of the underlying studies is quite good, so it must be publication bias. I couldn't think of a better demonstration as to why preregistration is a critical necessity, even if it is difficult to get going. All of your questions are very good ones however, and I agree with your conclusion that it isn't going to be a silver bullet. Who trawls the (now huge) database Only the authors of the meta-analyses, as part of their lit review. It's simply a criteria for inclusion into meta-analysis. How do you move from very spotty preregistration to compulsory preregistration? Eventually, if meta-analyses only look at studies that are preregistered, this would boost citation count for preregistered studies and therefore incent scientists to preregister since their careers are tied to citation count. How to arrive at this end state is a bit of a chicken and egg problem and requires a cultural shift, so it'll be very difficult. But once the end state is arrived at it should be sticky without needing to be compulsory, since career incentives are aligned. How do you handle work on the margins of registerability? Preregistration is mostly helpful for research questions that are amenable to statistical meta-analysis, i.e. those questions where the hypotheses tested across papers are sufficiently similar so the quantitative results can be statistically combined and statistical significance & clinical significance can be evaluated. I think preregistration is mostly needed for these questions, although it's probably helpful elsewhere too.