4 ms·

Important limitations of the study are hidden behind the paywall. From the Methods section in the full paper: From Study 1: "Participants (N = 213, 109 women)

Important limitations of the study are hidden behind the paywall. From the Methods section in the full paper: From Study 1: "Participants (N = 213, 109 women) were recruited from a small Northeastern college in exchange for partial course credit or a lottery entry for a $40 gift card. Of these participants, 190 (95 women) completed all focal variables and were included in further analyses. Sample size was determined by the number of participants researchers were able to recruit over the course of one semester. Sensitivity analyses indicated we have 0.80 power to detect an effect size of partial η2 = 0.076 for focal predictions. Participants ranged in age from 18 to 23 (M = 19.82 years, SD = 1.31). The majority of participants identified as White (75%), 14% identified as Asian, 4% as Hispanic/Latino, and 3% as Black." From Study 2: "U.S. Participants (N = 306, 141 women) were recruited through Amazon's Mechanical Turk and received $1.00 compensation. Given our shift from ‘ideal’ to actual friends, we anticipated a reduction in the effect size of our predicted sex differences and aimed to recruit a 50% larger sample than that of Study 1. Sensitivity analyses indicated that our sample size allowed us to detect effect sizes of partial η2 = 0.003 with 0.80 power. Participants ranged in age from 18 to 84 (M = 35.22, SD = 11.29) and primarily identified as White (74%) or Black (9%)." From Study 3: "U.S. Participants (N = 250; 97 women) were recruited through TurkPrime and received $1.00 for completing the study. Participants ranged in age from 18 to 70 (M = 34.43, SD = 9.88). Sensitivity analysis indicated that we were able to detect small effects (f < 0.10) with 0.80 power assuming 0.5 correlation between measures. The majority of participants identified as White (70%), 11% identified as Black, 7% as Asian, 6% as Hispanic/Latinx, 2% as multi-racial, 1% as American Indian, and 1% as Pacific Islander." I hope that a user with more research experience than me can chime in. In the meantime, it looks like the study's conclusions mainly apply to: -Undergraduate students in a specific American university located in the Northeast and -People accepting paid jobs on Mechanical Turk.

aradox66 5y ago
xapata 5y ago
It's reasonable to worry about confounding factors, but one should hypothesize what they are as part of the criticism.
- openknot 5y ago
  To improve my criticism (open to correction, especially because I don't have a background in academic psychology), I hypothesize that the paper's results don't apply and can't be generalized to the broader U.S. population. For Study 1, I specifically think that one's views on friendship are shaped by the people around them (so the views of undergraduates on friendship in a small Northeastern college might be different than a large state school in Texas or California). Views on friendship may also change after graduating university (where it's harder to make friends), and may also be different than views from people who have never attended (e.g. people in small towns in a trade or who have spent a career enlisted in the military). For Studies 2-3 (this is likely where my reasoning is shakiest), I hypothesize that people on Mechanical Turk represent a small subset of the U.S. population, and most have specific shared beliefs (that motivate them to trade time doing fairly simple tasks for money). It's possible that some people are on Mechanical Turk for fun or specifically to learn more about research by participating in online studies, but I hypothesize that this is a negligible part of the population. In short, I'm not convinced that the study's design makes its conclusions applicable to most women and men in the United States population.
  - xapata 5y ago
    The fact that a sample is a tiny part of the population is irrelevant. The question is how that sample differs from the population in a way that correlates with the explained/dependant variable, but is not included as an explanatory/independent variable. Your point that views on friendship may change over time is a strong criticism. The criticism of the Turk selection is weak as written. How does spending time on Turk relate to friendship preferences?
    - openknot 5y ago
      To clarify the rationale: I have a hunch that men spending time on Mechanical Turk are either less financially secure or care more about maximizing the money one earns from spare time. This may cause these men to prefer male friends who can help them make more money or raise their social status to help them find better work. Therefore, a sample of men who do not need to worry about money (and thus do not feel a want to use Mechanical Turk) may not feel a need to choose friends based on their usefulness to their financial or social status, and instead prefer male friends who are trustworthy and provide a higher level of emotional support. My hunch is backed up by this segment of a review paper from 2017 [0]: "Further, half of the MTurkers are employed part-time or unemployed. This is consistent with past findings (e.g., Ross et al., 2010; Behrend et al., 2011). A population of unemployed or underemployed participants offers a potentially interesting sampling opportunity for research questions related to job search behavior, underemployment, and unemployment (Roulin, 2015). At the same time, this could explain why MTurkers have a lower income on average as compared to nationally representative samples. "Taken together, our conclusions echo the conclusions of other researchers (cf., Behrend et al., 2011; Buhrmester et al., 2011; Berinsky et al., 2012; Casler et al., 2013; Brandon et al., 2014; Woo et al., 2015): MTurk samples are not representations of the general population." [0] From "Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations" published in the Frontiers in Psychology journal: https://doi.org/10.3389/fpsyg.2017.01359 https://doi.org/10.3389/fpsyg.2017.01359

aradox66 5y ago

xapata 5y ago

It's reasonable to worry about confounding factors, but one should hypothesize what they are as part of the criticism.

openknot 5y ago
To improve my criticism (open to correction, especially because I don't have a background in academic psychology), I hypothesize that the paper's results don't apply and can't be generalized to the broader U.S. population. For Study 1, I specifically think that one's views on friendship are shaped by the people around them (so the views of undergraduates on friendship in a small Northeastern college might be different than a large state school in Texas or California). Views on friendship may also change after graduating university (where it's harder to make friends), and may also be different than views from people who have never attended (e.g. people in small towns in a trade or who have spent a career enlisted in the military). For Studies 2-3 (this is likely where my reasoning is shakiest), I hypothesize that people on Mechanical Turk represent a small subset of the U.S. population, and most have specific shared beliefs (that motivate them to trade time doing fairly simple tasks for money). It's possible that some people are on Mechanical Turk for fun or specifically to learn more about research by participating in online studies, but I hypothesize that this is a negligible part of the population. In short, I'm not convinced that the study's design makes its conclusions applicable to most women and men in the United States population.
- xapata 5y ago
  The fact that a sample is a tiny part of the population is irrelevant. The question is how that sample differs from the population in a way that correlates with the explained/dependant variable, but is not included as an explanatory/independent variable. Your point that views on friendship may change over time is a strong criticism. The criticism of the Turk selection is weak as written. How does spending time on Turk relate to friendship preferences?
  - openknot 5y ago
    To clarify the rationale: I have a hunch that men spending time on Mechanical Turk are either less financially secure or care more about maximizing the money one earns from spare time. This may cause these men to prefer male friends who can help them make more money or raise their social status to help them find better work. Therefore, a sample of men who do not need to worry about money (and thus do not feel a want to use Mechanical Turk) may not feel a need to choose friends based on their usefulness to their financial or social status, and instead prefer male friends who are trustworthy and provide a higher level of emotional support. My hunch is backed up by this segment of a review paper from 2017 [0]: "Further, half of the MTurkers are employed part-time or unemployed. This is consistent with past findings (e.g., Ross et al., 2010; Behrend et al., 2011). A population of unemployed or underemployed participants offers a potentially interesting sampling opportunity for research questions related to job search behavior, underemployment, and unemployment (Roulin, 2015). At the same time, this could explain why MTurkers have a lower income on average as compared to nationally representative samples. "Taken together, our conclusions echo the conclusions of other researchers (cf., Behrend et al., 2011; Buhrmester et al., 2011; Berinsky et al., 2012; Casler et al., 2013; Brandon et al., 2014; Woo et al., 2015): MTurk samples are not representations of the general population." [0] From "Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations" published in the Frontiers in Psychology journal: https://doi.org/10.3389/fpsyg.2017.01359 https://doi.org/10.3389/fpsyg.2017.01359