9 ms·

Teenager on TikTok Disrupted Thousands of Scientific Studies with a Single Video

akjssdk 5y ago
From the article "Prolific, a tool for scientists conducting behavioral research, had no screening tools in place to make sure that it delivered representative population samples to each study.". I would argue that studies from this tool were already disrupted if you are not screening your users in some way.
- milliams 5y ago
  Yes, that just means that any studies based on this platform were already wrong, it's just that no one knew.
corobo 5y ago
The TikToker sent too many female-identifying people aged around 21 To fix it they've released a press release to tech blogs which will bring older and more male-identifying people in Very meta
- implements 5y ago
  Off-topic, but do you always use the “sex identifying” construction in place of “men / women”? - it seems a bit inelegant, and if you hold “TWAW” as true also unnecessary. Not intended as snarky, but I was slightly jarred by the ACLU’s removal of the word “woman” from a Ruth Bader Ginsburg quote yesterday, and wondered whether this was now ‘a thing’ (at least in the US).
  - corobo 5y ago
    Just trying to be inclusive tbh I try to remove gender references in all of my comments, but in this instance it was relevant so I went with the most inclusive option I could think of No idea if that's a thing for you folks, I'm in the UK. I'd have to look up ACLU to see what it stands for
    - greyivy 5y ago
      Your mindfulness is appreciated!
    - robbedpeter 5y ago
      Should probably just use "men" and "women", boys and girls, unless otherwise necessary, for clarity and simplicity. Someone offended by a lack of preemptive qualifying verbiage is not worth placating. If they ask for different words, you can add them in later conversation. Preemptively tacking on qualifiers can bury the actual meaning of the idea you're trying to communicate.
      - corobo 5y ago
        Can or did? I'd rather placate the people that want to be involved than placate those that would exclude them, even if that is what is happening. If I'm doing that wrong, I'll change my ways. e.g the person saying that it hints towards folks not being men or women if they have transitioned and we use different words which is a fair point, I'll have to look into that. As for losing the meaning of my comment - A lot of my comments get downvoted because I'm crap at getting an idea down without editing it (we're on the 5th edit as I write). That is nothing new. If my comment is meaningless give it a downvote
    - ASalazarMX 5y ago
      Wouldn't it be detrimental if the topic was sex? you'd have to specify "female-identifying people born with a vagina". I'm not kidding, recently I read advice for lube for "people born with a vagina". Sex is not gender, it's detrimental for communication to mix the two.
      - corobo 5y ago
        Two people have said it is detrimental to communication so far yet nobody has requested any form of clarification on what I said If anything my frequent lack of punctuation needs correcting first, that has caused infinitely more confusion. Weirdly in 20 years of forum use I've never had anyone start a discussion about it.. :)
niij 5y ago
If linking to a public study can "break" it, it was already broken.
- rtkwe 5y ago
  The tool was bad but the data could still be good if they were getting representative samples.
sct202 5y ago
This is odd considering Prolific makes users input a bunch of demographic info (age, sex, location, education, ethnicity/race, etc) so that researchers can target specific demographics for their surveys. Did the researchers just assume that Prolific would deliver something balanced on it's own?
- indrax 5y ago
  I just signed up to prolific. Their onboarding tutorial/example survey references prescreening and eligibility. I'm guessing it was on researchers to filter the demographics to what was relevant to their study. I think this is actually the right approach. Some 'balancing' algorithm would have to make assumptions about what's normal, then then the excluded participants wouldn't even show up in the research data.
  - psb31 5y ago
    I think yes, since that’s what it roughly did normally if you didn’t balance. This was a change in the normal behaviour, so if you ran a study from one week the next you wouldn’t have known that your data would change dramatically without additional screening.
major505 5y ago
“Young women who enjoy TikTok are people too" I don't think people who use tik tok should be classified as a human. I'm ready to die in this hill.
- depaya 5y ago
  What a stupid hill to die on.
  - robbedpeter 5y ago
    No, no, he said "die in." It's a great hill to die in, probably lots of twisty tunnels to explore before being eaten by a gru.
Zababa 5y ago
There's a silver lining: a few decades ago, we mostly knew about the psychology of college students. After this, we knew about the psychology of Mechanical Turk workers. Now we also know the psychology of 21 years old female TikTok users. That's what I call progress! In a few centuries we might know about actual human psychology.

From the article "Prolific, a tool for scientists conducting behavioral research, had no screening tools in place to make sure that it delivered representative population samples to each study.". I would argue that studies from this tool were already disrupted if you are not screening your users in some way.

milliams 5y ago
Yes, that just means that any studies based on this platform were already wrong, it's just that no one knew.

corobo 5y ago

The TikToker sent too many female-identifying people aged around 21 To fix it they've released a press release to tech blogs which will bring older and more male-identifying people in Very meta

implements 5y ago
Off-topic, but do you always use the “sex identifying” construction in place of “men / women”? - it seems a bit inelegant, and if you hold “TWAW” as true also unnecessary. Not intended as snarky, but I was slightly jarred by the ACLU’s removal of the word “woman” from a Ruth Bader Ginsburg quote yesterday, and wondered whether this was now ‘a thing’ (at least in the US).
- corobo 5y ago
  Just trying to be inclusive tbh I try to remove gender references in all of my comments, but in this instance it was relevant so I went with the most inclusive option I could think of No idea if that's a thing for you folks, I'm in the UK. I'd have to look up ACLU to see what it stands for
  - greyivy 5y ago
    Your mindfulness is appreciated!
  - robbedpeter 5y ago
    Should probably just use "men" and "women", boys and girls, unless otherwise necessary, for clarity and simplicity. Someone offended by a lack of preemptive qualifying verbiage is not worth placating. If they ask for different words, you can add them in later conversation. Preemptively tacking on qualifiers can bury the actual meaning of the idea you're trying to communicate.
    - corobo 5y ago
      Can or did? I'd rather placate the people that want to be involved than placate those that would exclude them, even if that is what is happening. If I'm doing that wrong, I'll change my ways. e.g the person saying that it hints towards folks not being men or women if they have transitioned and we use different words which is a fair point, I'll have to look into that. As for losing the meaning of my comment - A lot of my comments get downvoted because I'm crap at getting an idea down without editing it (we're on the 5th edit as I write). That is nothing new. If my comment is meaningless give it a downvote
  - ASalazarMX 5y ago
    Wouldn't it be detrimental if the topic was sex? you'd have to specify "female-identifying people born with a vagina". I'm not kidding, recently I read advice for lube for "people born with a vagina". Sex is not gender, it's detrimental for communication to mix the two.
    - corobo 5y ago
      Two people have said it is detrimental to communication so far yet nobody has requested any form of clarification on what I said If anything my frequent lack of punctuation needs correcting first, that has caused infinitely more confusion. Weirdly in 20 years of forum use I've never had anyone start a discussion about it.. :)

niij 5y ago

If linking to a public study can "break" it, it was already broken.

rtkwe 5y ago
The tool was bad but the data could still be good if they were getting representative samples.

sct202 5y ago

This is odd considering Prolific makes users input a bunch of demographic info (age, sex, location, education, ethnicity/race, etc) so that researchers can target specific demographics for their surveys. Did the researchers just assume that Prolific would deliver something balanced on it's own?

indrax 5y ago
I just signed up to prolific. Their onboarding tutorial/example survey references prescreening and eligibility. I'm guessing it was on researchers to filter the demographics to what was relevant to their study. I think this is actually the right approach. Some 'balancing' algorithm would have to make assumptions about what's normal, then then the excluded participants wouldn't even show up in the research data.
- psb31 5y ago
  I think yes, since that’s what it roughly did normally if you didn’t balance. This was a change in the normal behaviour, so if you ran a study from one week the next you wouldn’t have known that your data would change dramatically without additional screening.

major505 5y ago

“Young women who enjoy TikTok are people too" I don't think people who use tik tok should be classified as a human. I'm ready to die in this hill.

depaya 5y ago
What a stupid hill to die on.
- robbedpeter 5y ago
  No, no, he said "die in." It's a great hill to die in, probably lots of twisty tunnels to explore before being eaten by a gru.

Zababa 5y ago

There's a silver lining: a few decades ago, we mostly knew about the psychology of college students. After this, we knew about the psychology of Mechanical Turk workers. Now we also know the psychology of 21 years old female TikTok users. That's what I call progress! In a few centuries we might know about actual human psychology.