3 ms·
Oh yes, I do agree that the sample size is problematic, but that's honestly not uncommon in anything with medical research. Samples sizes are always crazy small
by godelski 3y ago
Oh yes, I do agree that the sample size is problematic, but that's honestly not uncommon in anything with medical research. Samples sizes are always crazy small and people are making wildly too large of conclusions from them given that. But it's also not like you can expect to form a hypothesis and then get 50k people to participate in a study. You gotta show that there's something to the idea first. That's what these kinds of papers are about.
And for the dropoff rate, I'm less concerned with that than 1) the (different) bias introduced by the people who chose to drop out vs not and 2) the overall sample size being smaller. Were people that dropped out uniformly distributed and there was still a large sample size, it wouldn't be of any concern. The statistical error you're making here is actually the same as the Monte Hall one. Basically, ignore the previous group and pretend they don't exist. Though there probably is a bias introduced from the dropout (which isn't biased in the Monte Hall problem), but there's so many other effects going on here that it's hard to say. (And medical students and human phys students aren't known for having the strongest stats skills. To be fair, it's a fucking tough field and they got a lot of extra factors that don't make it any easier)
But again, let's never think of these works as proofs of effects but rather like a proof of concept. Even if they didn't have any dropout I'd say the same thing here. Evidence is good, more people makes the evidence stronger, but you gotta keep pushing for stronger and stronger evidence but university research papers in human studies are just never strong evidence.
I am a researcher. I do think it is good that you are being critical, but it's often too easy to be overly critical and is a mistake a lot of junior people make. There are ALWAYS mistakes and are ALWAYS things to poke holes in. But that's not the point of publishing. The point of publishing is to communicate to your peers. So in that respect being overly critical is a hindrance to scientific progress. It's because the context matters. This actually ties into my other comment in the thread fwiw. Just make sure to see things for what they are (it's why I basically ignore articles that talk about papers now, because they do the same thing but usually vastly over exaggerating the conclusions.)