3 ms·

I looked at the paper's abstract [1], and this is a conclusion made from just two talks/interviews by the authors of the study? Should probably replicate it bef

I looked at the paper's abstract [1], and this is a conclusion made from just two talks/interviews by the authors of the study? Should probably replicate it before telling millions of people to buy hundreds of dollars of audio equipment... I wasn't able to find an open access copy of the study, though I suppose there's always sci-hub. 1: https://journals.sagepub.com/doi/10.1177/1075547018759345 https://journals.sagepub.com/doi/10.1177/1075547018759345

gwern 5y ago
Yes, Sci-Hub works fine here. Since you apparently don't deign to use it, I've mirrored it here: https://www.gwern.net/docs/design/2018-newman.pdf https://www.gwern.net/docs/design/2018-newman.pdf
- Strilanc 5y ago
  Oh no I love sci-hub but I second-guess linking it. Thanks for the mirror! So, reading the paper proper, I was wrong about it being the authors' talks. They picked existing engineering talks and interviews. It'd be nice to be able to actually see the clips to get a sense of how much they changed the quality. They say they put the clips on youtube for the test but don't take the opportunity to link the clips. In the Science Friday graph all the error bars overlap so meh. For the combined technical talks the error bars don't overlap and they're all in the same direction, which is way more encouraging, though I really wish they'd tried it with a pool of more talks to remove that systemic effect. I also worry about the fact that random mechanical turks may differ from the usual audience for a technical talk in important ways (e.g. not knowing the subject and so having nothing to go on besides audio quality). They say in a note at the end of the paper that they tried filtering by correctly answering a question at the end, and that the effect was still there, but they don't show the data for it or say if it affected the strength. Overall I feel it's a reasonable methodology and certainly suggestive results, if a bit lacking in the exact methodological details and the number of talks they tried it on. And it's not like I have a prior that audio quality doesn't matter...
  - gwern 5y ago
    Yeah, that was my overall take: reasonably convincing, neither a dumpster fire nor a slamdunk. These days I don't believe these sorts of psychology effects until I see a Registered Report, but this also isn't the hot steaming garbage you'd expect from the OP summary. It's more or less what you'd expect from a small but not super-small sample investigating a moderate real effect across the board which does seem pretty plausible a priori.

gwern 5y ago

Yes, Sci-Hub works fine here. Since you apparently don't deign to use it, I've mirrored it here: https://www.gwern.net/docs/design/2018-newman.pdf https://www.gwern.net/docs/design/2018-newman.pdf

Strilanc 5y ago
Oh no I love sci-hub but I second-guess linking it. Thanks for the mirror! So, reading the paper proper, I was wrong about it being the authors' talks. They picked existing engineering talks and interviews. It'd be nice to be able to actually see the clips to get a sense of how much they changed the quality. They say they put the clips on youtube for the test but don't take the opportunity to link the clips. In the Science Friday graph all the error bars overlap so meh. For the combined technical talks the error bars don't overlap and they're all in the same direction, which is way more encouraging, though I really wish they'd tried it with a pool of more talks to remove that systemic effect. I also worry about the fact that random mechanical turks may differ from the usual audience for a technical talk in important ways (e.g. not knowing the subject and so having nothing to go on besides audio quality). They say in a note at the end of the paper that they tried filtering by correctly answering a question at the end, and that the effect was still there, but they don't show the data for it or say if it affected the strength. Overall I feel it's a reasonable methodology and certainly suggestive results, if a bit lacking in the exact methodological details and the number of talks they tried it on. And it's not like I have a prior that audio quality doesn't matter...
- gwern 5y ago
  Yeah, that was my overall take: reasonably convincing, neither a dumpster fire nor a slamdunk. These days I don't believe these sorts of psychology effects until I see a Registered Report, but this also isn't the hot steaming garbage you'd expect from the OP summary. It's more or less what you'd expect from a small but not super-small sample investigating a moderate real effect across the board which does seem pretty plausible a priori.