4 ms·
I wrote a comment debunking OP's arguments, but it was too long for a HN comment and my noprocast settings kicked in so I can't post it as two comments; so I co
by gwern 12y ago
I wrote a comment debunking OP's arguments, but it was too long for a HN comment and my noprocast settings kicked in so I can't post it as two comments; so I copied it over to G+: https://plus.google.com/103530621949492999968/posts/1PqPdLyzXhn https://plus.google.com/103530621949492999968/posts/1PqPdLyz...
tl;dr: Let's summarize his complaints and my counter-objections:
1. no consent: irrelevant to whether this was good science or 'lousy social science'
2. crossed boundaries between corporations and academia: likewise irrelevant; also, welcome to the modern Internet
3. small effect size: misunderstood the statistical design of study and why it was designed & expected to have small effects
4. used LIWC with high error rate for measuring emotionality of posts: if random error, biases effect to zero and so is not an argument against statistically-significant findings
5. and LIWC may have systematic error towards positivity: apparently not an issue as negative & positive conditions agreed, and the studies he cites in support of this claim are mixed or unavailable
6. also, other methods are better than LIWC: sure. But that doesn't mean the results are wrong
7. maybe LIWC has large unknown biases applied to short social media texts: possible, but it's not like you have any real evidence for that claim
8. Facebook news posts are a biased source of mood anyway: maybe, but they still changed after random manipulation
9. experience sampling is sooooooo awesome: and also brings up its own issues of biases and I don't see how this would render the Facebook study useless anyway even if we granted it (like complaints #1, 2, 6, 7)
Now, I don't want to overstate my criticisms here. The author has failed to show the Facebook study is worthless (I'd wager much more money on the Facebook results replicating than 95% of the social science research I've read) and it would be outright harmful for Facebook to aim for large effect sizes in future studies, but he does at least raise some good points about improving the followup work: Facebook certainly should be providing some of its cutting-edge deep networks for sentiment analysis for research like this after validating them if it wants to get more reliable results, and it would be worthwhile to run experience sampling approaches to see what happens there, in addition to easier website tests (in addition, not instead of).
- kevingadd 12y agoI don't feel like you make an adequate case for some of your counter-objections. #1 is particularly troubling since you seem to argue it by misrepresenting his argument instead of addressing it directly. The alternative is to perform the experiment w/informed consent, not to keep the results of your unethical experiment secret. I think this is actually an issue that is important when considering whether an experiment is 'good' or 'acceptable'. Tuskegee is what I think of when people try to argue that informed consent is unimportant. Even though anyone can tell you that the stakes in this kind of experiment are incredibly small in comparison, there IS a measurable risk of harm when manipulating emotions, so ethics matter here. #2 - you seem to make no effort to address this either. To me, it is particularly troubling if the business and academic sides of FB freely interacted here because each side has an incomplete understanding of the other side's motives and priorities and constraints. The business side is focused on revenue and actionable metrics and legal liability (that's their job), while the research side is naturally going to have different objectives and have a perspective that is more rooted in the practice of good, ethical science. There's real trouble if research ethics & principles end up subjugated to the interests of the business people (which they often do). Seeing you utterly dismiss the first two points with 'Oh good, so the author isn't a complete idiot.' is not exactly a revelatory argumentative triumph. You can do better.
- gwern 12y ago> #1 is particularly troubling since you seem to argue it by misrepresenting his argument instead of addressing it directly. The alternative is to perform the experiment w/informed consent, not to keep the results of your unethical experiment secret. No, the alternative is to go through a long rigmarole which may or may not approve the experiment and enforce conditions which may or may not themselves constitute severe biases of their own (knowing one is in an experiment, even if the subjects have been deceived as to the content, is itself a problem). > #2 - you seem to make no effort to address this either. To me, it is particularly troubling if the business and academic sides of FB freely interacted here because each side has an incomplete understanding of the other side's motives and priorities and constraints. There's nothing to be addressed here. Businesses have always allied with scientists when they find common interests, from astronomers preparing tables for shipping to Gosset inventing a good chunk of modern statistics for optimizing a beer brewery. All he's done is talk vaguely about 'boundaries' and insinuate dark things. > Seeing you utterly dismiss the first two points with 'Oh good, so the author isn't a complete idiot.' is not exactly a revelatory argumentative triumph. You can do better. Those points are fundamentally irrelevant to the overarching claim he makes that it's bad science. I think he's wrong about the more... philosophical aspects of things, but I don't have to show he's wrong - he has to show they materially affect the truth of the results. He has to show that #1 and #2 actually matter, and he hasn't. Instead, he spent an incredible amount of verbiage on vaguely related topics and he flubbed the technical criticisms. I don't give him a pass on that and neither should you.
- yid 12y ago> No, the alternative is to go through a long rigmarole which may or may not approve the experiment and enforce conditions which may or may not themselves constitute severe biases of their own (knowing one is in an experiment, even if the subjects have been deceived as to the content, is itself a problem). Having gone through some IRBs, there is a good reason and enough historic precedent to warrant this "long rigamarole". The point is that you can't simply say "but everyone on the Internet does it" and expect people who support traditional research ethics not to say "wait a minute, that shouldn't make it OK".
- qu4z-2 12y agoRegarding your point about the percentages, the OP states that between 10-90% of positive or negative posts are removed, depending on the group. Looking at just the positive posts for a moment, if around 50% of the posts in the news feed are classified as positive (46.8%), surely removing 10% of those is precisely the 5% of posts you claim is different. Note that the percentage removal is based on the uid, so the range of posts removed would be between about 5 and 45%, depending on the poster. The 4.68% quoted is for a user on the extreme low end of the removal range, and is percentage of total posts, not positive posts. I don't see how this is different from the claimed 10-90% of positive posts. Similarly for negative posts. Am I misunderstanding something?
- gwern 12y agoThe OP may or may not have misunderstood it depending on how cleverly you parse what he wrote, but he presents it in a way that no reader would be able to understand, compared to the extract from the study which actually does specify the final numbers so one understands that it was a small intervention.