It's not evidence at the scientific level, but not far from it either. Raw data would be nice, but given the dramatic effect size reported I don't see it as abs

It's not evidence at the scientific level, but not far from it either. Raw data would be nice, but given the dramatic effect size reported I don't see it as absolutely crucial (it wouldn't be in a scientific context either). Also the methodology is described reasonably clearly (change the chat avatar and nothing else). Further the blog post even does discuss that this is not a scientific study, and what some of the possible confounders were. I'll concede that my statement is to general. But the article to me is strong evidence that the women involved experienced a strong sexist reaction from her customers. Are you not convinced that she did? The set up is as close to perfect for confounding variables as possible in an evolving production environment (as someone noted, it's basically A/B testing). I'm not saying it's convincing on its own in a vacuum of course. The prior formed by all the other studies (including the one you mentioned) is important in making it convincing. But it's certainly as clear evidence as 99% of what get's posted here. Put another way: If this was a blog post that shows that one configuration of a database leads to latencies consistently 10 times smaller than another configuration, with experiments run over three months of production use, I would call that convincing evidence. It's certainly a lot more coherent than the post I was replying to which somehow argues that this is not objective because chat boxes are annoying. (????) Luckily it's no longer the top comment either.