That’s not at all an argument for dropping AB testing, it’s an argument for being more rigorous, especially since in principle the circumstances under which ser

That’s not at all an argument for dropping AB testing, it’s an argument for being more rigorous, especially since in principle the circumstances under which services with a large active user base can test are downright luxurious. Sociologists will frequently not have as good access to such a large participant pool under near ideal experimental conditions with such good ways to observe behavior. And the stuff you have to keep in mind when running experiments is not terribly complex. A bit of statistics, a few things you absolutely have to get right, that’s it. Obviously there are reasons why AB tests are often not run rigorously (statistical illiteracy and pressure to get things done quick as well as to produce tangible results as often as possible – all three of which might lead you to run underpowered experiments with too few participants and to stop testing early which will lead to too many false positives). However, stopping to do experiments (and instead just releasing new stuff and observing the reaction) isn’t really an improvement that leads to better outcomes compared to that.