4 ms·

Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistical

Most decent companies and especially tech do AB testing for everything including having people whose only job is to make sure those test results are statistically valid.

staunton 3y ago
The magic words here are make sure.
- marcinzm 3y ago
  In my experience they make sure a ton more in industry than in academia.
  - staunton 3y ago
    Any anecdotes you can share? Also, I meant "make sure" in a negative way. As in you "make sure things are statistically significant" by e.g. p-hacking. Not that this isn't done in science but I think you're more in danger of being embarrassed during peer review than by the C-suite reading your executive summary...
    - marcinzm 3y ago
      Most companies that care will run everything though an AB test, the number of AB tests is physically limited by traffic volume and the team in charge of measuring the results of the AB tests is not the team that created the experiment. That makes it much harder to p-hack since you cannot re-run experiments infinitely on a laptop, and the measurement team is judged on the accuracy of their forecasts versus the revenue impact.
      - figassis 3y ago
        The worse things can and often do generate more revenue. So what are the goals of the AB test?
        marcinzm 3y ago
        >The worse things can and often do generate more revenue. The goal of a business is usually to generate revenue so I'm confused about your question.