4 ms·

I wonder what they'd be A/B testing for... which 'Decline' button gets the least clicks?

• ratww 5y ago
  My guess is they just A/B test everything by default, even stuff that's court ordered.
  • mattnewton 5y ago
    Basically true but not the terms I'd use; they aren't really A/B tests but staged rollouts, though the process and tooling required is similar. We did staged rollouts of _everything_ back when I worked on google search that wasn't a trivial bug fix. We'd move it to 1% for a day, check metrics, increase to 10%, hold a couple more days and check metrics, then to 100%. Very sensitive or risky launches might hold at a full 50% for some time. UI changes were "dark launched" behind a flag that we incrementally flipped on. The reason is that no test suite captures reality and this discipline forces you to account for easy rollbacks (just turn the flag off) and handle "skew" (the case where I user starts a session where the flag is off but then starts talking to a machine where it is on, or vice/versa). This was in addition to the binary that released multiple times a week and rolled out slowly over the course of the day, and often this happened after multiple versions were tested in experiments with statistically significant samples.
• robbedpeter 5y ago
  Or which version results in the least engagement / fastest click. You don't necessarily want to prompt introspection about privacy.
  • sodality2 5y ago
    This is absolutely true. I took a survey through MTurk from Google, from back when they were reworking the "ad" icon on search results. They wanted me to quickly select the first non-ad, with different "ad" icon styles (same color as rest of text, different shapes, etc).
• fistynuts 5y ago
  To check it's not broken before they roll it out Europe-wide.
  • mbesto 5y ago
    That's not an A/B test per se, thats a blue/green release.