4 ms·
This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomiza
by awhitby 6y ago
This is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time.
But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour."
So given that the author mostly observes long consistent blocks of time with the same headline, that suggests the NYT is allocating them to a subgroup in a persistent way (by IP or whatever). Then perhaps the cases where they didn't observe A/B testing were just cases where they were randomized into the optimal (hence final) headline subgroup by accident at the beginning, and never saw any different.
- asdfaoeu 6y agoIn addition, there's only one case too where it changes back to the original and that's a minor In, in change. Edit: The only example of clickbait here is OP's title.
- rosstex 6y ago+1, they didn't explain that at all.
- tqi 6y agoAgree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales. Assignments are also almost certainly sticky based on a browser cookie.
- megablast 6y agoNo. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour. I’m sure you would agree that would be a good representation.
- hntrader 6y agoThe traffic to a restaurant isn't uniformly distributed over an hour window. There's a peak 30-min in the morning just before work starts for most people, for example.
- tqi 6y agoSequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time. Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.
- deleted 6y ago[deleted]
- baking 6y agoAlso, if the content of the story changes with the headline, then it is probably just reflecting a developing story. That's what the SpaceX and Cuomo stories seemed to be doing. Assuming that all headline changes are just because an algorithm thought they were more dramatic seems to be jumping the gun.
- h00kwurm 6y agoyou’re right! as someone who has built (and builds the future of) the testing/algorithmic machinery at the NYT, this is one of the “misses” in the post.
- dang 6y agoOk, I've consed 'investigating' onto the title above, which hopefully adds an appropriate degree of uncertainty. Thanks!
- lolc 6y agoIt took me a while to understand this, but I think the long consistent blocks are a sorting artifact. As the observed headlines are bucketed by the hour. You don't see the title flip within the hour, you just see the percentage. It would have been more interesting to see the title flipping back and forth, because that would reveal how long those tests last. Less neat though.
- tomjcleveland 6y agoOP here. I don't know if anyone reads old HN threads but thought I'd clarify some things: 1. My scraper runs every 5 minutes, with a randomly-generated user-agent and never sends cookie headers 2. The charts are bucketed by half-hour periods, so even if the headline flips back and forth many times in half an hour, the colors are grouped together 3. Agreed that in the SpaceX situation (and maybe the Cuomo situation) the headlines change because the stories change. But, e.g., in the Meghan Markle situation, the first headline appears _after_ the interview is over. But that's something to watch out for! And charts like this one[0] look (to me) like a clear example of A/B testing. But would be interested to hear other explanations! [0] https://nyt.tjcx.me/articles/1163e0c4-e609-5cfa-aff1-b0945f193679 https://nyt.tjcx.me/articles/1163e0c4-e609-5cfa-aff1-b0945f1...
- rattray 6y agoDid you send from the same IP? (Worth including in the post either way). They may also conduct certain A/B tests only on cookied users, in which case your cookieless bot would not have seen everything.
- remram 6y agoPeople read week-old posts on Fridays, when the hacker newsletter comes out. Thanks for the clarification!