9 ms·
Investigating how the New York Times A/B tests their headlines
- ggm 6y agoGreat article. Stories also change under time and press engagement. Some of these may be feedback loop outcomes (cuomo) where others (spacex land explode story) i don't think are.
- ftio 6y agoIf you're interested in observing this in realtime, there's a great Twitter account called @nyt_diff[0] that automatically posts when headlines on the front page change. Really cool to watch articles get updated as stories evolve. It's particularly interesting to witness headline writers grapple with how and how much to editorialize. [0] https://twitter.com/nyt_diff https://twitter.com/nyt_diff
- TeeMassive 6y agoCool bot. I'm a keyboard warrior and I've been burned by the NYT and other "mainstream" news site stealth changes multiple times. Now when a big story comes out I save the page using SingleFile to Dropbox and when the edited part comes up in the conversation I compare both versions copied paste with Copy PlainText and Meld which allows me to post nice screenshots of the diff.
- exikyut 6y agoAgh, this is using a horribly noisy diffing algorithm. With quite a few of the posts I'd just present them side by side, with the differences highlighted.
- drak0n1c 6y agoThere's a project inspired by nyt diff and NewsDiffs and working on improving the algorithm and supporting many more news outlets: https://github.com/DocNow/diffengine https://github.com/DocNow/diffengine
- Waterluvian 6y agoCBC did this years ago. Whenever I'd load the website, sometimes all the headlines would suddenly switch to different wordings, once sufficiently loaded.
- motohagiography 6y agoValuable anecdata. News is defined by conflict, (literally, no conflict = not news) and this shows how headlines that express the most conflict get the most clicks. The integrity issues come up when even though the facts reported are real, the conflict that frames them is manufactured - and this is why people reject news. They don't reject it because of fake facts, they reject it because of fake conflict. When I want the real news, I go to fringe websites, because they get the real conflict right, and if I need details and facts, I can look those up. The reason they get the real conflict right is because by definition the fringe lives in that conflict, and they are the real anti-establishment that creates a counter balance narrative, whereas a conflict produced by setting mundane events and facts against the backdrop of an ideology designed to manufacture conflict is unreadable tripe. If I needed to sustain the dissonance of establishment narratives, I would read the NYTimes to keep up the appearances, but since my livelihood and aspirations do not depend on that, I have the freedom not to engage it. If you think you are being played and manipulated, watching these A/B tests should be enlightening.
- ARandomerDude 6y agoMeh. This is the narrative of Marx and Foucault, and one I think is demonstrably false. For multiple counter-examples, see pretty much everything on https://www.msn.com/en-us/news/good-news https://www.msn.com/en-us/news/good-news
- motohagiography 6y agoEvery single one of those headlines asserts a conflict, and the only difference is these MSN ones appeal to sentiment instead of outrage. Arguably, newspapers as a concern are a post-Marx phenomenon, and it's what people get taught in j-school, and what editors accept. Students of both are peddlers of pernicious nonsense, but their students also produced some entertaining and powerful things, notably recycled recipes for accelerating and reverting democracies into their inevitable tyrannies. Anyway, the NYTimes doesn't register as meaningful to me, and if they were looking for a reason why people are leaving them and other mainstream outlets behind, it's becuase their conflicts are contrived.
- 6y ago
- trident5000 6y ago"And yet it rarely attracts the kind of close scrutiny of, say, a Fox News. And that’s totally reasonable! Fox News is an absurd clown show and deserves every criticism it gets" I find it comical that any write-up critical of a left wing organization needs to include something like this to avoid being flagged, down-voted, whatever, on the platform its being posted on.
- iujjkfjdkkdkf 6y agoI don't usually read any US news as a rule, because of how sensational it all is. But I think I see most NYT headlines because someone inevitably posts them to NH. And honestly, when I compare them to what I saw on Fox News (which admittedly I only looked at for about a week around the election), I see no difference in the level of partisanship or spin. If anything there is less on Fox News just because the headline writers are not trying to show their undiscovered writing talent in the same way as NYT.
- sodality2 6y agoIf excluded, I doubt anyone would have cared or complained.
- deleted 6y ago[deleted]
- JasonFruit 6y agoShould a news source be optimizing for engagement, or for accuracy in communicating facts? I understand it's a business, but it's analogous to a bakery that labels all its goods as its best-selling items, only to confuse and disappoint the buyer when they open the box and find something different inside. It may sell that product that time, but it goes against the purpose of the organization as a whole, and doesn't seem like a sustainable practice.
- refurb 6y agoThis is my takeaway. There is a huge incentive for the media to make things more sensational or dramatic because it drives clicks and views. I might also argue that online news drives this more so than print - with print you tend to buy the paper based on the totality of their reporting and reputation versus a zero-cost click of a headline on your screen.
- justapassenger 6y agoIt’s fundamentally possible to have both. Although, yes, it’s very tempting to sacrifice one for the other, once you start to optimize.
- moscovium 6y agoIs it not the goal of a headline to engage?
- 6y ago
- freebuju 6y agoCorrect me if am wrong but doesn't this mean they are eventually going to move entirely to a click-bait model of acquiring readers. If this level of desperation is actually seen as acceptable for a media house, then journalism or whatever is left of it, is in dire need of help.
- rossdavidh 6y agoWhat we measure, determines what gets maximized (or minimized, if it's something we don't like). The factfulness of an article, or how well informed the reader is after they read it, are not easily measured. The clicks, are now easy to measure. What gets maximized, is the clicks. It would be nice if we had a news source with a feature (opt in) where the reader got an email quiz the next day, asking just a few questions about the facts in the article. The feedback loop would tell us which articles actually make the reader better informed.
- Rastonbury 6y agoEmotionally charged, slanted or dramatic headlines is one of the reasons I cancelled my NYT subscription a couple of years ago. I'd picked up a student NYT subscription back then and was had FT subscription from school, reading them back to back makes their slant and editorialization pretty obvious even if it isn't the most in your face.
- minimaxir 6y agoThe article is trying to paint NYT's A/B testing as nefarious, but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity, and A/B testing headlines is something every media publisher with a sufficient tech infra does. As frequent Hacker News submitters know, headlines alone can determine whether a post gets upvoted.
- bhupy 6y agoI think the article attempts to demonstrate with its examples that some of the titles don't actually accurately represent what ought to be the core takeaways of a news story. Selected excerpts: "I actually watched this interview—all two hours of it—and I can tell you that the first two headlines are a much better summary of what went down. Yes, Meghan does reveal that she contemplated suicide, but it’s a five-minute interlude in an interview that has a lot more going on." "Trump starts off addressing conservatives and claiming leadership of G.O.P. but in the final headline Trump has a hit list and is firing a warning shot. And sure enough, the bombastic rhetoric propels this article onto the “most viewed” list." In other words, while you're correct that titles are important for clarity, the claim in the article is that "clarity" can be at odds with what gets clicks (or gets upvoted). I'm able to viscerally internalize this because, as a tech worker, I've typically used A/B testing as a tool to maximize engagement, but I've never in my career used it as a tool to maximize correctness or "clarity".
- malandrew 6y ago> headlines are very, very important for clarity I'm pretty sure engagement is more important than clarity. I don't see how A/B testing helps you optimize for clarity. What are they measuring to determine a clarity score? Sadly, most of the time engagement is highest when something is more outrage inducing.
- chiefalchemist 6y ago> but unfortunately in the current social-media-oriented landscape, headlines are very, very important for clarity, Clarity? Or click-thrus? It's definitely not the former. Which in turn discounts the NYT as a beacon for journalisms and associated standards. They're desperate for traffic. Clarity isn't a metric in the playbook. Witin the world of high volume publishing, this type of A/B testing was popularized BuzzFeed.
- awhitby 6y agoThis is interesting but unless I missed it the author doesn't really explain why they believe they observe all A/B tests. They kind of assume that the randomization is over time (so every reader within a window sees the same headline, and then it changes) rather than within cohorts at a fixed time. But the quote included suggests the NYT does do the latter: "Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour." So given that the author mostly observes long consistent blocks of time with the same headline, that suggests the NYT is allocating them to a subgroup in a persistent way (by IP or whatever). Then perhaps the cases where they didn't observe A/B testing were just cases where they were randomized into the optimal (hence final) headline subgroup by accident at the beginning, and never saw any different.
- asdfaoeu 6y agoIn addition, there's only one case too where it changes back to the original and that's a minor In, in change. Edit: The only example of clickbait here is OP's title.
- rosstex 6y ago+1, they didn't explain that at all.
- tqi 6y agoAgree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales. Assignments are also almost certainly sticky based on a browser cookie.
- megablast 6y agoNo. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour. I’m sure you would agree that would be a good representation.
- 6y ago
- jeffgreco 6y agoIn several of these the shift is due to a headline written about an upcoming event being changed during and after the event, especially the Trump and Cuomo examples. Interesting tracking but not really A/B, which would be testing two different headlines roughly simultaneously.
- Spooky23 6y agoI would say that learning how the NY Post writes their headlines would be a more fun story.
- arielm 6y agoIntegrity and such aside, as someone who runs A/B tests I love seeing this data. Thanks for sharing it!
- reilly3000 6y agoGood piece, but damn I was hoping for some details on their implementation. Are the variations coming from the server, CDN/edge, or client? How does this impact SEO? Do they change the article slug if the title is a winner? How much is automated and how much is human curated? When beset with new assignments who has the time to babysit these things?
- novemberyankee 6y agocdn; seo impact is also tested for; canonical URL doesn't change; titles are written by humans, testing is entirely automated; small parts of larger UX team just do testing so babysitting these is their job
- reilly3000 6y ago+++ thank you! A stack to aspire to!
- patleeman 6y agoIs the headline testing some sort of multi-arm bandit system or a/b tests that are updated manually? It seems like MAB would be the ideal approach for something like headlines.
- reilly3000 6y agoHow do you measure SEO impact in a title split test? Scraping SERPs or monitoring query volume real-time?
- Hippocrates 6y agoThe value in A/B testing headlines is 30% boosting engagement directly and 70% creating a feedback loop where the newsroom can learn how to write a more engaging headline. Avoiding clickbait headlines requires some restraint and they are well aware of that slippery slope.
- exikyut 6y agoFeature request: Service that scrapes websites known to A/B test things, caches the A/B test content, and presents all of the variants at once. In this context, yes, I want all the headlines. Maybe they could be presented as a diff. With timestamps!
- wp381640 6y agoThis is very reminiscent of how the YouTube algorithm when optimized for views and clicks drove a lot of people into deep rabbit holes.
- bArray 6y agoI would suggest that this can be more 'evil' that just trying to get more clicks. A number of times I have seen on some place like Facebook where the initial article has some extreme headline, and then hours after when engagement is up, the headline is swapped to one that is less inflammatory. A few more extreme-perspective friends will send me an article saying "see?!?!" - and by the time I see it it's already been rewritten. A few times I would click on an article going viral and find the headline in the article itself doesn't match the one cached by Facebook. Remember that most people aren't even reading the headlines and just assume that NYT are trustworthy. The first headline is the one they end up internalizing. Bare in mind, there is zero consequence for doing this either. The newspaper and claim they were "correcting an editorial error", whilst openly spreading misinformation about some hot topic. I think at the very least it should be mandatory to maintain a list of edits to an article once published - and to indicate to the reader clearly that the article has received a number of edits.
- systemvoltage 6y agoI’d agree with you but it seems like these alterations are pretty benign. They’re still professional and not at all fake/misinformation.
- unethical_ban 6y agoThe entire thesis of the OP is that it is not benign and the intent is malice/inflammation rather than accurate summary of content. It may not be "fake news" in the Fox srtain, but the OP's point is that it is not for the good of the reader or society.
- refenestrator 6y agoIf they consistently go from "max outrage to grab attention" to "more clinical, for posterity", that's not benign. It's dishonest. Either be a tabloid or don't be.
- xwolfi 6y agoHaving worked in A/B testing in a loan marketing company, where we were pushing 300% payday loans in Mexico (you read that right), I can talk of evil. I think, when you're in the day to day of a web platform, you really just simplified everything from your hiring to your board meeting to one metric: click. Everything is number of click, it becomes no more evil than a Lion in the Savanna, you just must increase your number of clicks somehow and you forgot why. It can lead to interesting "let's remove the legal text away and see what gives", "let's color ever buttons blue vs pink and see who wins", let's reorder the search result independently of what a normal human would expect (as in... cheapest first?)". I see what you mean, but it's not evil anymore. It's now part of nature, they must hunt your click. They can't stop and they can't explain why. They could put one headline "Myanmar military coup" then replace it with "Myanmar cabinet reshuffle" to see which one you click on with little input from any writer and no philosophical debate whatsoever.
- unethical_ban 6y agoI think there is some bad intent at times - I think a reputable institution should be optimizing for accuracy and engagement, not sensation for its own sake. Different headline styles are suitable for different sections of the paper, as well. I saw one the other day from Nick Kristof: "America is not designed for the anatomically correct". It was about the need for public restrooms, but the headline didn't make me read it. It sounded goofy and, other than knowing Kristof writes decent articles, I passed it over. Later, the headline was "America isn't designed for those who pee". A bit coarse, but very much more clear as to what the topic of the editorial was.
- h00kwurm 6y agoI’m really happy that folks are digging into the perceptual outcomes of the AB testing from headlines. Frankly, it provides context to the NYT’s own research and development of the algorithms and human processes that go into such efforts. If the narrative is entirely that if we dont actively consider capturing interest, we’d be doddering and hard to track, if we do we’re abusive, then media is forever doomed to be unsatisfactory. We all hope to improve. In the world of headlines, the “spiciness” that’s been advanced as a function of engagement hunting is something that’s currently contended with through human intervention. All headlines are human created and the outcomes of AB tests are more about improving the understanding between author/editor and captured audience than manipulation or future interest conditioning. [disclosure: i lead ML platforms and the algo related eng products at NYT. all thoughts are my own presentation of what i have experienced, and not company opinions]
- 02020202 6y agonews today is merely a psychology-based domestic terrorism on massive scale without any oversight. hence why people today are constantly stressed out. news became "tabloid" because in the end, it is not about OBJECTIVE reporting of FACTS but rather SUBJECTIVE manipulation and selective truths that will yield the most clicks for the ad-based revenue. it is sickening and one of the most horrid things we have in the 21st century.
- deleted 6y ago[deleted]
- benlumen 6y agoDoesn't surprise me in the least. I've long felt that "journalism" has more in common with marketing and SEO than much else these days.