6 ms·
How to create a blog post title optimizer with GPT-3 and Hacker News data
- sharemywin 4y agoIn the article the author used GPT-3 babbage instead of Curie or Davinci. It got a 68% prediction rate on the validation set. I wonder if the other versions scored better? Also wonder how much it cost to train it?
- minimaxir 4y agoI mentioned in the article it cost $2. Finetuning on curie would be 5x that ($10) which is a bit much to experiment, and also increase the overall generation cost notably. It's possible there would be an accuracy improvement but it's arguable if it's worth it.
- goodside 4y agoThis is a cool demo, but note the author doesn’t actually show the rewritten headlines are better in any way but spot-checking. The conclusion here isn’t that GPT-3 can optimize titles on its own, but that it generates ideas that, when reviewed by a human, can be usefully cherry-picked for inspiration. The task of ensuring, e.g., that the headline is factually accurate is left to the human.
- minimaxir 4y agoThere isn't an easy way to prove one headline is better than another outside of A/B testing, unfortunately.
- RosanaAnaDana 4y agoEven then, is there even a way to do so with A/B testing? Even in A/B testing you can't fully control for the environment a headline sits in. Whats happening in the world? What styles of headlines are trending? Proving things is very difficult, and probably not necessary.
- minimaxir 4y ago"Proving", in this case, is whether the performance of one title is better than another at statistically significant level. Performance can be faceted by referrer if necessary if there's a need to control the results further, or exclude sources where title A/B testing can't feasibly be done or has noticeably different behavior. (an example is using a different title for social sharing vs. the article title itself) This is generally how title A/B testing works at every major online publisher.
- motoboi 4y agoThis is so funny because the article is actually on the first page, but GPT-3 told him it's not a good title. Lol So, it's GPT-3 wrong, is his code wrong or it's just that GPT-3 doesn't like hacker news titles?
- minimaxir 4y agoThe model predicted a 1/5 chance that it turned out to be a good title, so those aren't terrible odds for a random chance. Although if this post does hit 100 points, I'll add an update. That's the difficulty of interpreting the results of AI models and one of the dark secrets of DS/ML practicioners is that they don't always have the right answers, just the best-case scenarios.
- ruuda 4y agoWell, can you imagine how much more upvoted it would have been with a good title?
- andreyk 4y agoAlso, those were variations on a title about this particular topic. I bet if the model were used to score the likelihood that this blog post (with any reasonable title) were to hit front page, it would predict that to be likely.
- behnamoh 4y agoGoodhart's Law: “When a measure becomes a target, it ceases to be a good measure.” Optimizing for HN frontpage is a terrible idea. (1) HN users are usually hostile against titles that even remotely look clickbait and flag them, (2) It attracts the wrong audience to the blog post.
- WithinReason 4y agoBut what if you optimize on data from HN?
- behnamoh 4y agoMy point is: "why optimize in the first place"??
- adriancooney 4y agoAnecdotally I agree with you but doesn't this blog post suggest the reverse - click bait does well? The model was trained on a fairly comprehensive set of HN titles and it scores click-bait-y titles with a high "Good" probability. e.g. `"Beware! Uninstalling this PC game deletes your hard drive"` with a `62.0% Good prob`. There's a ton of hidden complexity involved here but if click-bait was generally downvoted by the HN community, we should expect a low "Good" score, right?
- a4isms 4y agoSpeaking as someone who has been writing forever and blogging-as-we-currently-know-it since 2004... It always depends on what you want out of it. Some people are outcome-oriented. They want reputation and/or the affirmation of crowds, and for them, writing is a process of reverse-engineering what people want to read. At a deep level, they don't want to write, they want to have written something popular. Others are journey-oriented. They write to work ideas out. paulg championed this o̶v̶e̶r̶ ̶a̶ ̶d̶e̶c̶a̶d̶e̶ ̶a̶g̶o̶ almost twenty years ago when he wrote an essay explaining that the word "essay" is not just a noun, but also a verb suggesting an attempt or experiment: http://www.paulgraham.com/essay.html http://www.paulgraham.com/essay.html. For some ideas, trying to explain them forces us to confront the idea cheaply and identify areas where we haven't thought things through all the way. And of course, sometimes it's a little of column A, a little of column B. Speaking for myself, my taste is almost entirely for column B: http://braythwayt.com/posterous/2014/10/30/write.html http://braythwayt.com/posterous/2014/10/30/write.html. But this is a big world, and if someone wants what's in column A or to sprinkle some column A on their column B essay, well, this post provides some hints.
- Labo333 4y agoShould we be concerned that the author is "Data Scientist at BuzzFeed"?
- minimaxir 4y agoIn what way?
- manquer 4y agoNot OP, I don’t think it is concerning if anything makes you more qualified, Buzz Feed has enormous subject matter expertise on titles that work. However I did it find it amusing after reading your rejection of click baity versions of the title because HN crowd wouldn’t go for it
- gwph 4y agoReposts are very common on Hacker News. I wonder if incorporating the performance of the same links with differing titles might've yielded a better training set.
- LordDragonfang 4y agoPosting guidelines on HN generally require you to not editorialize titles, so most reposts have the exact same title (since it's the title of the original article).
- ruuda 4y agoI wonder, when sampling to shrink the set of bad titles, it samples uniformly. But wouldn't it be better to take the n worst titles (with the least upvotes)? After all, titles near the good/bad threshold are a bit arbitrary and should be hard for the model to get right, would increasing the contrast make it perform better? Or does that limit the model's exposure too much that it ends up harming more than it helps?
- minimaxir 4y agoIt does sample uniformly to shrink the bad titles. Sampling nonuniformly may introduce an unexpected bias, so there's a tradeoff there.
- earthboundkid 4y agoOkay, now optimize for controversial.
- cwillu 4y ago“Apple releases M3 processor with embedded NFT of rust compiler written in go”
- cweill 4y agoHey Minimaxir again! This is really funny because I built an entire company around this idea: optimizing titles for YouTube videos: https://CreatorML.com https://CreatorML.com. Similarly, the user can use GPT-3 to generate titles, and a secondary ranker model scores them for estimated viewership. I use a regression model however. Anyway, just funny that "great minds think alike" :)
- behnamoh 4y agoSo it's people like you that turn the internet into a clickbait jungle... No thanks.
- coffee_beqn 4y agoThat is pretty powerful. I didn’t realize we had so many tools already that something like this looks.. easy?
- potatoman22 4y agoIs this typically how large language models are used for classification? With prompts?
- minimaxir 4y agoIt's one approach. In general you get better performance when the input data is aligned with what the model is trained on.
- iamcrazyyounus 4y agoIn place of Curie or Davinci, the author of the piece utilised GPT-3 Babbage. On the validation set, it had a prediction rate of 68%. Do the other versions have higher ratings? I also wonder how much training that cost.
- techniko 4y agoThe first thing I thought of when reading this article was https://news.ycombinator.com/item?id=15567707 https://news.ycombinator.com/item?id=15567707 where someone created a model that filters Hacker News for articles potentially interesting for them