3 ms·

I would model things differently. In particular, I disagree with point #3: "3. Engagement is negatively related to quality." On Netflix, the two most watched

I would model things differently. In particular, I disagree with point #3: "3. Engagement is negatively related to quality." On Netflix, the two most watched TV seasons ever were Stranger Things Season 4 and Wednesday Season 1 [1]. Both were higher quality compared to "sensitive" shows like 365 Days. Generally, I don't think engagement is negatively correlated to quality - rather, I think it's only negatively correlated (sometimes) when we restrict the domain to the most engaging content. If you took random samples of YouTube uploads, I'm confident you'd see a positive correlation between quality and engagement, while simultaneously finding that the highest engagement videos are not the highest quality. I believe I've heard this called splitting tails of the distribution, where if X and Y are correlated, the highest Ys are still unlikely to be the highest Xs. I'd amend "engagement is negatively related to quality" as "the highest engagement items are not the highest quality items", which is a much narrower claim. [1] https://www.whats-on-netflix.com/what-to-watch/most-watched-series-movies-of-all-time-hours-watched/ https://www.whats-on-netflix.com/what-to-watch/most-watched-...

n2d4 3y ago
> I believe I've heard this called splitting tails of the distribution Diverging tails [1], or Berkson's paradox [2]. One of the (imo) most underrated fallacies in statistics. [1] https://slatestarcodex.com/2018/09/25/the-tails-coming-apart-as-metaphor-for-life/ https://slatestarcodex.com/2018/09/25/the-tails-coming-apart... [2] https://en.wikipedia.org/wiki/Berkson%27s_paradox https://en.wikipedia.org/wiki/Berkson%27s_paradox
- agnosticmantis 3y ago
  Very interesting reads, thanks for sharing!
Eridrus 3y ago
This seems heavily impacted by the specific proxy being used for "engagement". In fact the article notes "In 2012 YouTube switched from maximizing clicks to maximizing watch-time. They found it led to a short-term decrease in clicks but a long-term increase in retention. I believe Netflix similarly has invested a lot of time in developing “deep” measures of engagement." If you were to rewrite this article to not use the word engagement and say "user clicks" it would ring a lot more true, but the attempt to generalize from clicks to things like likes, etc, actually muddies the issue and impliesthis issue is more fundamental than it is. In many ways, RL promises to solve this problem by letting us optimize metrics like user retention directly, even when there is not a way to decompose it into a per-item score, but RL is hard to deploy efficiently and has not seen huge takeup. But to go back to the article, I think it's argument that estimating quality is important is clearly true, though often you can implicitly estimate quality from some other metric (e.g. aggregate watch time), but the article implicitly argues that you have to start from estimating clicks and everything else is downstream of that, which is not fundamentally true.
omoikane 3y ago
I think Netflix might be categorized as "other media" further down, where it says "social media platforms rank by predicted engagement ... ranking by popularity is common for other media", so the negative quality correlation might not apply. Quality being negatively correlated with engagement sounds like social media version of people preferring junk food over gourmet.
opmelogy 3y ago
You bring up a good point. When I read the post I applied a filter of "social media" so vetted everything from the perspective of Twitter, Facebook, Insta, etc. And a lot of the ideas hold up when compared to that. It breaks down as you say with Netflix which probably means there's a line between user-generated content and curated content that requires a higher bar.

n2d4 3y ago

> I believe I've heard this called splitting tails of the distribution Diverging tails [1], or Berkson's paradox [2]. One of the (imo) most underrated fallacies in statistics. [1] https://slatestarcodex.com/2018/09/25/the-tails-coming-apart-as-metaphor-for-life/ https://slatestarcodex.com/2018/09/25/the-tails-coming-apart... [2] https://en.wikipedia.org/wiki/Berkson%27s_paradox https://en.wikipedia.org/wiki/Berkson%27s_paradox

agnosticmantis 3y ago
Very interesting reads, thanks for sharing!

Eridrus 3y ago

This seems heavily impacted by the specific proxy being used for "engagement". In fact the article notes "In 2012 YouTube switched from maximizing clicks to maximizing watch-time. They found it led to a short-term decrease in clicks but a long-term increase in retention. I believe Netflix similarly has invested a lot of time in developing “deep” measures of engagement." If you were to rewrite this article to not use the word engagement and say "user clicks" it would ring a lot more true, but the attempt to generalize from clicks to things like likes, etc, actually muddies the issue and impliesthis issue is more fundamental than it is. In many ways, RL promises to solve this problem by letting us optimize metrics like user retention directly, even when there is not a way to decompose it into a per-item score, but RL is hard to deploy efficiently and has not seen huge takeup. But to go back to the article, I think it's argument that estimating quality is important is clearly true, though often you can implicitly estimate quality from some other metric (e.g. aggregate watch time), but the article implicitly argues that you have to start from estimating clicks and everything else is downstream of that, which is not fundamentally true.

omoikane 3y ago

I think Netflix might be categorized as "other media" further down, where it says "social media platforms rank by predicted engagement ... ranking by popularity is common for other media", so the negative quality correlation might not apply. Quality being negatively correlated with engagement sounds like social media version of people preferring junk food over gourmet.

opmelogy 3y ago

You bring up a good point. When I read the post I applied a filter of "social media" so vetted everything from the perspective of Twitter, Facebook, Insta, etc. And a lot of the ideas hold up when compared to that. It breaks down as you say with Netflix which probably means there's a line between user-generated content and curated content that requires a higher bar.