14 ms·

I worked at Facebook for some time, and did a bunch of data work. We had this culture of building something, looking at the results between the experiment group

I worked at Facebook for some time, and did a bunch of data work. We had this culture of building something, looking at the results between the experiment groups, and then choosing the statistically more successful one -- i.e. a newsfeed algorithm that had better engagement. This sounds great at first, and certainly is straightforward if you want a promotion. But behind the scenes some of us had this thought that our observations only amounted to short-term gains. Although we had small long-term experiment holdout groups, the truth is they were rarely reviewed because it was unsexy. My current thinking is that features like the echo chamber effects from Facebook's algorithms, Snapchat's snap streaks, and clickbait like this, all serve to optimize short-term engagement. Yeah, I want to watch that sexy new show or keep my streak going or have my opinions validated. But there's a diminishing return on clickbait, hollow articles isn't there? I can only fill up so much time with garbage like that before I'm bored. I can only like so many posts before I feel like they're all the same. And once my snap streak is broken I hate snapping. The data/engineering/product loops at tech companies favor boosting short-term metrics; The employees are incentivized to do so and this is what they measure, so this is what they build. That's why we end up with features like this. That's why Snapchat fell off. That's why Facebook fell off. And that's why Netflix feels increasingly stale (despite there being a lot of quality content if you dig).

vasco 5y ago
They don't seem to have fallen off, are still the most widely used platforms in the world with billions of users in total. It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The only thing I've seen work is something like Pixar's brain trust - you do testing but you also rely on the good judgement of a small group of people with a strong sense of vision. Judging good judgement is pretty hard though, and you won't know if they're just full of it or winging it unless you try and trust. And so everyone does A/B testing which to be honest sounds much better than your average PM making decisions that are pulled out of their ass.
- deleted 5y ago
  [deleted]
- HWR_14 5y ago
  > It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? I mean, if corporate leadership isn't able to better direct the path of a product than the results of A/B tests, what value do they really have?
- piva00 5y ago
  > It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The alternative involves creativity to create a vision to achieve. A/B testing are pretty good tools to evaluate differing opinions on some vision, as the only deciding tool for what is worth to keep or not it's pretty short-sighted. Is it really that much better? I lived through the rise of A/B testing, professionally I've been responsible for implementing platforms for experimentation in at least 5 different companies and over the years I saw it transition from a tool to a crutch. Everything now is "data-based" which means incessant A/B testing, it still depends on your average PM deciding the parameters of the experiment, leading to the current state of affairs: a bunch of uninspired features that are solely judged on the data produced, no soul, no vision behind most of it. And it feels exactly like that, I can feel how tech products just became that, an incessant stream of features being tested, implemented or tossed. I'd much rather have data-supported intuition than this statistical machine in motion.
- fisf 5y ago
  The problem is not only that it could be a local Maxima. It might also measure the wrong metric all together. Is the platform widely used and the business successful because content makes users happy and improves their lives? Or is engagement high, because users have become accustomed to whatever a company is feeding them and look for something shallow to fill up their time. If you only measure engagement as metric, without any deeper reflection and judgment, then heroin is a perfect product in terms of A/B testing.
lukestevens 5y ago
Yes, I'd love to see a more robust approach to novelty decay rates (to make up a metric) in A/B testing, especially around engagement. You'd think that would be of interest to the business to know that X "successful" intervention had a typical average lifetime of Y before reverting to the mean, for example. Then again, there's the idea that competitive advantage for social networks is just finding enough novel interventions before the novelty of the platform itself is exhausted, and FB has no short supply of other platforms to milk in that regard.
dreamer7 5y ago
What was the duration for which you collected results? Was it 2 weeks or a month etc?
- mgraczyk 5y ago
  Depends on the change and effect size, also on product. Typically somewhere between 1 week and 1 quarter. Almost always there is a backtest after shipping in which a small percent of users do not receive the change for 1-6 months (a "holdout")
keiferski 5y ago
This is the consequence of lack of vision. What is Netflix's long term goal? Ideally, it would be to fund and broadcast fantastic cinema. Yet, I really don't get the impression that contributing to the art form of film is very high up on the metrics chart.
- pizza234 5y ago
  I'm of the opposite opinion. They're executing their vision perfectly. Netflix is very aggressively pursuing the bottom line, and they're not doing it differently from a drug dealer. It's not random that the term "Netflix binge" exists. Netflix interface is designed to shove content at the users' face at all costs, and in a very disrespectful way. Here there are some disgusting dark patterns: - it took them a long time to give the user the option to remove movies from the watching now list - there's no way to remove already watched movies from the panel; their trailer is even sometimes displayed in the top frame - they randomize the global order of the horizontal lists, so that one needs every time to hunt the user list (watching now/my list) through all the panel For a short time, I even experienced a bug where movies with (my) negative rating were not greyed out. I can imagine the engineering thinking "LOW PRIORITY!". On my (fictional) "drug-dealing practices" rating, Netflix is high on top with Amazon.
  - john_minsk 5y ago
    100% agree. I was awaiting for Netflix thread to pop-up to ask (hopefully someone who works there): Why there is no way to tell Netflix that I already watched the show and I'm not planning to rewatch it again any time soon or hide the ones I tried and decided not watch (or may be watched before)? No matter how many times you show me them I'm not going to watch... You can still have nice features around it: separate list at the end of scroll with my old movies/shows (may be with number of times I watched it?). Once new seasons released send them all the way to to the top etc. And I will be able to discover something new. At the moment it is cluttered with things I don't want to watch so lately I was not happy with Netflix at all. You should see such patterns in your data.
  - i2shar 5y ago
    Add one more to the list - and this one drives me crazy to no end - on my Chromecast with GTV the wretched thing starts autoplaying even if I just want to read the synopsis. This, despite turning off all autoplay options in my profile. What is the friggin rush?! Why won't you let me browse/read distraction free?!
2Gkashmiri 5y ago
i do not understand snapchat and their business model. they started out as a sexting app that deletes the photos and that is where it remained for a long time. i remember news of how "deleted snapchat photos can be undeleted" tutorials and i tried to investigate for fun but i could not understand the UI back then and the tutorials didnt work so i dropped it. Apparently its pretty big these days in the kids and they have "streaks" as you mention. i hear multiple hundred thousand streaks and other things but to what end? how is snapchat justifying the storage, bandwidth and processing of data? i mean facebook has ads which they earn, same for twitter but what about snapchat? do they have a sustainable revenue stream because no where i heard snapchat had a paid option. reddit was user funded for a long time with daily goals and all that so that was not an issue mostly but what snapchat done to address that. snapchat is all the rage in my city, kids who are now on online classes send snaps because i hear about it. everyone knows how to make that twistface pout or whatever but why? genuinely curious about it
- cosmie 5y ago
  > i do not understand snapchat and their business model. Ads[1]. Initially it was just promotional lenses/filters and required a high touch process and substantial media spend commitment. But looking at that site, they now have a full on self-service ad platform with minimal spend commitment, helper software to make lenses/filters, and have added several other ad formats beyond that one. So they've essentially monetized the same way as every other "free" platform has – capitalize on all the eyeballs they have to siphon off a cut of marketing dollars for themselves. [1] https://forbusiness.snapchat.com/ https://forbusiness.snapchat.com/
  - 2Gkashmiri 5y ago
    cool. i did not know that
- ladon86 5y ago
  $2.5bn revenue last year. You've heard of streaks (which were a very early feature), and you've probably heard of Stories too, right? Video ads appear in between Stories video rolls.
tomnipotent 5y ago
> The data/engineering/product loops at tech companies favor boosting short-term metrics I think this hits the nail on the head. Leadership is hard, and executives can be many layers removed from the frontline. It seems that A/B tests and family have proven a really effective vehicle to communicate across leadership levels what has happened, why it happened, and the impact it had on the business. I'm definitely guilty as charged of having participated in this "pyramid scheme". Teams get a tool to structure work and determine success/fail. Managers get a tool to detail exact impact to leadership on how they're solving business problems. Executives get a tool showing how teams are being deployed on thoughtful projects with an iterative and rigorous scientific approach that helps increase confidence around deploying capital. The board gets some slides during the quarterly meeting and maybe might even see some charts showing growth and less cash burn. Eventually over longer periods like a year, executives will notice different trends than what the short-term numbers are showing and will have to make real hard decisions about how to steer the ship. It's a hell of a hamster wheel.
- dr_dshiv 5y ago
  How might a streaming service be designed to increase user happiness and wellbeing? From a positive design perspective, could certain streaming features give users a greater sense of purpose or goal accomplishment (e.g., mastering a genre) or help users meaningfully connect to others (e.g., seeing what friends have recommended lately)? While measuring the effects of a feature on wellbeing outcomes is substantially harder than measuring the effects of a feature on engagement (time spent), it is worth measuring/optimizing in the long run. The hypothesis is that capitalist enterprises that make their users feel happier and more fulfilled will, in the end, be more successful than enterprises that merely provide empty satisfaction. If not, the future could be very bleak.
  - verve_rat 5y ago
    Maybe... make good shows?
    - dr_dshiv 5y ago
anon9001 5y ago
> I can only fill up so much time with garbage like that before I'm bored. Don't sell yourself short! You're surrounded by tons of people that never get their fill of content. Just relax and let it happen ;)
mgraczyk 5y ago
As another former Facebook employee, I mostly agree with this. One nitpick is that although you may have not looked at the holdouts often, somebody definitely did. Different teams use their holdouts differently, but leadership probably looked at your holdouts at least a few times per quarter. And now with recent infra and changes, it's very likely that data scientists or product managers somewhere in your org are responsible for explaining results to higher ups at least quarterly. All that is to say, the cause isn't that Facebook (and Netflix) aren't thinking about or monitoring things in the long term. It's that they are measuring the wrong things, because it's very hard to measure things like "this is clickbait".
beerandt 5y ago
Netflix has somehow conditioned me to expect disappointment in movies with interesting looking cover art, which I now actively ignore, and don't even click through for descriptions anymore. "Huh, that looks interesting. Not falling for that old trick again! What's that half-off-screen ambiguous cover on the next row down?"
roystonvassey 5y ago
Interesting insights. Online Analytics is now really nuanced - you need to know what metrics are important for * your * business, not just use boilerplate kpis. For instance, an e-commerce website is clearly looking to lead the user to a lot purchase and the more they purchase, it’s good for the business so kpis around sales conversion help and recommenders help your business increase sales. For Netflix though, the users have already paid for the service after which they land on the website. Most users I imagine then expect to be provided all that Netflix has to offer in an easy way. So if I was a Netflix product owner, I’d be more interested in Kpis around search-ability, having an anti algorithm that “suggests” completely random obscure shows, “switchability” of users - how less of a time do users spend on a movie or show. I imagine they’re doing this but as a user I don’t see this at least - they show the same old stale recommendations for me, I’m always trying to hack their search to find what I want and they continue to invest in content that’s mostly miss than hit. I wish they at least had a directory for me to browse through (at least I’ll be driving their engagement metrics to help them drive their valuations)
- kofejnik 5y ago
  They don’t want you to watch obscure shows, they want you to watch the top ones (in your region!) as they are preloaded into your cdn already.
  - roystonvassey 5y ago
    Right - but why would they assume I want to watch the "top" show? I've already paid for it and they benefit by having me browse and allowing myself to find that I like right? And what I like is not necessarily what they think others might me like - what one watches is highly variable (can change day to day, even time of the day) and influenced by so many factors that are personal.
    - pnut 5y ago
      Netflix is now a major player in the entertainment business, and I would guess there is a pay-to-play element behind the scenes as well as ROI on new productions, forcing them to jam these options down our throats. User viewing preferences seem to be a secondary consideration.
qwertox 5y ago
> i.e. a newsfeed algorithm that had better engagement. Sometimes (but rarely) I click on something because I ask myself "Why would these a---- show me this? I want to know more about what's behind it in order to understand what could have made their algorithm chose me as a candidate for this content". Or to check where the scam-URL would lead me to, because this also happens, that ads sold on Google and Facebook are scams, links leading to malware. Congratulations to them, if they think that this is the kind of engagement which should be valued, that any kind of engagement is a good and healthy engagement. These companies are so rotten and their engineers just don't care.
- bertil 5y ago
  There were tools to help answer your question (predictably called “Why am I seeing this?” after the button copy). Those had so little engagement they were not worth maintaining.
uxcolumbo 5y ago
Off topic: To me they seem to focus on quantity over quality. Can you recommend some of that quality content? I always quit after browsing their catalogue for 20 minutes or so.
Fiahil 5y ago
I think you can maintain the feedback loop without degrading the user experience if you actually ask them what they want to be shown. The problem lies in that "engagement" definition : if the user clicked on a thumb, then they must have been attracted to it. In reality, there is considerably more explanation for that click : They could be looking for a specific title, mistakenly thought _this_ actor was playing in the movie, clicked on it to reset the search, wondered if this was a show or a film, missclicked, just picked the least annoying in an ocean of shitty propositions... An algorithm alone can never find its way for better suggestions if it lacks _intent_. The outcome of this lack of data and the lack of data scientists' imagination, will always be stuck in a local minima. So, what solution do you have to gather this precious intent and act on it ? Well, you start small on predictable things : "what would you recommend to Agatha if they liked _Expendables_ ?" "Is _RED_ a good addition to the list if I'm looking for Action/Comedy films ?" And later "Are you looking for Action/Comedy films ? here is some proposals" It works the same for different approaches, for picking thumbnails, or promoting movies. Just ask user for what's motivate them. If Netflix had asked me last week, I would have told them their catalog is getting worse and worse so we prefer using disney+ instead.
- notafraudster 5y ago
  The idea is that these kinds of sources of error will be noise, since they should be random with respect to experimental assignment in any kind of A/B test. In a small sample you could imagine chance imbalances, but given that I'm experimentally assigning someone to see either a thumbnail of John Krasinski or Steve Carell, it's unclear why the "I misclicked" scenario is going to have a higher rate in one rather than the other. And experiments at large companies like Netflix are almost infinitely powered. I would think a bigger threat to this approach would be being stuck in a local maxima.
antiterra 5y ago
It seems even worse than that. During my stints in social media, there would be ‘research projects’ that were simply fishing expeditions for meaningful data. Something like: which user demographics have a correlation to behavior x. And then you’d just search and search and search and hope you found something. If you didn’t find something, then you’d be encouraged by a team level manager to adjust or tweak some approach to take advantage of volatility to force the correlation. Maybe you shift the sample start date, maybe you skip a normalization dimension. The directors would ask about things like normalization and if things were measured over a ncertain time period, but if you had some excuse or answer prepared, it was enough to get a nod and some recognition for your presentation. Heck, sometimes your team manager would do the presentation on your behalf to lay claim to some of that precious impact. Then, after the new review cycle starts, priorities change and you never have to follow up on the holes in your data. Or, you can even admit it has holes, you already got credit.
m12k 5y ago
Humans are instinctively drawn to a lot of things that are really bad for us in significant amounts, because we are designed to fit into an environment where those things are scarce. Sugar, fat, inactivity, "interesting news", outrage. These all steal our attention because that response helped our ancestors survive and pick out these rare but important treats. But in typical human fashion we've now crafted a world that gives us these things all the time, and it's making us sick and miserable as a result. "But we're just giving the people what they want" some might say. Well, depends on the definition of "want". In a taste test, junk food would win over broccoli for me, hands down - but it's broccoli that ends up on my table more often than not, because it's what makes me happier and healthier in the long run. It wasn't until my thirties before I realized just how sluggish I got after eating junk food. But if Facebook was running our diets, their algorithm would long since have "optimized" its way to junk food for all of us. Sometimes the worst thing you can do is give people exactly what they ask for. Being healthy in a world like the one we've created requires much, much more restraint and self-discipline than it used to. "The algorithm" is basically the digital incarnation of the little devil on our shoulder whispering that we should treat ourselves.
- agent008t 5y ago
  I agree with you, particularly when it comes to information consumption. On-demand services are bad enough; cleverly optimized services that push content are even worse. But in some ways, if you hang out in the right circles, being healthy is a lot easier today. Alcohol consumption seems to be down, people exercise a lot more, being out of shape in your 30s is not seen as 'the norm'. So it is probably easier psychologically to go for a jog now when parks are full of joggers than to do it in 1950s when people would think a 30-something person out for a jog is a bit strange. In this way, healthy fads can actually be very positive. I find it very likely that in the near future a culture will emerge that tries to minimize the use of technology and information hygiene becomes a part of 'normal' life.
- wepple 5y ago
  > But if Facebook was running our diets, their algorithm would long since have "optimized" its way to junk food for all of us. I really, really like this comparison. If we fed our bodies in the same way we let tech company algorithms feed our minds, we’d have three Big Macs for breakfast washed down with a shot of whiskey and a couple lines of cocaine
bertil 5y ago
I’ve worked on similar experiments (starting about Stories) and we included long-term holdout experiment to measure the compounded impact of sharing. The fear at the time was more about professional content (with millions of views) vs. your friends. The worry was because people engaged more with the former but posted more if they saw the later. I left before conclusions were drawn, but (according to press reports) those experiments changed the goals to more leading indicator of long-term trend, like posting rather than Likes. I joined again more recently and expected to see that it influenced the company. I’m not working there at the moment.
pdinny 5y ago
> The employees are incentivized to do so and this is what they measure, so this is what they build I also work in data teams and this is directly my observation too. Closely related are Conway's Law (you ship your org chart) and Goodhart's Law (commonly paraphrased as 'When a measure becomes a target, it ceases to be a good measure.') In Netflix's case I certainly feel the shipped product is too close of a reflection of org structure and incentives and user experience suffers as a result. This feels particularly evident in the treatment of thumbnails and title descriptions. Often I find myself clicking through on a thumbnail in order to read the description only to find that the I've already read the description and didn't want to watch that title, but the thumbnail has since changed. I'm sure there is some team in Netflix whose sole purpose is increasing thumbnail clickthrough rates. And they are probably succeeding in that respect by changing the thumbnails. They get to win at their portion of some funnel, even if the net result is a lousy user experience.
- denton-scratch 5y ago
  > and title descriptions To my mind, it's the title descriptions that are the most clickbaity. I get the sense that Netflix are beding over backwards to ensure that you can't form an impression of the flic from that description. I cancelled my Netflix subscription. Less and less of the titles are well-known 3rd-party productions; more and more are Netflix productions that I have found to be rather dull. Their ability to "divine my preferences" was never much good, but it's got worse. Of course, they don't really care about my preferences; what matters for Netflix is what they would prefer me to watch, apparently. Youtube's ability to discern my preferences also seems to have nosedived over the last year. I watched one programme about a certain WWII aircraft; YT then started pushing at me an unending diet of aircraft restoration docs and amateur historian channels.
omniscient_oce 5y ago
Spotify's UI feels like it has succumbed to this.
- stronglikedan 5y ago
  I recently got a very gracious trial period for Youtube Music Premium, so I'm giving it a shot. It's UI is what Spotify used to be, before they crippled it nearly completely for my use cases. I'm seriously considering switching because of this. I'm just hoping that YT Music can replicate Spotify's one killer feature - Spotify Connect - before my trial is over.
  - city41 5y ago
    I really like Amazon Music. The interface is simple and not gimmicky. Just search for music and play it.
- criddell 5y ago
  Definitely has. I remember when podcasts first showed up. Spotify isn't my podcast player so that is pretty much useless to me. I looked through the options for how to hide podcasts and couldn't find anything. I wish I could at least move the podcasts section to the bottom of the page.
dangus 5y ago
Unfortunately it probably works because the business model depends on people forgetting about their subscription, e.g., Planet Fitness. If you have a nice experience for a week or two and the price is low enough you'll probably never watch Netflix (e.g., ME) and keep on paying for it.
Vrondi 5y ago
Yes, Facebook is now almost useless. The only reason to be there is to keep track of friends and family, but it has gotten to the point where if a family member has a big life event, I'm hearing about it through traditional in-person or on-voice-call word of mouth _faster_ than through Facebook, because Facebook keeps deciding for me what it wants me to see. Netflix has kept trying to make it harder for me to find what I want for the last few years, and I use it less and less, and now pay for fewer accounts than in the past. They are in danger of losing customers, with their annoying mouseover/autoplay/clickbait interface.
- relativ575 5y ago
  > Yes, Facebook is now almost useless. T Not my experience. A contractor recently told me almost all of his business were driven by his company's FB page. I'm also aware of a few small businesses who do most of their business via FB connections. FB is well and alive in the mainstream, even if it skews toward more older group of people.
TuringNYC 5y ago
>> But behind the scenes some of us had this thought that our observations only amounted to short-term gains. I'm convinced this must be the case at Facebook. I used to love the newsfeed but it has steadily declined so much for me that I stopped using it. It used to highlight a wide variety of friends updates. Then, only starred "close friends" and now mostly junk news items. Amazing things happen for friends (weddings, births) and I dont hear about it (no they aren't blocking me, i've confirmed). I can to individually to each person's profile and see everything, but how realistic is that. I'm sure there was a short boost as people checked more and more to news that didn't appear. Now, you check five times, you get the same 30 newsfeed items over and over and then "you're all caught up". I can totally see the short term boost and long term decline with such a strategy.
aeoleonn 5y ago
Excellent. There is a silver lining: It will lead to the end of Facebook and hopefully these practices you’ve described... once the lesson is learned. The end of Facebook, Snap, and other scientifically-designed-to-be-attention-exploiting apps/sites would be very positive.
- ckosidows 5y ago
  Or, conversely, apps like TikTok, etc, etc will learn from past failures and develop even stronger algorithms to capture attention ad infinitum.
  - seph-reed 5y ago
    I tried out TikTok for a few nights. It very, very quickly got repetitive. I think the short-form content sphere can only fit so much of an arc in what little time it has. And getting people to sit through it requires being a bit click-baity. And everything was obviously staged, and a lot of times it seemed like... Okay, so when I was a kid, we made stupid videos too. But we never expected anyone to watch them. And they were often 5-10 minutes long in the end. On TikTok you can tell that the priority of being filmed is to be seen. They aren't just having fun. It's like TV, and weirds me out.

vasco 5y ago

They don't seem to have fallen off, are still the most widely used platforms in the world with billions of users in total. It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The only thing I've seen work is something like Pixar's brain trust - you do testing but you also rely on the good judgement of a small group of people with a strong sense of vision. Judging good judgement is pretty hard though, and you won't know if they're just full of it or winging it unless you try and trust. And so everyone does A/B testing which to be honest sounds much better than your average PM making decisions that are pulled out of their ass.

deleted 5y ago
[deleted]
HWR_14 5y ago
> It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? I mean, if corporate leadership isn't able to better direct the path of a product than the results of A/B tests, what value do they really have?
piva00 5y ago
> It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The alternative involves creativity to create a vision to achieve. A/B testing are pretty good tools to evaluate differing opinions on some vision, as the only deciding tool for what is worth to keep or not it's pretty short-sighted. Is it really that much better? I lived through the rise of A/B testing, professionally I've been responsible for implementing platforms for experimentation in at least 5 different companies and over the years I saw it transition from a tool to a crutch. Everything now is "data-based" which means incessant A/B testing, it still depends on your average PM deciding the parameters of the experiment, leading to the current state of affairs: a bunch of uninspired features that are solely judged on the data produced, no soul, no vision behind most of it. And it feels exactly like that, I can feel how tech products just became that, an incessant stream of features being tested, implemented or tossed. I'd much rather have data-supported intuition than this statistical machine in motion.
fisf 5y ago
The problem is not only that it could be a local Maxima. It might also measure the wrong metric all together. Is the platform widely used and the business successful because content makes users happy and improves their lives? Or is engagement high, because users have become accustomed to whatever a company is feeding them and look for something shallow to fill up their time. If you only measure engagement as metric, without any deeper reflection and judgment, then heroin is a perfect product in terms of A/B testing.

lukestevens 5y ago

Yes, I'd love to see a more robust approach to novelty decay rates (to make up a metric) in A/B testing, especially around engagement. You'd think that would be of interest to the business to know that X "successful" intervention had a typical average lifetime of Y before reverting to the mean, for example. Then again, there's the idea that competitive advantage for social networks is just finding enough novel interventions before the novelty of the platform itself is exhausted, and FB has no short supply of other platforms to milk in that regard.

dreamer7 5y ago

What was the duration for which you collected results? Was it 2 weeks or a month etc?

mgraczyk 5y ago
Depends on the change and effect size, also on product. Typically somewhere between 1 week and 1 quarter. Almost always there is a backtest after shipping in which a small percent of users do not receive the change for 1-6 months (a "holdout")

keiferski 5y ago

This is the consequence of lack of vision. What is Netflix's long term goal? Ideally, it would be to fund and broadcast fantastic cinema. Yet, I really don't get the impression that contributing to the art form of film is very high up on the metrics chart.

pizza234 5y ago
I'm of the opposite opinion. They're executing their vision perfectly. Netflix is very aggressively pursuing the bottom line, and they're not doing it differently from a drug dealer. It's not random that the term "Netflix binge" exists. Netflix interface is designed to shove content at the users' face at all costs, and in a very disrespectful way. Here there are some disgusting dark patterns: - it took them a long time to give the user the option to remove movies from the watching now list - there's no way to remove already watched movies from the panel; their trailer is even sometimes displayed in the top frame - they randomize the global order of the horizontal lists, so that one needs every time to hunt the user list (watching now/my list) through all the panel For a short time, I even experienced a bug where movies with (my) negative rating were not greyed out. I can imagine the engineering thinking "LOW PRIORITY!". On my (fictional) "drug-dealing practices" rating, Netflix is high on top with Amazon.
- john_minsk 5y ago
  100% agree. I was awaiting for Netflix thread to pop-up to ask (hopefully someone who works there): Why there is no way to tell Netflix that I already watched the show and I'm not planning to rewatch it again any time soon or hide the ones I tried and decided not watch (or may be watched before)? No matter how many times you show me them I'm not going to watch... You can still have nice features around it: separate list at the end of scroll with my old movies/shows (may be with number of times I watched it?). Once new seasons released send them all the way to to the top etc. And I will be able to discover something new. At the moment it is cluttered with things I don't want to watch so lately I was not happy with Netflix at all. You should see such patterns in your data.
- i2shar 5y ago
  Add one more to the list - and this one drives me crazy to no end - on my Chromecast with GTV the wretched thing starts autoplaying even if I just want to read the synopsis. This, despite turning off all autoplay options in my profile. What is the friggin rush?! Why won't you let me browse/read distraction free?!

2Gkashmiri 5y ago

i do not understand snapchat and their business model. they started out as a sexting app that deletes the photos and that is where it remained for a long time. i remember news of how "deleted snapchat photos can be undeleted" tutorials and i tried to investigate for fun but i could not understand the UI back then and the tutorials didnt work so i dropped it. Apparently its pretty big these days in the kids and they have "streaks" as you mention. i hear multiple hundred thousand streaks and other things but to what end? how is snapchat justifying the storage, bandwidth and processing of data? i mean facebook has ads which they earn, same for twitter but what about snapchat? do they have a sustainable revenue stream because no where i heard snapchat had a paid option. reddit was user funded for a long time with daily goals and all that so that was not an issue mostly but what snapchat done to address that. snapchat is all the rage in my city, kids who are now on online classes send snaps because i hear about it. everyone knows how to make that twistface pout or whatever but why? genuinely curious about it

cosmie 5y ago
> i do not understand snapchat and their business model. Ads[1]. Initially it was just promotional lenses/filters and required a high touch process and substantial media spend commitment. But looking at that site, they now have a full on self-service ad platform with minimal spend commitment, helper software to make lenses/filters, and have added several other ad formats beyond that one. So they've essentially monetized the same way as every other "free" platform has – capitalize on all the eyeballs they have to siphon off a cut of marketing dollars for themselves. [1] https://forbusiness.snapchat.com/ https://forbusiness.snapchat.com/
- 2Gkashmiri 5y ago
  cool. i did not know that
ladon86 5y ago
$2.5bn revenue last year. You've heard of streaks (which were a very early feature), and you've probably heard of Stories too, right? Video ads appear in between Stories video rolls.

tomnipotent 5y ago

> The data/engineering/product loops at tech companies favor boosting short-term metrics I think this hits the nail on the head. Leadership is hard, and executives can be many layers removed from the frontline. It seems that A/B tests and family have proven a really effective vehicle to communicate across leadership levels what has happened, why it happened, and the impact it had on the business. I'm definitely guilty as charged of having participated in this "pyramid scheme". Teams get a tool to structure work and determine success/fail. Managers get a tool to detail exact impact to leadership on how they're solving business problems. Executives get a tool showing how teams are being deployed on thoughtful projects with an iterative and rigorous scientific approach that helps increase confidence around deploying capital. The board gets some slides during the quarterly meeting and maybe might even see some charts showing growth and less cash burn. Eventually over longer periods like a year, executives will notice different trends than what the short-term numbers are showing and will have to make real hard decisions about how to steer the ship. It's a hell of a hamster wheel.

dr_dshiv 5y ago
How might a streaming service be designed to increase user happiness and wellbeing? From a positive design perspective, could certain streaming features give users a greater sense of purpose or goal accomplishment (e.g., mastering a genre) or help users meaningfully connect to others (e.g., seeing what friends have recommended lately)? While measuring the effects of a feature on wellbeing outcomes is substantially harder than measuring the effects of a feature on engagement (time spent), it is worth measuring/optimizing in the long run. The hypothesis is that capitalist enterprises that make their users feel happier and more fulfilled will, in the end, be more successful than enterprises that merely provide empty satisfaction. If not, the future could be very bleak.
- verve_rat 5y ago
  Maybe... make good shows?
  - dr_dshiv 5y ago

anon9001 5y ago

> I can only fill up so much time with garbage like that before I'm bored. Don't sell yourself short! You're surrounded by tons of people that never get their fill of content. Just relax and let it happen ;)

mgraczyk 5y ago

As another former Facebook employee, I mostly agree with this. One nitpick is that although you may have not looked at the holdouts often, somebody definitely did. Different teams use their holdouts differently, but leadership probably looked at your holdouts at least a few times per quarter. And now with recent infra and changes, it's very likely that data scientists or product managers somewhere in your org are responsible for explaining results to higher ups at least quarterly. All that is to say, the cause isn't that Facebook (and Netflix) aren't thinking about or monitoring things in the long term. It's that they are measuring the wrong things, because it's very hard to measure things like "this is clickbait".

beerandt 5y ago

Netflix has somehow conditioned me to expect disappointment in movies with interesting looking cover art, which I now actively ignore, and don't even click through for descriptions anymore. "Huh, that looks interesting. Not falling for that old trick again! What's that half-off-screen ambiguous cover on the next row down?"

roystonvassey 5y ago

Interesting insights. Online Analytics is now really nuanced - you need to know what metrics are important for * your * business, not just use boilerplate kpis. For instance, an e-commerce website is clearly looking to lead the user to a lot purchase and the more they purchase, it’s good for the business so kpis around sales conversion help and recommenders help your business increase sales. For Netflix though, the users have already paid for the service after which they land on the website. Most users I imagine then expect to be provided all that Netflix has to offer in an easy way. So if I was a Netflix product owner, I’d be more interested in Kpis around search-ability, having an anti algorithm that “suggests” completely random obscure shows, “switchability” of users - how less of a time do users spend on a movie or show. I imagine they’re doing this but as a user I don’t see this at least - they show the same old stale recommendations for me, I’m always trying to hack their search to find what I want and they continue to invest in content that’s mostly miss than hit. I wish they at least had a directory for me to browse through (at least I’ll be driving their engagement metrics to help them drive their valuations)

kofejnik 5y ago
They don’t want you to watch obscure shows, they want you to watch the top ones (in your region!) as they are preloaded into your cdn already.
- roystonvassey 5y ago
  Right - but why would they assume I want to watch the "top" show? I've already paid for it and they benefit by having me browse and allowing myself to find that I like right? And what I like is not necessarily what they think others might me like - what one watches is highly variable (can change day to day, even time of the day) and influenced by so many factors that are personal.
  - pnut 5y ago
    Netflix is now a major player in the entertainment business, and I would guess there is a pay-to-play element behind the scenes as well as ROI on new productions, forcing them to jam these options down our throats. User viewing preferences seem to be a secondary consideration.

qwertox 5y ago

> i.e. a newsfeed algorithm that had better engagement. Sometimes (but rarely) I click on something because I ask myself "Why would these a---- show me this? I want to know more about what's behind it in order to understand what could have made their algorithm chose me as a candidate for this content". Or to check where the scam-URL would lead me to, because this also happens, that ads sold on Google and Facebook are scams, links leading to malware. Congratulations to them, if they think that this is the kind of engagement which should be valued, that any kind of engagement is a good and healthy engagement. These companies are so rotten and their engineers just don't care.

bertil 5y ago
There were tools to help answer your question (predictably called “Why am I seeing this?” after the button copy). Those had so little engagement they were not worth maintaining.

uxcolumbo 5y ago

Off topic: To me they seem to focus on quantity over quality. Can you recommend some of that quality content? I always quit after browsing their catalogue for 20 minutes or so.

Fiahil 5y ago

I think you can maintain the feedback loop without degrading the user experience if you actually ask them what they want to be shown. The problem lies in that "engagement" definition : if the user clicked on a thumb, then they must have been attracted to it. In reality, there is considerably more explanation for that click : They could be looking for a specific title, mistakenly thought _this_ actor was playing in the movie, clicked on it to reset the search, wondered if this was a show or a film, missclicked, just picked the least annoying in an ocean of shitty propositions... An algorithm alone can never find its way for better suggestions if it lacks _intent_. The outcome of this lack of data and the lack of data scientists' imagination, will always be stuck in a local minima. So, what solution do you have to gather this precious intent and act on it ? Well, you start small on predictable things : "what would you recommend to Agatha if they liked _Expendables_ ?" "Is _RED_ a good addition to the list if I'm looking for Action/Comedy films ?" And later "Are you looking for Action/Comedy films ? here is some proposals" It works the same for different approaches, for picking thumbnails, or promoting movies. Just ask user for what's motivate them. If Netflix had asked me last week, I would have told them their catalog is getting worse and worse so we prefer using disney+ instead.

notafraudster 5y ago
The idea is that these kinds of sources of error will be noise, since they should be random with respect to experimental assignment in any kind of A/B test. In a small sample you could imagine chance imbalances, but given that I'm experimentally assigning someone to see either a thumbnail of John Krasinski or Steve Carell, it's unclear why the "I misclicked" scenario is going to have a higher rate in one rather than the other. And experiments at large companies like Netflix are almost infinitely powered. I would think a bigger threat to this approach would be being stuck in a local maxima.

antiterra 5y ago

It seems even worse than that. During my stints in social media, there would be ‘research projects’ that were simply fishing expeditions for meaningful data. Something like: which user demographics have a correlation to behavior x. And then you’d just search and search and search and hope you found something. If you didn’t find something, then you’d be encouraged by a team level manager to adjust or tweak some approach to take advantage of volatility to force the correlation. Maybe you shift the sample start date, maybe you skip a normalization dimension. The directors would ask about things like normalization and if things were measured over a ncertain time period, but if you had some excuse or answer prepared, it was enough to get a nod and some recognition for your presentation. Heck, sometimes your team manager would do the presentation on your behalf to lay claim to some of that precious impact. Then, after the new review cycle starts, priorities change and you never have to follow up on the holes in your data. Or, you can even admit it has holes, you already got credit.

m12k 5y ago

Humans are instinctively drawn to a lot of things that are really bad for us in significant amounts, because we are designed to fit into an environment where those things are scarce. Sugar, fat, inactivity, "interesting news", outrage. These all steal our attention because that response helped our ancestors survive and pick out these rare but important treats. But in typical human fashion we've now crafted a world that gives us these things all the time, and it's making us sick and miserable as a result. "But we're just giving the people what they want" some might say. Well, depends on the definition of "want". In a taste test, junk food would win over broccoli for me, hands down - but it's broccoli that ends up on my table more often than not, because it's what makes me happier and healthier in the long run. It wasn't until my thirties before I realized just how sluggish I got after eating junk food. But if Facebook was running our diets, their algorithm would long since have "optimized" its way to junk food for all of us. Sometimes the worst thing you can do is give people exactly what they ask for. Being healthy in a world like the one we've created requires much, much more restraint and self-discipline than it used to. "The algorithm" is basically the digital incarnation of the little devil on our shoulder whispering that we should treat ourselves.

agent008t 5y ago
I agree with you, particularly when it comes to information consumption. On-demand services are bad enough; cleverly optimized services that push content are even worse. But in some ways, if you hang out in the right circles, being healthy is a lot easier today. Alcohol consumption seems to be down, people exercise a lot more, being out of shape in your 30s is not seen as 'the norm'. So it is probably easier psychologically to go for a jog now when parks are full of joggers than to do it in 1950s when people would think a 30-something person out for a jog is a bit strange. In this way, healthy fads can actually be very positive. I find it very likely that in the near future a culture will emerge that tries to minimize the use of technology and information hygiene becomes a part of 'normal' life.
wepple 5y ago
> But if Facebook was running our diets, their algorithm would long since have "optimized" its way to junk food for all of us. I really, really like this comparison. If we fed our bodies in the same way we let tech company algorithms feed our minds, we’d have three Big Macs for breakfast washed down with a shot of whiskey and a couple lines of cocaine

bertil 5y ago

I’ve worked on similar experiments (starting about Stories) and we included long-term holdout experiment to measure the compounded impact of sharing. The fear at the time was more about professional content (with millions of views) vs. your friends. The worry was because people engaged more with the former but posted more if they saw the later. I left before conclusions were drawn, but (according to press reports) those experiments changed the goals to more leading indicator of long-term trend, like posting rather than Likes. I joined again more recently and expected to see that it influenced the company. I’m not working there at the moment.

pdinny 5y ago

> The employees are incentivized to do so and this is what they measure, so this is what they build I also work in data teams and this is directly my observation too. Closely related are Conway's Law (you ship your org chart) and Goodhart's Law (commonly paraphrased as 'When a measure becomes a target, it ceases to be a good measure.') In Netflix's case I certainly feel the shipped product is too close of a reflection of org structure and incentives and user experience suffers as a result. This feels particularly evident in the treatment of thumbnails and title descriptions. Often I find myself clicking through on a thumbnail in order to read the description only to find that the I've already read the description and didn't want to watch that title, but the thumbnail has since changed. I'm sure there is some team in Netflix whose sole purpose is increasing thumbnail clickthrough rates. And they are probably succeeding in that respect by changing the thumbnails. They get to win at their portion of some funnel, even if the net result is a lousy user experience.

denton-scratch 5y ago
> and title descriptions To my mind, it's the title descriptions that are the most clickbaity. I get the sense that Netflix are beding over backwards to ensure that you can't form an impression of the flic from that description. I cancelled my Netflix subscription. Less and less of the titles are well-known 3rd-party productions; more and more are Netflix productions that I have found to be rather dull. Their ability to "divine my preferences" was never much good, but it's got worse. Of course, they don't really care about my preferences; what matters for Netflix is what they would prefer me to watch, apparently. Youtube's ability to discern my preferences also seems to have nosedived over the last year. I watched one programme about a certain WWII aircraft; YT then started pushing at me an unending diet of aircraft restoration docs and amateur historian channels.

omniscient_oce 5y ago

Spotify's UI feels like it has succumbed to this.

stronglikedan 5y ago
I recently got a very gracious trial period for Youtube Music Premium, so I'm giving it a shot. It's UI is what Spotify used to be, before they crippled it nearly completely for my use cases. I'm seriously considering switching because of this. I'm just hoping that YT Music can replicate Spotify's one killer feature - Spotify Connect - before my trial is over.
- city41 5y ago
  I really like Amazon Music. The interface is simple and not gimmicky. Just search for music and play it.
criddell 5y ago
Definitely has. I remember when podcasts first showed up. Spotify isn't my podcast player so that is pretty much useless to me. I looked through the options for how to hide podcasts and couldn't find anything. I wish I could at least move the podcasts section to the bottom of the page.

dangus 5y ago

Unfortunately it probably works because the business model depends on people forgetting about their subscription, e.g., Planet Fitness. If you have a nice experience for a week or two and the price is low enough you'll probably never watch Netflix (e.g., ME) and keep on paying for it.

Vrondi 5y ago

Yes, Facebook is now almost useless. The only reason to be there is to keep track of friends and family, but it has gotten to the point where if a family member has a big life event, I'm hearing about it through traditional in-person or on-voice-call word of mouth _faster_ than through Facebook, because Facebook keeps deciding for me what it wants me to see. Netflix has kept trying to make it harder for me to find what I want for the last few years, and I use it less and less, and now pay for fewer accounts than in the past. They are in danger of losing customers, with their annoying mouseover/autoplay/clickbait interface.

relativ575 5y ago
> Yes, Facebook is now almost useless. T Not my experience. A contractor recently told me almost all of his business were driven by his company's FB page. I'm also aware of a few small businesses who do most of their business via FB connections. FB is well and alive in the mainstream, even if it skews toward more older group of people.

TuringNYC 5y ago

>> But behind the scenes some of us had this thought that our observations only amounted to short-term gains. I'm convinced this must be the case at Facebook. I used to love the newsfeed but it has steadily declined so much for me that I stopped using it. It used to highlight a wide variety of friends updates. Then, only starred "close friends" and now mostly junk news items. Amazing things happen for friends (weddings, births) and I dont hear about it (no they aren't blocking me, i've confirmed). I can to individually to each person's profile and see everything, but how realistic is that. I'm sure there was a short boost as people checked more and more to news that didn't appear. Now, you check five times, you get the same 30 newsfeed items over and over and then "you're all caught up". I can totally see the short term boost and long term decline with such a strategy.

aeoleonn 5y ago

Excellent. There is a silver lining: It will lead to the end of Facebook and hopefully these practices you’ve described... once the lesson is learned. The end of Facebook, Snap, and other scientifically-designed-to-be-attention-exploiting apps/sites would be very positive.

ckosidows 5y ago
Or, conversely, apps like TikTok, etc, etc will learn from past failures and develop even stronger algorithms to capture attention ad infinitum.
- seph-reed 5y ago
  I tried out TikTok for a few nights. It very, very quickly got repetitive. I think the short-form content sphere can only fit so much of an arc in what little time it has. And getting people to sit through it requires being a bit click-baity. And everything was obviously staged, and a lot of times it seemed like... Okay, so when I was a kid, we made stupid videos too. But we never expected anyone to watch them. And they were often 5-10 minutes long in the end. On TikTok you can tell that the priority of being filmed is to be seen. They aren't just having fun. It's like TV, and weirds me out.