4 ms·

As a computer program, this is probably not feasible. While in theory you are possibly capable of writing the algorithms for your own recommendation engine usin

As a computer program, this is probably not feasible. While in theory you are possibly capable of writing the algorithms for your own recommendation engine using whatever ML tools are popular today, the issue you'd have is data. YouTube's recommendation algorithm effectively only works because it's capable of aggregating millions of users behavior to feed into their own system. Getting that data personally would be very difficult both because YouTube doesn't want to enable competitors, and because consumers would be extra freaked out to find out about some random engineer outside of YT getting their watch history. As a non-program, you can always depend on the means that humans have used for content discovery since time immemorial: word of mouth.

imglorp 6y ago
Well just for me I don't care what's trending so don't need the feed of other people's views. But absolutely needing a feed of new posts might be problem.
- creato 6y ago
  That's really missing the point. Recommendation engines are a lot more dumb than you are thinking. They don't understand the content at all. They only know that thousands of users that watched the same videos as you in the past also watched this other new video, so you might like it as well. A recommendation engine that works solely based on your own watch history would require a totally different approach using algorithms that are years/decades/forever away from existing.
  - imglorp 6y ago
    Good points. It does make sense that the job is simpler with everyone's history and inputs, if you can keep PII out of it. Are there any good references on recommendation aggregation algorithms? Here's a simple proposal that keeps everyone's stuff anonymous and allows a rec service to pay bills. Distribute the pool of everyone's watch histories/prefs/etc using IPFS, which is a DHT with static names and some storage. User clients put the share data into IPFS and register into a public pool. The client remembers their own hash. Rec services (there can be multiple) pull everyone's sets periodically and aggregate/indexe them. Users can then retrieve a batch of new recs by sending a small fee to the service of their choice along with their hash and the server will place some new recs into their bucket. If the recs are not suitable, they can try another service.
    - halflings 6y ago
      If I send you links to 2-3 rare videos and you view them, I can probably uniquely identify you (or restrict the list of potential users to ~5 users then recognize you based on other videos you would have watched). I then have access to your full video watching history. Even without this, I could just check videos you shared on Twitter, follow the same process, and have a decent chance of identifying you. Anonymization is very very hard. Maybe federated learning is more promising here, keeping all your data privately stored on your own devices.
  - halflings 6y ago
    > That's really missing the point. Recommendation engines are a lot more dumb than you are thinking. They don't understand the content at all. I think you're the one underestimating modern recommender systems, especially the ones built by advanced teams in big tech companies. They absolutely use content features. Among other advantages, this counters the cold start problem (how will you learn about a new video if it doesn't get recommended to anyone).
- aasasd 6y ago
  Have you ever seen the torrent of garbage that is new posts on e.g. Reddit or Imgur? You only see decent posts on the ‘front page’ because some people sift through that and filter out teenagers' low-effort attempts. YouTube has hours of video uploaded every minute. A non-centralized, context-less recommendation system would mean that you are offered those crap videos together with ones that are the standard for ‘good’ now. Just get into the long tail on any search results and see if you want those. A non-monopolized video discovery would be splendid, only who knows how to achieve that. Perhaps it would be more realistic to do the recommendations based on the numbers of ‘retweets’ in federated networks—in a more ‘social’ style instead of views (dunno if PeerTube already offers that). Also, an open system will be incessantly gamed.
luplex 6y ago
Open Data is a thing, I can imagine some PeerTube instances would publicly share their data (anonymously etc), in order to build open recommendation systems. Personally, I would gladly donate my (anonymous etc) data for this noble goal.
- Andrex 6y ago
  Open data + federated learning (somewhat ironically a Google invention) could help with this, but it has a long way to go, and a lot of opportunities for incumbents to fight back.
- danielheath 6y ago
  The problem is (as with all open/federated systems) spam. How do you figure out which data has been faked to drive views to particular content?
dredmorbius 6y ago
Data offers diminishing returns, and at sufficient quantity introduces vetting and validity problems --- see Amazon's reviews crisis or fake Yelp reviews, as examples. Data quality and relevance matter far more.
spatley 6y ago
Except that youtube’s recommendation engine It terrible and getting worse. I am on youtube all the time and all it ever recommends is things I already subscribe to. And it seems to be steering further into things I have already watched.
csande17 6y ago
This is maybe a tangent, but do these ML-powered recommendation algorithms actually, like, work well for people? I use both YouTube and Netflix pretty extensively. These companies are both held up as ML success stories. But, at least in my personal experience, they suck! My front-page feeds consist of (1) content that the platform knows I have already seen because IT IS IN MY WATCH HISTORY, (2) content that is trivially related, like by having the the same channel/director/actors, and (3) random junk I have no interest in watching and always regret clicking on. It's not that the content isn't out there -- I'm able to find stuff I like through "big spreadsheet of all the Netflix movies" websites, online recommendations from humans, and YouTube channel crossovers/cameos. But I've never had that magical "the algorithm knows you better than you know yourself" experience that every ML engineer I've ever spoken to seems to be convinced their systems achieve. Are all ML engineers mindless drones with exactly average movie preferences, or am I some kind of incomprehensible weirdo?
- SifJar 6y ago
  > random junk I have no interest in watching and always regret clicking on if you regret clicking on it, that means you did click on it and the recommendation did it's job; remember that the job of the recommendation engine is not (directly) to suggest content you'll like, it's to suggest content that will get you to click. Ideally those goals would be aligned, but not necessarily. YouTube is simply incentivised (via ad revenue) to get you to click on as many videos as possible.
  - csande17 6y ago
    At least in my case, it's not that I'm constantly clicking on things that it turns out I dislike. Rather, it's that the few times I have clicked on a recommendation (because hey, maybe they know something I don't, maybe Sonic the Hedgehog actually is a good suggestion for people who liked Eternal Sunshine of the Spotless Mind), I've regretted it.
    - SifJar 6y ago
      Right, but my point is that from YouTube's perspective it doesn't "matter" if you like or dislike the video, if they can get you to click. And clearly this strategy works on some people, or they wouldn't be doing it. So while it may not be overly successful on you (i.e. you only click occasionally), if it works on a significant portion of people it could still be a successful strategy overall.

imglorp 6y ago

Well just for me I don't care what's trending so don't need the feed of other people's views. But absolutely needing a feed of new posts might be problem.

creato 6y ago
That's really missing the point. Recommendation engines are a lot more dumb than you are thinking. They don't understand the content at all. They only know that thousands of users that watched the same videos as you in the past also watched this other new video, so you might like it as well. A recommendation engine that works solely based on your own watch history would require a totally different approach using algorithms that are years/decades/forever away from existing.
- imglorp 6y ago
  Good points. It does make sense that the job is simpler with everyone's history and inputs, if you can keep PII out of it. Are there any good references on recommendation aggregation algorithms? Here's a simple proposal that keeps everyone's stuff anonymous and allows a rec service to pay bills. Distribute the pool of everyone's watch histories/prefs/etc using IPFS, which is a DHT with static names and some storage. User clients put the share data into IPFS and register into a public pool. The client remembers their own hash. Rec services (there can be multiple) pull everyone's sets periodically and aggregate/indexe them. Users can then retrieve a batch of new recs by sending a small fee to the service of their choice along with their hash and the server will place some new recs into their bucket. If the recs are not suitable, they can try another service.
  - halflings 6y ago
    If I send you links to 2-3 rare videos and you view them, I can probably uniquely identify you (or restrict the list of potential users to ~5 users then recognize you based on other videos you would have watched). I then have access to your full video watching history. Even without this, I could just check videos you shared on Twitter, follow the same process, and have a decent chance of identifying you. Anonymization is very very hard. Maybe federated learning is more promising here, keeping all your data privately stored on your own devices.
- halflings 6y ago
  > That's really missing the point. Recommendation engines are a lot more dumb than you are thinking. They don't understand the content at all. I think you're the one underestimating modern recommender systems, especially the ones built by advanced teams in big tech companies. They absolutely use content features. Among other advantages, this counters the cold start problem (how will you learn about a new video if it doesn't get recommended to anyone).
aasasd 6y ago
Have you ever seen the torrent of garbage that is new posts on e.g. Reddit or Imgur? You only see decent posts on the ‘front page’ because some people sift through that and filter out teenagers' low-effort attempts. YouTube has hours of video uploaded every minute. A non-centralized, context-less recommendation system would mean that you are offered those crap videos together with ones that are the standard for ‘good’ now. Just get into the long tail on any search results and see if you want those. A non-monopolized video discovery would be splendid, only who knows how to achieve that. Perhaps it would be more realistic to do the recommendations based on the numbers of ‘retweets’ in federated networks—in a more ‘social’ style instead of views (dunno if PeerTube already offers that). Also, an open system will be incessantly gamed.

luplex 6y ago

Open Data is a thing, I can imagine some PeerTube instances would publicly share their data (anonymously etc), in order to build open recommendation systems. Personally, I would gladly donate my (anonymous etc) data for this noble goal.

Andrex 6y ago
Open data + federated learning (somewhat ironically a Google invention) could help with this, but it has a long way to go, and a lot of opportunities for incumbents to fight back.
danielheath 6y ago
The problem is (as with all open/federated systems) spam. How do you figure out which data has been faked to drive views to particular content?

dredmorbius 6y ago

Data offers diminishing returns, and at sufficient quantity introduces vetting and validity problems --- see Amazon's reviews crisis or fake Yelp reviews, as examples. Data quality and relevance matter far more.

spatley 6y ago

Except that youtube’s recommendation engine It terrible and getting worse. I am on youtube all the time and all it ever recommends is things I already subscribe to. And it seems to be steering further into things I have already watched.

csande17 6y ago

This is maybe a tangent, but do these ML-powered recommendation algorithms actually, like, work well for people? I use both YouTube and Netflix pretty extensively. These companies are both held up as ML success stories. But, at least in my personal experience, they suck! My front-page feeds consist of (1) content that the platform knows I have already seen because IT IS IN MY WATCH HISTORY, (2) content that is trivially related, like by having the the same channel/director/actors, and (3) random junk I have no interest in watching and always regret clicking on. It's not that the content isn't out there -- I'm able to find stuff I like through "big spreadsheet of all the Netflix movies" websites, online recommendations from humans, and YouTube channel crossovers/cameos. But I've never had that magical "the algorithm knows you better than you know yourself" experience that every ML engineer I've ever spoken to seems to be convinced their systems achieve. Are all ML engineers mindless drones with exactly average movie preferences, or am I some kind of incomprehensible weirdo?

SifJar 6y ago
> random junk I have no interest in watching and always regret clicking on if you regret clicking on it, that means you did click on it and the recommendation did it's job; remember that the job of the recommendation engine is not (directly) to suggest content you'll like, it's to suggest content that will get you to click. Ideally those goals would be aligned, but not necessarily. YouTube is simply incentivised (via ad revenue) to get you to click on as many videos as possible.
- csande17 6y ago
  At least in my case, it's not that I'm constantly clicking on things that it turns out I dislike. Rather, it's that the few times I have clicked on a recommendation (because hey, maybe they know something I don't, maybe Sonic the Hedgehog actually is a good suggestion for people who liked Eternal Sunshine of the Spotless Mind), I've regretted it.
  - SifJar 6y ago
    Right, but my point is that from YouTube's perspective it doesn't "matter" if you like or dislike the video, if they can get you to click. And clearly this strategy works on some people, or they wouldn't be doing it. So while it may not be overly successful on you (i.e. you only click occasionally), if it works on a significant portion of people it could still be a successful strategy overall.