2 ms·

> a stream processing tool for end users This is it! This is the fundamental problem that we need to solve, and what RSS is very poor at addressing today. I th

> a stream processing tool for end users This is it! This is the fundamental problem that we need to solve, and what RSS is very poor at addressing today. I think that being able to process massive content streams in a human-friendly way is one of those problems that would be world-changing if solved effectively. Facebook and Twitter have failed at this so far, with far-reaching negative consequences for societies. My gut feeling is that this probably means that we need a paradigm shift in how we relate to content streams. I think your move towards algorithmic transparency is a step in the right direction, though it may have to be bigger than that. I think the fundamental design flaw of modern content aggregation sites such as YouTube, Facebook, and Twitter, is that lack of transparency. They think that they know what's best for you. 1. You don't know why you're being recommended what you're seeing. 2. You can't teach them to avoid topics. You can only mark individual videos/posts as "not interested" and hope The Algorithm takes the hint. Likewise, you can't teach them to recommend more of a certain topic. 3. You can't see a feed as someone else, so everyone is in their own bubble by default. Reddit and Hacker News avoids this issue by showing everyone the same feed. 4. They tend to have poorly thought-out metrics for relevancy. For example on Facebook comments, users tend to use the Laugh and Angry reacts as negative signal. These comments also get a large number of negative comments. Facebook is completely blind to this, and shows them at the top since it treats all interactions as effectively equal. Cool, they've just built a system that boosts toxicity.

jacobobryant 6y ago
Yes, exactly! The nice thing about framing information discovery as a stream-processing problem is that you can plug other methods into it. For a long time I was thinking just in terms of user-item rating matrices, i.e. collaborative filtering. You've got a list of users, a list of items, and ratings for some of those pairs. Pick 10 items to show a given user. But that doesn't give the user a whole lot of control. If you think in terms of stream processing instead, you can treat collaborative filtering/other types of recommender systems as some black box that generates a stream. For example, Findka's ML recommendation algorithm picks a handful of articles for each user every day. It's just another stream, and it can be treated exactly the same way as any of the RSS feeds that users subscribe to. So you get a list of sources, some of which are "regular" RSS feeds, some of which are algorithmically generated--and they can come from within Findka or from outside--and then the user should be able to decide how those feeds are aggregated. Bandit algorithms I think are a great fit (including contextual bandits!), but you can also include manual controls. e.g. you could say "feed X should be 10% of my main sink feed", or "feed Y should be no more than 5% of my main sink feed." Or give priorities--sample from all the feeds in priority 1 until the contents are exhausted, then move to priority 2 etc. There are probably lots of interesting things that could be done in this space. Recommendation algorithms are often opaque out of necessity (I guess more generally, ML algorithms), but at least this way they wouldn't be in control of everything.

jacobobryant 6y ago

Yes, exactly! The nice thing about framing information discovery as a stream-processing problem is that you can plug other methods into it. For a long time I was thinking just in terms of user-item rating matrices, i.e. collaborative filtering. You've got a list of users, a list of items, and ratings for some of those pairs. Pick 10 items to show a given user. But that doesn't give the user a whole lot of control. If you think in terms of stream processing instead, you can treat collaborative filtering/other types of recommender systems as some black box that generates a stream. For example, Findka's ML recommendation algorithm picks a handful of articles for each user every day. It's just another stream, and it can be treated exactly the same way as any of the RSS feeds that users subscribe to. So you get a list of sources, some of which are "regular" RSS feeds, some of which are algorithmically generated--and they can come from within Findka or from outside--and then the user should be able to decide how those feeds are aggregated. Bandit algorithms I think are a great fit (including contextual bandits!), but you can also include manual controls. e.g. you could say "feed X should be 10% of my main sink feed", or "feed Y should be no more than 5% of my main sink feed." Or give priorities--sample from all the feeds in priority 1 until the contents are exhausted, then move to priority 2 etc. There are probably lots of interesting things that could be done in this space. Recommendation algorithms are often opaque out of necessity (I guess more generally, ML algorithms), but at least this way they wouldn't be in control of everything.