4 ms·

If you don't want to see the same story twice, you need to implement your own filter. If it is titles only, keep index of last 1000 titles posted and apply some

If you don't want to see the same story twice, you need to implement your own filter. If it is titles only, keep index of last 1000 titles posted and apply something like tf-idf to it in order to find the most similar. You can peek in the articles bodies, but it requires proper scraping which is one more level of difficulty. So, if you find that two titles are quite similar (tune the value), just drop the one with less upvotes/comments. Additional things to look up are stop words and embeddings. Hope that helps

PaulHoule 5y ago
I helped build a search engine for patents that used a neural network embedding, I also wrote https://ontology2.com/essays/ClassifyingHackerNewsArticles/ https://ontology2.com/essays/ClassifyingHackerNewsArticles/ so I know what the data looks like. However that above article has never gotten as many votes as this article did https://ontology2.com/essays/HackerNewsForHackers/ https://ontology2.com/essays/HackerNewsForHackers/ which catches the emotions better than a solution does. I would retract some things from that article pair, for instance I put the anti-Apple filter in at the time and I did not need to, the site was just flooded with Apple articles at that moment in time because WWDC was a hit that year. A lot is know about ranking algorithms where "better content sort higher" but not ones where the absolute value of the ranking score is meaningful, which is important for filtering applications where you want to pick out "new and interesting" articles and set some threshold for what interesting is. As for consolidating topics that I think is better addressed through clustering than embeddings that were trained on somebody else's task.
- gostsamo 5y ago
  What you want is a personalized recommending system with lots of serendipity. Maybe you can try feeding it what you've liked, accept large amount of recommendations, and remove those that are too close to the input.

PaulHoule 5y ago

I helped build a search engine for patents that used a neural network embedding, I also wrote https://ontology2.com/essays/ClassifyingHackerNewsArticles/ https://ontology2.com/essays/ClassifyingHackerNewsArticles/ so I know what the data looks like. However that above article has never gotten as many votes as this article did https://ontology2.com/essays/HackerNewsForHackers/ https://ontology2.com/essays/HackerNewsForHackers/ which catches the emotions better than a solution does. I would retract some things from that article pair, for instance I put the anti-Apple filter in at the time and I did not need to, the site was just flooded with Apple articles at that moment in time because WWDC was a hit that year. A lot is know about ranking algorithms where "better content sort higher" but not ones where the absolute value of the ranking score is meaningful, which is important for filtering applications where you want to pick out "new and interesting" articles and set some threshold for what interesting is. As for consolidating topics that I think is better addressed through clustering than embeddings that were trained on somebody else's task.

gostsamo 5y ago
What you want is a personalized recommending system with lots of serendipity. Maybe you can try feeding it what you've liked, accept large amount of recommendations, and remove those that are too close to the input.