3 ms·
I've dabbled with using RSS feeds to inform the crawling of my search engine as well. I think the big issue with leaning too heavily into RSS without much huma
by marginalia_nu 4y ago
I've dabbled with using RSS feeds to inform the crawling of my search engine as well.
I think the big issue with leaning too heavily into RSS without much human intervention is that many CMS platforms generate an absurd number of RSS feeds, and it can be hard bordering on impossible to figure out which of them is canonical. Some (but not all) of these sites also tend to be fairly spammy low-value domains.
- derekzhouzhen 4y agoCMSs are not bad. Worse are the various forums. But the worst is nitter.net, which basically re-exposed the cesspool of twitter in RSS.
- zaik 4y agoNitter RSS feeds are the only way I follow people on Twitter. I'm glad they exist.
- derekzhouzhen 4y agoI understand that nitter serve a purpose. However, for OP who is trying to crawl high quality content such as blogs and news automatically, nitter is a nightmare.
- marginalia_nu 4y agoNitter is not that big of a deal, as they're just one website. Easy to block.