3 ms·
This is a cool idea. It would be interesting to add a spider onto it as well. Check outlinks from known articles and see if they have feeds, then add that to th
by kevincox 4y ago
This is a cool idea. It would be interesting to add a spider onto it as well. Check outlinks from known articles and see if they have feeds, then add that to the index.
It also seems like feed archiving/pagination is not supported. This means that new feeds won't have much history searchable. It may be a good idea to support archiving and pagination to index old articles. (Although admittedly support for either of these protocols is rare).
- marginalia_nu 4y agoFinding "the RSS feed" of a website can be an annoyingly non-trivial problem, since many CMS:es and forum software produces an absolute shit-ton of RSS feeds dynamically.
- kevincox 4y agoI'm sure you can't get 100% of sites that have feeds but support for <link rel=alternate> is very widespread. If you want to be more aggressive you can try fetching links that sound like RSS feeds from link text or URL but in practice I have found that only gets you a small amount more.
- marginalia_nu 4y agoNo I mean you'll commonly encounter several RSS feeds on the same page, all rel=alternate. There doesn't seem to be any convention for indicating what the feed is about. You may get an RSS feed for the site, one for each the topic, and one for each comments section of the blog post, etc. These may then be duplicated in various ways since there's more than one path to the same page. It's a real pain in the ass.
- daviducolo 4y agoall list are paginated and you can search over the entire dataset.
- kevincox 4y agoI'm talking about the ingested feeds, not the search results.
- daviducolo 4y agoah ok no, API extracts feeds from March 8, 2023