I have myself dabbled a little bit in that subject. Some of my notes: - some RSS feeds are protected by cloudflare. It is true however that it is not necessary

I have myself dabbled a little bit in that subject. Some of my notes: - some RSS feeds are protected by cloudflare. It is true however that it is not necessary for majority of blogs. If you would like to do more then selenium would be a way to solve "cloudflare" protected links - sometimes even selenium headless is not enough and full blown browser in selenium is necessary to fool it's protection - sometimes even that is not enough - then I started to wonder, why some RSS feeds are so well protected by cloudflare, but who am I to judge? - sometimes it is beneficial to cover user agent. I feel bad for setting my user agent to chrome, but again, why RSS feeds are so well protected? - you cannot parse, read entire Internet, therefore you always need to think about compromises. For example I have narrowed area of my searches in one of my projects to domains only. Now I can find most of the common domains, and I sort them by their "importance" - RSS links do change. There need to be automated means to disable some feeds automatically to prevent checking inactive domains - I do not see any configurable timeout for reading a page, but I am not familiar with aiohttp. Some pages might waste your time - I hate that some RSS feeds are not configured properly. Some sites do not provide a valid meta "link" with "application/rss+xml". Some RSS feeds have naive titles like "Home", or no title at all. Such a waste of opportunity My RSS feed parser, link archiver, web crawler: https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive. Especially interesting could be file rsshistory/webtools.py. It is not advanced programming craft, but it got the job done. Additionally, in other project I have collected around 2378 of personal sites. I collect domains in https://github.com/rumca-js/Internet-Places-Database/tree/main/permanent https://github.com/rumca-js/Internet-Places-Database/tree/ma... . These files are JSONs. All personal sites have tag "personal". Most of the things are collected from: https://nownownow.com/ https://nownownow.com/ https://searchmysite.net/ https://searchmysite.net/ I wanted also to process domains from https://downloads.marginalia.nu/ https://downloads.marginalia.nu/, but haven't got time to read structure of the files