2 ms·
I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reli
by renegat0x0 1y ago
I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reliably has become increasingly difficult, as more and more content is protected or blocked. It’s especially frustrating when RSS feeds are inaccessible to bots.
404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers while exploring the web.
My scraping mechanism:
https://github.com/rumca-js/crawler-buddy https://github.com/rumca-js/crawler-buddy
Web crawler / RSS reader
https://github.com/rumca-js/Django-link-archive https://github.com/rumca-js/Django-link-archive
- vivzkestrel 1y agoyour requirements.txt lists feedparser but where are you actually using it? https://github.com/search?q=repo%3Arumca-js%2FDjango-link-archive%20feedparser&type=code https://github.com/search?q=repo%3Arumca-js%2FDjango-link-ar...
- renegat0x0 1y agoI use python poetry, i have not genereted requirements file for a long time. For current deps you can check pyproject file. I do not use feedparser, because it could not parse properly some rss files. I implemented my own lib for rss parsing.