3 ms·
For devs like us, archivebox? or browsertrix-crawler? for scraping entire sites for our own uses, maybe to keep contents behind pay walls while we have subscrip
by millvalleydev 2y ago
For devs like us, archivebox? or browsertrix-crawler? for scraping entire sites for our own uses, maybe to keep contents behind pay walls while we have subscriptions or maybe to feed them to local LLMs to ask?
- nikisweeting 2y agoFor scraping entire sites browserteix is currently more suited until we add full depth recursive crawling in v0.9. For feeding to LLMs ArchiveBox MIGHT BE better (imho) because we extract the raw content and you likely don't need the whole WARC.