3 ms·
For scraping entire sites browserteix is currently more suited until we add full depth recursive crawling in v0.9. For feeding to LLMs ArchiveBox MIGHT BE bette
by nikisweeting 2y ago
For scraping entire sites browserteix is currently more suited until we add full depth recursive crawling in v0.9. For feeding to LLMs ArchiveBox MIGHT BE better (imho) because we extract the raw content and you likely don't need the whole WARC.