2 ms·
The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features https://github.com/adbar/trafilatura#features Basically: you
by simonw 3y ago
The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features https://github.com/adbar/trafilatura#features
Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.