4 ms·
What’s recursive scraping (presumably, following internal links), and what’s broken about it?
by bcherny 5y ago
What’s recursive scraping (presumably, following internal links), and what’s broken about it?
- chaps 5y agoYep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.