3 ms·
How do you crawl the web? Do you follow links around? How do you reach a page that isn't linked from anywhere you've crawled?
by pooper 3y ago
How do you crawl the web? Do you follow links around? How do you reach a page that isn't linked from anywhere you've crawled?
- gregw134 3y agoI'm just using common crawl for now
- mmkos 3y agoI mean that's what web crawling is, right? By extension, you just can't reach a page unless you stumble upon a link to it _somewhere_. Google gives you an option to submit a link and schedule a crawl that way, so that's another option if it's not being linked to from anywhere.