3 ms·
> there is no need to spider/crawl if there are sitemap XML files listing all the content URLs you never know if these are all URLs tho, so I would always craw
by thr0wnawaytod4y 4y ago
> there is no need to spider/crawl if there are sitemap XML files listing all the content URLs
you never know if these are all URLs tho, so I would always crawl and brute force more paths than trusting sitemap.xml
- 1vuio0pswjnm7 4y agoIs that what Bing does. Doubtful.