8 ms·
If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
by metronius 12y ago
If you would crawled web using Foreach(links) { get(link)} you would end up crawling useless automatically generated webpages.
- psykovsky 12y agoYou say that like it doesn't happen all the time. googlebot included.