4 ms·
There is probably some highly non-obvious reason that sorting your queue of URLs by length is optimal, which was arrived at after a lot of modelling and testing
by JonnieCache 15y ago
There is probably some highly non-obvious reason that sorting your queue of URLs by length is optimal, which was arrived at after a lot of modelling and testing.
We're unlikely to ever know the answer unless someone from google explains it to us.
- JonnieCache 15y agoThinking about it more, its probably just a breadth-first search. Duh.
- frisco 15y agoBreadth-first search means crawling all of the links on the page and adding all of the links on the child pages to the queue at once rather than drilling down on one link-path first before moving to the other links on the first page. There's probably some highly non-obvious reason for crawling by url length.
- xyzzyz 15y agoThe thing is, the deeper you are, the longer are the urls, so if you do a breadth first search, you are more likely to visit short urls first.