4 ms·
@marginalia, Can you talk more about how you prioritize the URL frontier?
by billyhoffman 4y ago
@marginalia, Can you talk more about how you prioritize the URL frontier?
- marginalia_nu 4y agoI use a combination of incoming links and the average ranking of these linking sites to add new sites to the crawl queue. It's not super sophisticated, and I think it matters less the bigger the crawl is.
- stinkytaco 4y agoHow do you keep it "indie"? Even fairly small sites probably link to Forbes or the Atlantic once in a while. Do you have a specific "block" list to keep large, commercial sites out of the results?
- marginalia_nu 4y agoI'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch. Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites. [1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)