2 ms·
I'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and track
by marginalia_nu 4y ago
I'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch.
Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites.
[1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)