4 ms·
How do you keep it "indie"? Even fairly small sites probably link to Forbes or the Atlantic once in a while. Do you have a specific "block" list to keep large,
by stinkytaco 4y ago
How do you keep it "indie"? Even fairly small sites probably link to Forbes or the Atlantic once in a while. Do you have a specific "block" list to keep large, commercial sites out of the results?
- marginalia_nu 4y agoI'll crawl those sites too. Most of them will be weeded out in the processing stage, where I exclude websites that have too much heavy duty javascript and tracking and so on. I only index about 20% of the documents I fetch. Some still slip by, but the ranking algorithm takes care of the rest. I'm using personalized pagerank[1] biased toward a set of real human websites, which turns out to rather aggressively promote human websites. [1] http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf (see ch. 6)