4 ms·bulk filtering like that only cements FAANG hegemony; agree its a problem, do not agree this is the solutionby mistrial9 7y agobulk filtering like that only cements FAANG hegemony; agree its a problem, do not agree this is the solutiondTal 7y agoA whitelist of FAANG crawlers would cement their hegemony - a blacklist of known-badly-behaved crawlers doesn't.nkozyra 7y agoI don't filter out anyone but bad actors. If you abide by robots.txt you're free to scrape my sites