4 ms·

> After a "bad" page percentage threshold is exceeded just stop wasting your time crawling that domain entirely. In the words of Bush jr.: Mission accomplished

by krior 2y ago

> After a "bad" page percentage threshold is exceeded just stop wasting your time crawling that domain entirely. In the words of Bush jr.: Mission accomplished!