You can use the AOPIC algorithm that has a credit and taxation system to penalize these types of websites. https://stackoverflow.com/questions/5834808/designin

You can use the AOPIC algorithm that has a credit and taxation system to penalize these types of websites. https://stackoverflow.com/questions/5834808/designing-a-web-crawler https://stackoverflow.com/questions/5834808/designing-a-web-...