3 ms·
No one ever talks about a particular topic though when it comes to web crawling etc. How do you avoid all the "bad" sites as in, really bad shit? The stuff that
by supername 11y ago
No one ever talks about a particular topic though when it comes to web crawling etc. How do you avoid all the "bad" sites as in, really bad shit? The stuff that your ISP could use as evidence against you when in fact it was just your code running and it happened to come across one of these sorts of sites. How do you deal with all that? That is the only thing stopping me from experimenting around with web crawling.
- johnward 11y agoI used to do some crawling on comcast using what is now IBM Watson Explorer. I got a ton of phone calls from them. It sounded like those calls would go away if I just paid a little bit more for business class service.