3 ms·
Do you feel bad at all about apparently making a business out of crawlers, but still apparently viewing it as bad enough that you want countermeasures against i
by 59nadir 9y ago
Do you feel bad at all about apparently making a business out of crawlers, but still apparently viewing it as bad enough that you want countermeasures against it? Don't you feel a slight bit hypocritical about this?
- wrath 9y agoI don't think I'm being hypocritical. I have no issues if people crawl my site, I even identify who they are and give them access to a private API. I do not generate any income from that website though. I provide a service because I love doing it and I cover all the costs. Bots do increase my cost so I choose to limit their activity. Crawl me, but do so using my rules. One of the alternatives is charging for my service but bots are not my users problem, they are mine.