4 ms·

More bots, more EC2s, more money More bots, more noise, bad analytics More bots, IP theft at scale, competitors now have my content I don't understand how th

More bots, more EC2s, more money More bots, more noise, bad analytics More bots, IP theft at scale, competitors now have my content I don't understand how this is hard to wrap one's head around. This is why we can't have nice things. Edit - I offer 3 points to refute parent and I'm getting downvoted. This site is moderated by a bunch of bots!

neurobro 7y ago
With a few exceptions like HD video hosting or sites doing a ton of computation for every request, the best response to point 1 is to just lighten the site, which reduces the load from bots and also improves the UX for humans. Rate limiting can also help in some circumstances. Or maybe consider... charging money. If you don't force botters to take extreme camouflage measures, bots are easily filtered out of logs (and offer a potentially useful metric of their own). If a business is threatened by "competitors" simply scraping published information, it's probably already doomed from the start. And I would posit that most site owners with this mindset originally got their data by scraping other sites, which is why they feel vulnerable. Compete over elements that are actually valuable!