3 ms·
Just because it's a robot doesn't mean that it takes more resources to load a page. A robot that loads 1000x more pages than a normal user, sure. But then rate-
by diggan 9y ago
Just because it's a robot doesn't mean that it takes more resources to load a page. A robot that loads 1000x more pages than a normal user, sure. But then rate-limit everyone rather than blocking specifically bots.
And that bots don't lead to revenue depends on why the bot is navigating on your page no? If it's some indexer that links back to your website and it's a popular index, then you'll maybe end up with more revenue thanks to that bot than a normal user.
- ikeboy 9y agoAccepting robots + humans takes more resources than only accepting humans. Your arguments about revenue are website-dependant and it's the website owner who is in the best position to decide whether robots are good for them or not (and plenty of sites don't ban bots in their robots.txt). In this case, the company that ran the bots is directly competing with Linkedin's products that sell aggregated data to employers and such, and linkedin clearly decided it's not going to lead to more revenue for them.