4 ms·
I mostly agree. the traditional method [1] should be honored. but clearly, there are some bad agents out there that ignore the rules and mess things up for ever
by jfoutz 9y ago
I mostly agree. the traditional method [1] should be honored. but clearly, there are some bad agents out there that ignore the rules and mess things up for everyone. They are pretty clear about their expectations:
# Notice: The use of robots or other automated means to access LinkedIn without
# the express permission of LinkedIn is strictly prohibited.
[1] https://www.linkedin.com/robots.txt https://www.linkedin.com/robots.txt
- franze 9y agopuuuh, somebody should explain the concept of user-agent groups to them (would heavily simplify their robots.txt)