3 ms·
>except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access how does this work technically? i just tried
by devmunchies 7y ago
>except that certain entities, like the Google search engine, have express permission from LinkedIn for bot access
how does this work technically? i just tried crawling a friend's profile using curl and set my user agent to Google's bot and it still was blocked.
- phs318u 7y agoIP range whitelisting? MASSL? EDITED: corrected auto-correct.
- dx034 7y agoGoogle describes how to verify Googlebot here: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en Most other search engine crawlers provide similar methods.