2 ms·
Traditionally this is controlled by robots.txt, which does let you control which search engines can scrape your site. It's not enforceable, but generally consid
by skybrian 9y ago
Traditionally this is controlled by robots.txt, which does let you control which search engines can scrape your site. It's not enforceable, but generally considered fair play to respect it.
I think it's well within a website's rights to control who scrapes them. The problem with LinkedIn (and other social media sites) is that they don't let their users control this, in turn. It should be up to each user to decide whether they want to appear in search engines, and which ones.
It's pretty bizarre that this was never brought up in the article. The idea that users should control their own information was entirely ignored.