3 ms·
If they don’t want to be spidered maybe just use robots.txt They can’t have both: exposure AND content only available on their platform
by mccorrinall 4y ago
If they don’t want to be spidered maybe just use robots.txt
They can’t have both: exposure AND content only available on their platform
- chownie 4y agoCan you expand on that? Should Google have rights to use any of your content if you allow spidering? Including for commercial gain?
- mccorrinall 4y agoYes and yes. It’s okay to crawl a site and show the meta tags like short description, title, author etc. Why should it be illegal?