3 ms·
“Private links” should be covered by robots.txt. The only case I see this happening is for those “anyone with link” shares and those are easy to cover.
by Gigachad 4y ago
“Private links” should be covered by robots.txt. The only case I see this happening is for those “anyone with link” shares and those are easy to cover.
- rsbadger 4y agoAFAIK, most of those "private" links are actually just "unlisted" but they're still public. I'm sure Bing is indexing those too...
- Gigachad 4y agoAnything private should ideally be put behind authentication. If that isn't possible, than robots.txt. Search engines are _meant_ to index everything that is publicly accessible and not blacklisted on robots.txt.