3 ms·
> Does Mastodon have something to the effect of robots.txt Mastodon has a "Opt-out of search engine indexing: Affects your public profile and post pages" setti
by jefftk 4y ago
> Does Mastodon have something to the effect of robots.txt
Mastodon has a "Opt-out of search engine indexing: Affects your public profile and post pages" setting, which probably inserts a "robots" meta tag. [1]
But this is just for ordinary web crawling, and the bit isn't sent along with your posts that get federated to other instances.
[1] https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag https://developers.google.com/search/docs/crawling-indexing/...
- _as31 4y agoFurthermore, crawlers may not respect the opt-out tag. https://github.com/mastodon/mastodon/issues/13207 https://github.com/mastodon/mastodon/issues/13207 Another thing -- noindex is not federated. I used to enable noindex, and still found my content on Google (mirrored on other instances that I was federated with). I just allowed indexing.
- boyter 4y agoHence the crux of the issue. You can opt-out of indexing. The moment you federate, and your post leaves your server you no longer have that guarantee. All it takes is one instance to boost your post to someone who enables indexing.