I thought robots.txt was meant to be pulled from the domain and honoured anyway. At least that’s what used to happen. Just because someone links to you doesn’t

I thought robots.txt was meant to be pulled from the domain and honoured anyway. At least that’s what used to happen. Just because someone links to you doesn’t mean the spiders should crawl all the content