robots.txt is for web crawlers that follow links, it doesn't usually apply to robots that fetch a single page based on a user request. But they could invent and

robots.txt is for web crawlers that follow links, it doesn't usually apply to robots that fetch a single page based on a user request. But they could invent and adopt a new tag.