I should have just linked to the official post: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googl

I should have just linked to the official post: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...