It's not binding, but it's a hint they may listen to where you can say things like if there's a certain URL structure where your server would keep serving conte

It's not binding, but it's a hint they may listen to where you can say things like if there's a certain URL structure where your server would keep serving content infinitely and just waste the crawlers time and resources, you can hint to them they maybe they should avoid certain URLs, but hey if they wanna waste their resources crawling that anyway be my guest!