2 ms·
From what I've been told, the Google Bot absolutely tries to respect the robots.txt. The advice I got was to double check that my syntax and expressions were r
by freedomben 5y ago
From what I've been told, the Google Bot absolutely tries to respect the robots.txt. The advice I got was to double check that my syntax and expressions were really correct. A lot of people screw it up so it has to be basically perfect otherwise it won't be parsed correctly. Are you willing to share your robots.txt file?
- franze 5y agoGooglebot supports their set of robots.txt instructions https://developers.google.com/search/docs/advanced/robots/create-robots-txt https://developers.google.com/search/docs/advanced/robots/cr... Crawl Delay is not part of it.