10 ms·
That's maybe jumping to conclusion. I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones" So if you put a wi
by gutnor 11y ago
That's maybe jumping to conclusion.
I interpreted the sentence less literally and more like "in absence of rule, default to GoogleBot ones"
So if you put a wildcard rule forbidding access and a specific one allowing access to GoogleBot, AppleBot will honour the wildcard one.
That's how I would have coded it anyway: parse the rule for current agent string, if no rule applies, run it again with GoogleBot one before assuming that website does not contain restrictions.
- koyote 11y ago"If robots instructions don't mention Applebot" I would assume this means that it will follow GoogleBot unless you specifically mention AppleBot by name and not by using a wildcard. So a User-agent: * would be ignored if a User-agent: GoogleBot is found.