4 ms·
You could look for special treatment in robots.txt. But that wouldn't show you what the server would say if you had a different user-agent.
by danuker 5y ago
You could look for special treatment in robots.txt. But that wouldn't show you what the server would say if you had a different user-agent.