2 ms·
Ideally i would like these crawlers to access /robots.txt but nothing else. Only if they ignore robots.txt the access rules will stop them.
by Tepix 2y ago
Ideally i would like these crawlers to access /robots.txt but nothing else.
Only if they ignore robots.txt the access rules will stop them.
- jraph 2y agoYou can probably write a specific location block for robots.txt which will have a higher priority. See also https://stackoverflow.com/questions/5238377/nginx-location-priority https://stackoverflow.com/questions/5238377/nginx-location-p...