3 ms·
Do you know what proportion of the texty web instructs unknown crawlers to go away (or blocks them)?
by kews 5y ago
Do you know what proportion of the texty web instructs unknown crawlers to go away (or blocks them)?
- marginalia_nu 5y agoIt's hard to give numbers, it doesn't seem to be very many, but losing out on a few key sites does make a pretty big impact. You see stuff like this sometimes, makes me a bit sad. https://linux.die.net/robots.txt https://linux.die.net/robots.txt