3 ms·
He's not done his robots.txt properly, he's commented out the bit that actually disallows it # silly bing #User-agent: Amazonbot #Disallow: /
by 1317 3y ago
He's not done his robots.txt properly, he's commented out the bit that actually disallows it
# silly bing
#User-agent: Amazonbot
#Disallow: /
# buzz off
#User-agent: GPTBot
#Disallow: /
# Don't Allow everyone
User-agent: *
Disallow: /archive
# slow down, dudes
#Crawl-delay: 60
- haeffin 3y agoThe contents changed between then and now.
- deleted 3y ago[deleted]