4 ms·
No mention of robots.txt on the post, nor here in the comments. After a bit of poking around, it would appear Bing may need an allow block to crawl. I don't kn
by kordlessagain 4y ago
No mention of robots.txt on the post, nor here in the comments.
After a bit of poking around, it would appear Bing may need an allow block to crawl. I don't know what DDG does, but the author's site effectively has nothing in the robots.txt file, other than a commented out Disallow block. From doing this before in the past, I suggest include the following:
User-agent: *
Allow: /