3 ms·
Why would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?
by aunch 4y ago
Why would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?
- latchkey 4y agoA day later, I see you added the robots.txt and it shows up first now.
- latchkey 4y agoYou're certainly getting a lot of free SEO help today. I'm not the one who has problems getting my site indexed, but it seems like they are different things. # Allowing all web crawlers access to all content User-agent: * Disallow: https://moz.com/learn/seo/robotstxt https://moz.com/learn/seo/robotstxt At least you'll be able to look in your logs and see if BingBot is grabbing the file or not.
- CharlesW 4y agoBecause this is the lowest-hanging fruit imaginable, why not take the 5 minutes to add a robots.txt and then use https://www.bing.com/webmasters/robotstxttester https://www.bing.com/webmasters/robotstxttester to confirm that Bing can fetch/parse it?