3 ms·
Start with a robots.txt? https://www.codeium.com/robots.txt https://www.codeium.com/robots.txt Update: I also see that you serve your site off GitHub... so th
by latchkey 4y ago
Start with a robots.txt?
https://www.codeium.com/robots.txt https://www.codeium.com/robots.txt
Update: I also see that you serve your site off GitHub... so that might be part of it as well? Bing might not index those sites?
- aunch 4y agoWhy would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?
- latchkey 4y agoA day later, I see you added the robots.txt and it shows up first now.
- latchkey 4y agoYou're certainly getting a lot of free SEO help today. I'm not the one who has problems getting my site indexed, but it seems like they are different things. # Allowing all web crawlers access to all content User-agent: * Disallow: https://moz.com/learn/seo/robotstxt https://moz.com/learn/seo/robotstxt At least you'll be able to look in your logs and see if BingBot is grabbing the file or not.
- CharlesW 4y agoBecause this is the lowest-hanging fruit imaginable, why not take the 5 minutes to add a robots.txt and then use https://www.bing.com/webmasters/robotstxttester https://www.bing.com/webmasters/robotstxttester to confirm that Bing can fetch/parse it?
- joshmanders 4y agoNo robots.txt is basically the same as the default contents of a robots.txt saying "All bots welcome"
- vertis 4y agoNo robots.txt is unpredictable because it depends on the implementation of the bot.
- benhoyt 4y agoA missing robots.txt has always meant "allow all", so it would be really weird for this to be the cause. Robots.txt is primarily for disallowing (or really, discouraging) crawlers from crawling certain paths.