3 ms·
Great resource! I'm wondering why you put links to your site in the robots.txt example? It's like there's a clear utility you've made but then the links cheapen
by lancesells 3y ago
Great resource! I'm wondering why you put links to your site in the robots.txt example? It's like there's a clear utility you've made but then the links cheapen it a bit. Maybe I'm missing the reason?
- gavinhking 3y agoYou're not missing anything, they're just for any humans reading the file who want to know more about the agent. This is also just an example file.