3 ms·
I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probab
by ScottWhigham 13y ago
I actually did this exact robots.txt for my site this summer. The net effect? A massive loss of secondary market traffic - enough that, after a month, we probably lost 10-15% in revenue. It surprised us. We rolled the old robots.txt back into place and bam - life went back to normal. You can question my pithy 300-character explanation all you want but I'll leave you with this: we were 100% certain that the robots.txt change was the difference.
In the end, we switched to network blocks for the common bots/spiders. Much better.
- breadtk 13y agoDo you mind posting your old robots.txt?
- ScottWhigham 13y agoIt was almost identical to his
- csharp_gooru 13y agoThanks Scott. Please advice how to create network blocks. I run a affiliate site and need to do this for my site too!
- ScottWhigham 13y agoIt depends on your setup. We have a hardware firewall so I add/remove networks daily to it. If you are on apache, you can use htaccess. On a Windows server, you can add them to Windows Firewall.
- csharp_gooru 13y agoCool. Thank you for your response. My site runs on a Windows server. I'll look into your recommendation. Thanks again.