3 ms·
Here's the robots.txt file for hackerfactor.com: User-agent: ia_archiver Allow: / User-agent: ScoutJet Disallow: /blog User-Agent: Googlebot Allow:
by kordlessagain 8y ago
Here's the robots.txt file for hackerfactor.com:
User-agent: ia_archiver
Allow: /
User-agent: ScoutJet
Disallow: /blog
User-Agent: Googlebot
Allow: /
Disallow: /blog/index.php?/archives/2018/
Disallow: /blog/index.php?/archives/2017/
Disallow: /blog/index.php?/archives/2016/
Disallow: /blog/index.php?/archives/2015/
Disallow: /blog/index.php?/archives/2014/
Disallow: /blog/index.php?/archives/2013/
Disallow: /blog/index.php?/archives/2012/
Disallow: /blog/index.php?/archives/2011/
Disallow: /blog/index.php?/archives/2010/
Disallow: /blog/index.php?/archives/2009/
Disallow: /blog/index.php?/archives/2008/
Disallow: /blog/index.php?/archives/2007/
Disallow: /blog/index.php?/archives/2006/
Disallow: /blog/index.php?/archives/2005/
Disallow: /blog/index.php?/archives/2004/
Disallow: /blog/index.php?/archives/2003/
Disallow: /blog/index.php?/archives/2002/
Disallow: /blog/index.php?/archives/2001/
Disallow: /blog/index.php?/archives/2000/
#Disallow: /blog/index.php?/categories/
User-agent: *
Disallow: /blog
Disallow: /badbot-a2d0ac98abcaf3cafd9eff83b3cffa98fec7a390a6c5b9
I'm not 100% certain about that ? in there, but I think you can do this instead:
Disallow: /blog/index.php?/archives/*/