4 ms·
Glassdoor uses its for recruiting: http://www.glassdoor.com/robots.txt http://www.glassdoor.com/robots.txt
by d99kris 13y ago
Glassdoor uses its for recruiting:
http://www.glassdoor.com/robots.txt http://www.glassdoor.com/robots.txt
- te_chris 13y agoWhy would they disallow all their about content? (Bit of an SEO noob).
- JimmyM 13y agoOver-simply, because they do not want to indicate that their site is about those pages, for whatever reason. In practice, I'm not entirely sure but it looks like it's quite old since those pages don't seem to exist anymore, even as folders which don't exist as pages in themselves, and they aren't 301-redirected to the current relevant pages. In fact, they're all 404's. So perhaps they used to be pages, were deleted, and kept being crawled which made their site look crap (because of the 404's). Now they could use 301's, but I assume that the reason they didn't is because they might want to restructure the site in the future and re-use those pages. They don't use 302's because 302's are unreliable and freaky. Does that sound right to everyone else?
- mattmanser 13y agoThese guys are the best white-hat SEO growth hackers on the planet, just copy them and don't question it.
- JimmyM 13y agoOh, I wasn't questioning them - just asking if my assessment sounded right.
- coin 13y agoSadly they use Jobvite. Its usability is about as bad as Lotus Notes.
- avar 13y agoI love the note at the end of that file about how they can't figure out their own web framework.