3 ms·

Ask HN: What to do when Google ignores you?

2 points by Frozenlock 14y ago

Hello gentlemen, I've spent months building my tools and services. I'm actively trying to promote my website, but Google stopped crawling my site for 3 months now. I've checked all my commits around the time Google stopped crawling, but there's nothing there that should have an influence! I tested my website with proxies, web-sniffers, users from around the world: the website works for ALL of them. However if I try to 'fetch as Google' in the Google's webmaster tools, it always results in 'page unavailable'. I emailed Google, but all I received (even after multiple tries) was pre-made answers of this kind "Have you checked our forums?" YES, I did! I posted my question, and didn't get any answer, which is why I'm emailing you now! So yeah, I'm happy to see my website on DuckDuckGo, but I would really like to see it on Google; especially considering that my customers aren't necessary cutting edge... - I tested with multiple proxies and web-sniffer tools. - I checked my DNS - I checked if my ISP was blocking Googlebot IPs - I even checked with Wireshark and can see the Googlebot connection initiated!! I asked Google to get a more useful explanation than just 'page unavailable', but alas... So now I'm asking help from you. Here is my website: http://goo.gl/aSX2u Is there anything obvious that I'm missing? Why is Google ignoring me? Thank you very much in advance!

rcavezza 14y ago
Do you use Google webmaster tools? They should show potential reasons on there and which pages are currently indexed.
- Frozenlock 14y ago
  The reason shown in the crawl error is "Robots.txt fetch". This isn't very surprising as 'fetch as Google' returns a page unavailable for everything. (It's my understanding that a 404 on robots.txt is OK)
  - 27182818284 14y ago
    In the last few days I've seen errors with our robots.txt on sites that have NEVER had a problem with it and no errors. I even looked around on Twitter thinking it was a Google Bot bug. Then it just went away. Out of curiosity, why would a 404 on a robots.txt be OK? Shouldn't you want "Fetch as Google" to return the proper HTTP/1.1 200 OK?
    - Frozenlock 14y ago
      I don't have a robots.txt, so -for this url alone- I'm expecting a 404. What I've read in the google documents is that when a request to get the robots.txt returns a 404, it will consider it as OK to crawl.
AznHisoka 14y ago
Try changing from HTTPs to HTTP temporarily (a week or so) and see if that helps. Another solution: Come up with a new domain, do a redirect.
- Frozenlock 14y ago
  I did try without HTTPS, and even without GZIP. Not for a week, but enough to test the 'fetch as Google' option in the Google webmaster tools. I've also read that Googlebot shouldn't have any problems with HTTPS.
  - AznHisoka 14y ago
    Google claims they don't have problems all the time. Doesn't mean you should trust them. Give it a shot, what will it hurt?
    - Frozenlock 14y ago
      Tried again without HTTPS and this time it worked (What the f..). I hope this will be fixed soon, I would prefer to force HTTPS everywhere. Anyhow, thank you very much, at least now the bot will be able to crawl my site!
propercoil 14y ago
three problems off the bat: 1. you have only 10 backlinks from different root domains. 2. one of them is bacnetwiki.com (yours). This can signal a backlink pyramid. 3. your server is dropping requests when pinging from holland. Something is definitely wrong here
- Frozenlock 14y ago
  Thanks! The pinging is deactivated on my router, no surprise there. The problem is not the ranking, which backlinks influence, but the ability of the Googlebot to crawl the site. Also, I did not add the entry to BACnetWiki. In fact it's probably one of the only mention of the site that I did not made. :-)
orangethirty 14y ago
I'm going to point the Nuuton crawler towards your site. Shoot me an email to give you results.
- Frozenlock 14y ago
  Thank you! You can send me a mail at frozenlock@<the-domain-name>.com
neoterics 14y ago
Your site may be banned since it is not in the index, I would file a reconsideration request http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35843 http://support.google.com/webmasters/bin/answer.py?hl=en&#38...
- Frozenlock 14y ago
  Thanks. Any idea if this would also affect 'fetch as Google'?