4 ms·
Show HN: CrawlerCheck – A tool to check if a site is blocking crawlers
Hi HN,
I'm a long-time developer and SEO consultant. Over the years, I've seen clients suffer from a simple, costly mistake: accidentally blocking Googlebot or other important crawlers with a misplaced rule in robots.txt or a noindex tag.
Manually checking the robots.txt file, then the page's meta tags, then the X-Robots-Tag HTTP header is a tedious process. I wanted a tool that would do it all in one shot and give me a clear answer.
So, I built CrawlerCheck. You give it a URL, and it checks all three sources of crawler directives to tell you if a page is accessible.
The backend is written in Go, and the frontend is a lightweight Svelte app. The goal was to make it as fast and reliable as possible.
It's a brand new project, and I'd love to get some honest feedback from the HN community. Thanks for taking a look.
- 8organicbits 1y agoLooks pretty helpful, thanks for building this. Minor suggestion. Consider sorting the checks by status, or adding a summary at the top. I needed to scroll to find if anything was blocked. I don't know enough about the SEO space, but would a llms.txt check also help?
- bogozi 1y agoThanks so much for the great feedback! That's a fantastic point about adding a summary. I agree, it would make the results much easier to scan, especially because I'm planning to add even more rules. I'll definitely work on adding that. Good point on llms.txt too. I'm watching that standard evolve and plan to add it once it's more established. Appreciate you bringing it up!
- bogozi 1y agoHi. I have implemented the summary and indeed it looks and feels a lot quicker this way. I have also added scrolling functionalities so it's a bit easier to navigate as well. "but would a llms.txt check also help" - not sure about it as it is not standardized yet. Big players still need to adapt it or come up with a better solution. Will wait and see until then. I hope you'll find the updates useful. Thanks for the great feedback.
- bogozi 1y agoJust wanted to post a quick follow-up on this project for anyone who might see it. After the initial launch, I spent some time hardening the core logic and polishing the UI. I've just pushed a significant update with several improvements that make the tool much more reliable. Here’s what’s new: Smarter robots.txt parsing: It now correctly handles group precedence (so the most specific rule wins), wildcards (*), end-anchors ($), and rules with query strings (?) to better match how Googlebot actually interprets these files. Better error handling: I've polished the error messages on both the frontend and backend, so it's much clearer what's happening if a URL can't be fetched or a robots.txt file is malformed. Mobile & accessibility fixes: Pushed a few improvements for the experience on smaller screens and for better overall accessibility. Just wanted to log the progress here as I continue to work on the tool. Cheers.
- bogozi 1y agoIn the meantime I have implemented a publicly available changelog Here is what's new in v1.1.1 - Backend HTTP client now mimics real browsers more closely for improved compatibility with strict sites. - Automatic cookie handling and support for HTTP/2 enabled. - Referer header now transparently set to crawlercheck.com for all requests. - Improved diagnostics: HTTP status code and response body are logged for non-2xx responses to help troubleshoot blocks. - All previous features and compatibility with simpler sites are preserved.