3 ms·
Show HN : I have developed a robots.txt full text search engine.
As a sub project of my pet project DNSDigger.com i have developed a search engine that index robots.txt (http://en.wikipedia.org/wiki/Robots_exclusion_standard).
I actually don't know what poeple could use this for but i suspect poeple that are into SEO could use it and black/white-hat hackers(?).
What i need help with is VPS/$$ for spidering. And as i sit on a considereble index of domainnames (though DNSDigger.com 300m hosts) i thought that
if you want a TLD indexed sooner than later you could sponsor it with either VPS or Paypal. See it as a strange "kickstarter".
And if you think the "design" suck (which it does because there is a lot of things i suck at and design is one of them) you are free to submit anything better. I can put up a banner for your service a limited time.
I am adding new hosts/TLD all the time.
And if you have to ask what this is you will not have any use for it even if you did ;)
- Xeroday 13y agoVery cool project. I'm curious as to how you're getting all these robots - are you scraping them yourself?
- Joyfield 13y agoDownloading them as we speak. I have a big list of hosts/domains i have collected through spidering for my DNSDigger.com. This is a hobby project that has grown a bit over my head hehe. And there is no scraping needed. Robots.txt is just simple textfiles. Download and parse, repeat a couple of million times and build an index :)