4 ms·
Well, I created my own domain index. I have not crawled every page inside domains, but it is not my goal. I have 1542766 domains. Might not be much, but it is
by renegat0x0 1y ago
Well, I created my own domain index. I have not crawled every page inside domains, but it is not my goal.
I have 1542766 domains. Might not be much, but it is an honest work.
It is available as a github repo, so anybody that wants to start crawling has some initial data to kick off.
Links
https://github.com/rumca-js/Internet-Places-Database https://github.com/rumca-js/Internet-Places-Database
- hobs 1y agoCant you just request the ICANN’s zone files and have the canonical list of the day?
- egberts1 1y agoAvoiding GIGO (Garbage In, Garbage Out). This is why we have computer-variants of Library Science and Archeology, Forensic Science and a bunch of other advanced knowledge (not AI, mind you).
- hobs 1y agoI don't see how this applies as its aggregating a bunch of stuff from random crawlers - if you want to crawl a list of actual domains that's generally considered the list of things that could resolve, so seems like a good starting place.
- egberts1 1y agoSmashing stuff together by pure probablistic word association like AI do today is really, tsk tsk. That's why it is important to clearly define words like love, oh wait.
- renegat0x0 1y agoAny link list, or domain list is not worth much without any rating, or meta. I lead a hobby project, and I am not expert, so I provide ratings based on what kind of data pages provide (title, social, description), and my own manual voting system. It is not ideal, but it is something. Also I provide tags, so it is easily known what the domain provides, or domains can be filtered by tags. I know that you cannot count and visit every domain, so the list will never be finished, but I am happy with the results.
- hobs 1y agoWell, if you are curating every link them its a different story, and looks like a more classic webring - I missed that part of the work - I thought it looked like a big set of crawler data that wasn't as manually curated.
- beaugunderson 1y agoyou can, though you must provide a reason compelling enough to the person maintaining access (I provided a few sentences and was approved for most but maybe 20% of registrars declined my request): https://czds.icann.org/home https://czds.icann.org/home also be prepared for thousands of emails about status changes to your access.
- raybb 1y agoWhat a nice project. What inspired this initially? FYI there's a broken link in your readme: https://rumca-js.github.io/internet full internet search
- renegat0x0 1y agothanks, I replaced it with a other link demo
- didip 1y agoThis is amazing. Thanks for sharing!
- bufferoverflow 1y ago[dead]