4 ms·

Hey, we're still kind of new and kicking the tires. Any UX feedback or data feedback is appreciated!

• regecks 9y ago
  It's pretty much missing or empty data for the entire .au TLD (e.g. even google.com.au is not present). Living here, I know that their whois server rate limits heavily and the ccTLD zonefile is not available, so I'm guessing those are probably contributing reasons. Do you guys do any crawling at all? I think this is a fantastic concept though. Knowing that whois database and zonefile access is often protected for commercial motivations, it really irks me. Open it all up.
  • mxpxrocks10 9y ago
    right now we're getting .au domains from Open Crawl. It's tricky because the zone file is not available like you mentioned. If anyone has any ideas on how to get more .au domains, we'll gladly implement.
    • jlgaddis 9y ago
      CT logs come to mind, if you aren't using them already. DNS queries, if you have access to recursive resolvers.
      • mxpxrocks10 9y ago
        we're using CT logs for getting hostnames (you can see it at the top of the page of any domain). We have a great recursive resolver source but haven't been able to integrate it into the data pipeline yet. thanks for the note and ideas!
    • hughesey 9y ago
      There's some reverse engineered zone files for a lot of cctlds at http://viewdns.info/data/ http://viewdns.info/data/.