5 ms·

https://host.io https://host.io - an API to get domain name metadata (scraped web content, backlinks, redirects, dns data, ranking information and more). See ht

https://host.io https://host.io - an API to get domain name metadata (scraped web content, backlinks, redirects, dns data, ranking information and more). See https://host.io/google.com https://host.io/google.com for an example. We've been building out the infrastructure for a while (scrape all domains monthly, resolve all domains, progress the data etc), but have only recently launched the API (see htttps://host.io/docs). We'll soon be releasing a Top 10M Ranked Domain list, like the Alexa top 1M, but 10M, and based on our own ranking signals, instead of traffic data like Alexa. If you've got any interesting use cases for our data or any feedback I'd love to hear it! ben@ipinfo.io

KajMagnus 7y ago
That's interesting, especially this, I think: > Shared IP Address > There are 68,258 domains hosted on 216.58.217.36 (AS15169 Google LLC) including google.com. Here's a sample: [...] You wrote: > scrape all domains monthly, resolve all domains what does "all domains" mean? All domains worldwide? Or some sub set? As mentioned, I think your project sounds interesting — still I'm wondering, how did you verify there's an interest for what you're building? (I'd think/guess there is, just wondering if/how you verified this, or knew beforehand)
- coderholic 7y ago
  All domains is literally all domains :)
  - sepbot 7y ago
    How does one get a list of all domains?
    - nullpage 7y ago
      Rapid7's project sonar forward lookup database is a good start. https://opendata.rapid7.com/ https://opendata.rapid7.com/
KajMagnus 7y ago
> If you've got any interesting use cases What are some cases you know about this far?
- coderholic 7y ago
  Cybersecurity research / investigation (example here: https://twitter.com/host/status/1215709472066170880 https://twitter.com/host/status/1215709472066170880) Intellectual property enforcement Malicious traffic / account creation investigation Domain name or competitor research Market analysis (eg. find out how many customers publicly traded hosting providers have)
atmosx 7y ago
Some corps collect DNS logs from the infrastructure for domains that have been registered very recently. UPDATE: The free tier is static at 1k and then you have to buy credits or it is 1k / month?
- coderholic 7y ago
  Currently the API is priced at $0.01 per call, with $10 of free credit when you signup. We'll likely add monthly subscriptions in the future perhaps with a recurring monthly free tier, but we don't have them yet. If you do run out of the free $10 credit and want more time to see if the API is going to be useful for you shoot me an email and I'll be happy to issue more credit.
  - atmosx 7y ago
    Thanks
135792468 7y ago
Where are you getting the link data from? Common crawl? 3rd party? Same question for ranking info, assume you're not scraping yourself and using some like D4SEO?
- coderholic 7y ago
  We're doing the scraping ourselves. Just a single page per domain currently (so it's just links on the homepage), but we'll expand to a few pages per domain in the future. We're also doing the ranking ourselves. We bootstrapped it by doing a regression against Alexa and other top 1m lists using a bunch of our own features (like number of backlinks, some content features, some IP address related features etc). We'll be sharing more details around this soon.

KajMagnus 7y ago

That's interesting, especially this, I think: > Shared IP Address > There are 68,258 domains hosted on 216.58.217.36 (AS15169 Google LLC) including google.com. Here's a sample: [...] You wrote: > scrape all domains monthly, resolve all domains what does "all domains" mean? All domains worldwide? Or some sub set? As mentioned, I think your project sounds interesting — still I'm wondering, how did you verify there's an interest for what you're building? (I'd think/guess there is, just wondering if/how you verified this, or knew beforehand)

coderholic 7y ago
All domains is literally all domains :)
- sepbot 7y ago
  How does one get a list of all domains?
  - nullpage 7y ago
    Rapid7's project sonar forward lookup database is a good start. https://opendata.rapid7.com/ https://opendata.rapid7.com/

KajMagnus 7y ago

> If you've got any interesting use cases What are some cases you know about this far?

coderholic 7y ago
Cybersecurity research / investigation (example here: https://twitter.com/host/status/1215709472066170880 https://twitter.com/host/status/1215709472066170880) Intellectual property enforcement Malicious traffic / account creation investigation Domain name or competitor research Market analysis (eg. find out how many customers publicly traded hosting providers have)

atmosx 7y ago

Some corps collect DNS logs from the infrastructure for domains that have been registered very recently. UPDATE: The free tier is static at 1k and then you have to buy credits or it is 1k / month?

coderholic 7y ago
Currently the API is priced at $0.01 per call, with $10 of free credit when you signup. We'll likely add monthly subscriptions in the future perhaps with a recurring monthly free tier, but we don't have them yet. If you do run out of the free $10 credit and want more time to see if the API is going to be useful for you shoot me an email and I'll be happy to issue more credit.
- atmosx 7y ago
  Thanks

135792468 7y ago

Where are you getting the link data from? Common crawl? 3rd party? Same question for ranking info, assume you're not scraping yourself and using some like D4SEO?

coderholic 7y ago
We're doing the scraping ourselves. Just a single page per domain currently (so it's just links on the homepage), but we'll expand to a few pages per domain in the future. We're also doing the ranking ourselves. We bootstrapped it by doing a regression against Alexa and other top 1m lists using a bunch of our own features (like number of backlinks, some content features, some IP address related features etc). We'll be sharing more details around this soon.