3 ms·

Triv.io donates URL index to Common Crawl

51 points by LisaG 14y ago

soult 14y ago
Is it just me or is the data file not available for free despite being in the Amazon Public Dataset S3 bucket? *Edit: The problem seems to be fixed now.
- srobertson 14y ago
  Let me double check that for you. Were you using a valid aws-id and secret?
  - soult 14y ago
    No, usually you can download them without sending any aws-id if they are on the Public Datasets S3 bucket, e.g. wget https://s3.amazonaws.com/aws-publicdatasets/common-crawl/crawl-001/2008/06/19/0/1213886083018_0.arc.gz
brianr 14y ago
Nice work triv.io!
rb2k_ 14y ago
What I'd love to see: A simple list of domains No information about content, no full URLs, just the domainname.
- deleted 14y ago
  [deleted]
- soult 14y ago
  Extracing such a list from the generated index only takes a small script and a few hours to download the 200+ GB index file. Which is a lot less than the slightly bigger script and the months/years to download and process 80+ TB of arc files that it previously would have taken to extract all domains. Anyways, if you want a copy of domains from the index file just send me a mail to the address in my profile.
  - thefreeman 14y ago
    Where is the index file located in the s3 bucket?
    - srobertson 14y ago
      s3://aws-publicdatasets/common-crawl/projects/url-index/url-index.1356128792
      - thefreeman 14y ago
        thank you kindly!
  - deleted 14y ago
    [deleted]
- srobertson 14y ago
  That would be great. Out of curiosity, mind describing what you'd use such a list for?
  - rb2k_ 14y ago
    Usually as a starting point for a crawler when checking statistics about the web. Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not. With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)
- adventured 14y ago
  You can get the top million sites from Quantcast in a download for free. Alexa used to also offer something similar, but I don't see it on their site any longer (they try to sell their data through aws, so that's probably why).
  - gourneau 14y ago
    Can someone dig that up and link it please
    - adventured 14y ago
      http://ak.quantcast.com/quantcast-top-million.zip http://ak.quantcast.com/quantcast-top-million.zip
      - deleted 14y ago
        [deleted]
  - ghewgill 14y ago
    You can find the Alexa top 1 million domain list at http://www.alexa.com/topsites http://www.alexa.com/topsites (it's on the right hand side).
- aseidl 14y ago
  A number of the sponsoring orgs will grant you access to their zonefiles if you're willing to do a little paperwork (HelloFax is awesome) and have a valid reason for access. Granted it's not the complete list of registered domains (those without nameservers on file won't show up), but it's pretty close. The list of sponsoring orgs for each TLD is at: http://www.iana.org/domains/root/db http://www.iana.org/domains/root/db

51 points by LisaG 14y ago

soult 14y ago

Is it just me or is the data file not available for free despite being in the Amazon Public Dataset S3 bucket? *Edit: The problem seems to be fixed now.

srobertson 14y ago
Let me double check that for you. Were you using a valid aws-id and secret?
- soult 14y ago
  No, usually you can download them without sending any aws-id if they are on the Public Datasets S3 bucket, e.g. wget https://s3.amazonaws.com/aws-publicdatasets/common-crawl/crawl-001/2008/06/19/0/1213886083018_0.arc.gz

brianr 14y ago

Nice work triv.io!

rb2k_ 14y ago

What I'd love to see: A simple list of domains No information about content, no full URLs, just the domainname.

deleted 14y ago
[deleted]
soult 14y ago
Extracing such a list from the generated index only takes a small script and a few hours to download the 200+ GB index file. Which is a lot less than the slightly bigger script and the months/years to download and process 80+ TB of arc files that it previously would have taken to extract all domains. Anyways, if you want a copy of domains from the index file just send me a mail to the address in my profile.
- thefreeman 14y ago
  Where is the index file located in the s3 bucket?
  - srobertson 14y ago
    s3://aws-publicdatasets/common-crawl/projects/url-index/url-index.1356128792
    - thefreeman 14y ago
      thank you kindly!
- deleted 14y ago
  [deleted]
srobertson 14y ago
That would be great. Out of curiosity, mind describing what you'd use such a list for?
- rb2k_ 14y ago
  Usually as a starting point for a crawler when checking statistics about the web. Crawling is much faster if you don't have to "spider" all the links and check if you've already visited them or not. With a big enough list, you can just iterate over those domains. (average number of links on a website, how often does javascript framework x vs y get used, how many sites have an HTML5 doctype yet, ...)
adventured 14y ago
You can get the top million sites from Quantcast in a download for free. Alexa used to also offer something similar, but I don't see it on their site any longer (they try to sell their data through aws, so that's probably why).
- gourneau 14y ago
  Can someone dig that up and link it please
  - adventured 14y ago
    http://ak.quantcast.com/quantcast-top-million.zip http://ak.quantcast.com/quantcast-top-million.zip
    - deleted 14y ago
      [deleted]
- ghewgill 14y ago
  You can find the Alexa top 1 million domain list at http://www.alexa.com/topsites http://www.alexa.com/topsites (it's on the right hand side).
aseidl 14y ago
A number of the sponsoring orgs will grant you access to their zonefiles if you're willing to do a little paperwork (HelloFax is awesome) and have a valid reason for access. Granted it's not the complete list of registered domains (those without nameservers on file won't show up), but it's pretty close. The list of sponsoring orgs for each TLD is at: http://www.iana.org/domains/root/db http://www.iana.org/domains/root/db