5 ms·

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as

by Smerity 12y ago

[lightly modified version of a comment I put on the article as I love HN for discussion!] Great article -- we're excited there's so much interest in the web as a dataset! I'm part of the team at Common Crawl and thought I'd clarify some points in the article. The most important is that you can download all the data that Common Crawl provides completely for free, without the need to pay S3 transfer fees or process it only in an EC2 cluster. You don't even need to have an Amazon account! Our crawl archive blog posts give full details for downloading[1]. The main challenge then is storing it, as the full dataset is really quite large, but a number of universities have pulled down a significant portion onto their local clusters. Also, we're performing the crawl once a month now. The monthly crawl archives are between 35-70 terabytes compressed. As such, we've actually crawled and stored over a quarter petabyte compressed, or 1.3 petabytes uncompressed, so far in 2014. (The archives go back to 2008.) Comparing directly against the Internet Archive datasets is a bit like comparing apples to oranges. They store images and other types of binary content as well, whilst Common Crawl aims primarily for HTML, which compresses better. Also, the numbers used for Internet Archive were for all of the crawls they've done, and in our case the numbers were for a single month's crawl. We're excited to see Martin use one of our crawl archives in his work -- seeing these experiments come to life the best part of working at Common Crawl! I can confirm that optimizations will help you lower that EC2 figure. We can process a fairly intensive MR job over a standard crawl archive in afternoon for about $30. Big data on a small budget is a top priority for us! [1]: http://blog.commoncrawl.org/2014/11/october-2014-crawl-archive-available/ http://blog.commoncrawl.org/2014/11/october-2014-crawl-archi...

marktangotango 12y ago
Before Google's page rank algorithm there was a lot of research into document search. A favorite of mine was 'scatter gather'[1]. I often wanted ability to filter and group when staring a pages of results from redhat jboss support forums when trying to fix a dead jboss cluster. But I quit that job so haven't had the need recently. Edit; point being it would be nice if someone came up with a service that implemented the ideas in this paper :) [1] http://www-users.cs.umn.edu/~han/dmclass/scatter.pdf http://www-users.cs.umn.edu/~han/dmclass/scatter.pdf
curiously 12y ago
what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher? I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating an internet as a dataset, or an API. I can't help but feel like it's just solving another 'semantic web' problem that nobody asked for. It is funny that the most demanding customers of semantic web are also the ones who are willing to pay the least amount of time and money.
- pasteurquadrant 12y ago
  +1
- jobposter1234 12y ago
  Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO. Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG). You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment. Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.
jtoy 12y ago
anyway we can get an open sourced version of that optimized job for us to play with?
- Smerity 12y ago
  There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data! AWS optimizations: For that level of cost efficiency, you really need to use spot instances. A cluster of 100 m1.xlarge machines (1.5TB of RAM, 400 cores, and 168TB of magnetic disk storage) will only cost you $3 per hour using spot instances, rather than $30 on-demand. You should pay on-demand prices for the Hadoop master however. You'll also want to roll your own Hadoop cluster as opposed to using Elastic MapReduce (EMR). EMR is amazing but the cost overhead when using it on spot instances is ~100%. For the code itself, this is a situation where you'll want to stick with the programming language that has both the best performance and best ecosystem. I'm personally not a big fan of Java, but it really does win out here -- it's close to C or C++ for performance and has the advantage of the Hadoop ecosystem behind it. Other languages are certainly usable, but even if LanguageX only ran 4x slower than Java, the resulting job would be 4x more expensive due to paying by the hour. Other than that, it's really just a standard MapReduce job using Hadoop. You can see an three examples for the three different data formats we use at: https://github.com/commoncrawl/cc-warc-examples/ https://github.com/commoncrawl/cc-warc-examples/
  - elyase 12y ago
    Thanks for your comments. I see you get 400 cores for 3USD/h . Where you optimizing for processing power per dollar? Do you know if the m1.xlarge instances are the best for this use case?
    - Smerity 12y ago
      It really depends on the use case. I mentioned m1.xlarge as they provide a good general cluster setup. The mix of CPU, RAM, and disk space should work well for most experiments one might want to perform. m1.xlarge instances also have 1Gbps network interfaces when others in the same price range have 500Mbps -- a vestige of the older generation of machines. Finally, they excel at disk space. If you're utilizing HDFS heavily, newer instances are usually SSD (good) but have 10 to 20 times less disk storage (bad). For the same dollar amount, you can trade for other specs though. If you're more interested in CPU / RAM / SSD for example, paying $3USD/h for r3.xlarge gets you 3TB of RAM, about 1.5 times more computing power (same number of cores but more compute units), but far less disk space -- 8TB of SSD. In the end, it really depends on the task at hand, but your dollar does go quite far regardless!
stevesalevan 12y ago
So awesome to see that folks are discovering Common Crawl, it's a really great project and the amount of data they've crawled has been tremendous. If you're looking to get into machine learning, it's perhaps the most comprehensive dataset you can get your hands on.
adwf 12y ago
Is there any chance we might see an updated index for the Common Crawl? I've tried using the dataset before, but I found it difficult given that you have to process the entire thing in order to find the particular pages you are looking for. As an example, I was trying a project to look at the top news sites, like BBC, CNN, Al Jazeera, etc. Then matching articles about the same news topic on each site, before finally fact checking for differences between the stories (ie. 20,000 homes were without power vs. 50,000) That kind of project requires a load of crawling, but I can't look for specific pages without processing the entire CC set first. I love the project though, so thank you for doing it!

by Smerity 12y ago

marktangotango 12y ago

Before Google's page rank algorithm there was a lot of research into document search. A favorite of mine was 'scatter gather'[1]. I often wanted ability to filter and group when staring a pages of results from redhat jboss support forums when trying to fix a dead jboss cluster. But I quit that job so haven't had the need recently. Edit; point being it would be nice if someone came up with a service that implemented the ideas in this paper :) [1] http://www-users.cs.umn.edu/~han/dmclass/scatter.pdf http://www-users.cs.umn.edu/~han/dmclass/scatter.pdf

curiously 12y ago

what would be the benefit of having the web as a dataset when it is rife with copyright laws (see craigslist), monopoly businesses who viciously protect their human uploaded content and profiles, and majority of the data from the web being useless without a context and purpose of the searcher? I'm just curious as to how commoncrawl compares with kimonolabs and import.io as they seem to have the same goal of creating an internet as a dataset, or an API. I can't help but feel like it's just solving another 'semantic web' problem that nobody asked for. It is funny that the most demanding customers of semantic web are also the ones who are willing to pay the least amount of time and money.

pasteurquadrant 12y ago
+1
jobposter1234 12y ago
Regarding copyright issues, I believe you can still use copyrighted data as long as it's transformed. E.g., building language models, or doing a search engine like Google. In fact, I can think of more computational uses for copyrighted data, while on the "banned" side, I can only think of... SEO. Regarding point two: "monopoly businesses who viciously protect their human uploaded content". I spend a lot of time scraping these monopoly businesses, and it seems to me they do a decent job of letting their users decide what data is exposed. Facebook, linkedin, and google all are decent about letting me scrape their public info. That's all I have a right to -- private info should stay private, at the behest of the owner (the User in UCG). You are correct regarding the third point, but I don't see that as a problem. This isn't a solution in search of a problem -- it's a problem without a solution at the moment. Here's a toy example of something I'd like to do: calculate the positive / negative sentiment of commenters at particular baseball fan sites, so I can hide the content I don't like, and show that which I do. Having a common crawl of the site would be immensely useful (and is indeed a prereq) for this. I wouldn't need to republish it, just compute on it.

jtoy 12y ago

anyway we can get an open sourced version of that optimized job for us to play with?

Smerity 12y ago
There's are two levels of optimizations that come into play: AWS setup and the choice of primary language. I'm always happy to speak about both as we love seeing experiments run over the data! AWS optimizations: For that level of cost efficiency, you really need to use spot instances. A cluster of 100 m1.xlarge machines (1.5TB of RAM, 400 cores, and 168TB of magnetic disk storage) will only cost you $3 per hour using spot instances, rather than $30 on-demand. You should pay on-demand prices for the Hadoop master however. You'll also want to roll your own Hadoop cluster as opposed to using Elastic MapReduce (EMR). EMR is amazing but the cost overhead when using it on spot instances is ~100%. For the code itself, this is a situation where you'll want to stick with the programming language that has both the best performance and best ecosystem. I'm personally not a big fan of Java, but it really does win out here -- it's close to C or C++ for performance and has the advantage of the Hadoop ecosystem behind it. Other languages are certainly usable, but even if LanguageX only ran 4x slower than Java, the resulting job would be 4x more expensive due to paying by the hour. Other than that, it's really just a standard MapReduce job using Hadoop. You can see an three examples for the three different data formats we use at: https://github.com/commoncrawl/cc-warc-examples/ https://github.com/commoncrawl/cc-warc-examples/
- elyase 12y ago
  Thanks for your comments. I see you get 400 cores for 3USD/h . Where you optimizing for processing power per dollar? Do you know if the m1.xlarge instances are the best for this use case?
  - Smerity 12y ago
    It really depends on the use case. I mentioned m1.xlarge as they provide a good general cluster setup. The mix of CPU, RAM, and disk space should work well for most experiments one might want to perform. m1.xlarge instances also have 1Gbps network interfaces when others in the same price range have 500Mbps -- a vestige of the older generation of machines. Finally, they excel at disk space. If you're utilizing HDFS heavily, newer instances are usually SSD (good) but have 10 to 20 times less disk storage (bad). For the same dollar amount, you can trade for other specs though. If you're more interested in CPU / RAM / SSD for example, paying $3USD/h for r3.xlarge gets you 3TB of RAM, about 1.5 times more computing power (same number of cores but more compute units), but far less disk space -- 8TB of SSD. In the end, it really depends on the task at hand, but your dollar does go quite far regardless!

stevesalevan 12y ago

So awesome to see that folks are discovering Common Crawl, it's a really great project and the amount of data they've crawled has been tremendous. If you're looking to get into machine learning, it's perhaps the most comprehensive dataset you can get your hands on.

adwf 12y ago

Is there any chance we might see an updated index for the Common Crawl? I've tried using the dataset before, but I found it difficult given that you have to process the entire thing in order to find the particular pages you are looking for. As an example, I was trying a project to look at the top news sites, like BBC, CNN, Al Jazeera, etc. Then matching articles about the same news topic on each site, before finally fact checking for differences between the stories (ie. 20,000 homes were without power vs. 50,000) That kind of project requires a load of crawling, but I can't look for specific pages without processing the entire CC set first. I love the project though, so thank you for doing it!