6 ms·

Blekko donates search data to Common Crawl

114 points by dwynings 14y ago

greglindahl 14y ago
If you'd like to see some examples of what you can do with Common Crawl data, here are the winning projects from a code contest held last September: http://commoncrawl.org/announcing-the-winners-of-the-code-contest/ http://commoncrawl.org/announcing-the-winners-of-the-code-co... Some code libraries for using Common Crawl data: https://github.com/commoncrawl/ https://github.com/commoncrawl/ Some clues for getting started: http://commoncrawl.org/get-started/ http://commoncrawl.org/get-started/
- chiph 14y ago
  This is a pretty cool thing that you're doing, Greg.
LisaG 14y ago
I am part of Common Crawl and I just wanted to say that we are super excited about blekko's donation! This is yet another demonstration how much blekko values openness and transparency.
mtgx 14y ago
Last I checked Blekko was pretty good, better than DuckDuckGo in search relevancy I thought, but that was like a year ago.
- jggonz 14y ago
  It has gotten even better! =)
- NonEUCitizen 14y ago
  Does Blekko also NOT track you, like DuckDuckGo ?
  - sp332 14y ago
    The terms seem pretty reasonable, which means they are not as extreme as DDG's. https://blekko.com/about/privacy-policy https://blekko.com/about/privacy-policy
- COKing 14y ago
  Well no one really uses either one. They should merge into one useless web site.
hosay123 14y ago
The point worth note is that it's not an archive of downloaded pages, it's data for the Blekko equivalent of PageRank (i.e. computed relationships, not just the pages). To generate this independently would not only require access to a large crawl, but also robust code and most probably a large cluster to compute it in reasonable time, not to mention legal advice to avoid stepping on Google (et al) patents. If only certain other companies weren't so precious about their publicly derived data. Fabulous donation.
- jacquesm 14y ago
  > not to mention legal advice to avoid stepping on Google (et al) patents. You may be right, you may not be right, but if this is the equivalent of PageRanked data then you may not be in the clear to use this as is. After all if 'PageRank' went into producing it buying it and using it makes you the beneficiary of patent infringement. Personally I'd say so-su-mi, but I still think that it should be noted that the fact that someone else did the infringing does not put you automatically in the clear when using the end product. http://en.wikipedia.org/wiki/Patent_infringement_under_United_States_law http://en.wikipedia.org/wiki/Patent_infringement_under_Unite... Section indirect infringement.
  - AznHisoka 14y ago
    No, this is not the equivalent of PageRank. it could be an approximation however, and they are probably calling it a different name. If that's not legal, then SEOMoz would've been sued a long long time ago (see OpenSiteExplorer, Page Authority, Domain Authority, etc)
    - greglindahl 14y ago
      blekko doesn't compute PageRank, and we don't compute anything similar to it, either. It's highly gamed and less useful than you might think. (The academic equivalent of PageRank for research papers is highly gamed, too, by citation clubs...) By the way, the original PageRank patent is owned and licensed by Stanford University, not by Google.
ssalevan 14y ago
Common Crawl has a really neat mission, as there isn't a whole lot of free and open data out in the world right now and they're trying to change that. With this donation it looks like their commons will be augmented with some great stuff and that can only mean awesome things.
chuhnk 14y ago
I wondered how Google's foothold in search would ever be overcome. I think this might just be the start of something.
- iroy 14y ago
  Google has had years to tweak and tune things. I think it is practically impossible for anyone to match them, let alone surpass them in index quality. Google needs to open its index and create a search market place. There are millions of domain/location specific apps that can be built around that index.
graue 14y ago
This is great, but I'm confused by the part about avoiding porn: > Common Crawl will use blekko’s metadata to improve its crawl quality, while avoiding webspam, porn, and the influence of excessive SEO (search engine optimization) Why avoid porn? Millions of people deliberately search for porn on the internet every day. It's hardly less worthy of crawling than any other content. The next sentence goes on to suggest that porn is not "useful to humans", which is obviously false. If Common Crawl is indeed filtering out content they determine to be pornographic, I hope they are taking care not to also remove information on sexual and relationship health and LGBT rights, which are often collateral damage of porn-blocking systems. And it would be nice to see an open acknowledgement that filtering is going on - I couldn't find any references to this at commoncrawl.org.
- LisaG 14y ago
  Graue I am from Common Crawl. We don't filter for porn. A corpus of web data needs to include porn or it wouldn't be a representative sample of the web ;) We do want to enrich our sample of the web with high-value sites and that is where the blekko data will be so incredibly valuable. I really appreciate your mention of LGBT and sexual health sites being collateral damage - we need to draw more attention to that problem. I would love to see someone work with Common Crawl to improve methods of distinguishing. Lisa
  - graue 14y ago
    Glad to have brought up the issue. Sounds like you should talk to 'randomstring, above.
- jmillikin 14y ago
  > The next sentence goes on to suggest that porn is not > "useful to humans", which is obviously false. It's worth noting that porn is, in fact, of use only to humans.
  - graue 14y ago
    So are webspam and SEO abuse, but just to a small number of humans and not the ones performing the search. :)
ot 14y ago
This is going to be invaluable for information retrieval researchers. Google, MSR, and Yahoo! have an edge on research over universities because of the large amount of data they collect from the users; all the other institutions are left with either small-size benchmark datasets or synthetic data, which are usually not representative of the actual usage scenarios. I myself had to synthetize a query log from the Wikipedia request logs to test some of my data structures on large-scale data. I expect to see a huge number of papers which will use these data in their experiments in the immediate future. Thanks, Blekko!
- robrenaud 14y ago
  As far as I can tell, this contribution from Blekko doesn't have any user data/queries in it. As far as I can tell, this is the best resource for publicly available search engine query logs. http://jeffhuang.com/search_query_logs.html http://jeffhuang.com/search_query_logs.html I don't intend to downplay the contribution, having a large collection of spam/porn classified web docs is still a very nice thing to have for researchers.
  - wumpus 14y ago
    This is our first donation. We have a lot more we plan on giving, but for user queries, for example, the privacy issues are a lot more difficult to work through. We have no interest in being the next privacy scandal.
kjackson2012 14y ago
I've tried Blekko a couple of times, and thought they were okay, but I don't find them so much better than Google to make me switch. I'm curious, how Blekko can stay in business? Do they get enough traffic and revenue from ads, etc, to maintain some sort of positive cash flow, or are they simply burning through cash from investors?
- frederi 14y ago
  For me, blekko doesn't have to be "so much better" to make me use it over Google, it just has to be just as good or slightly better because I respect and support blekko's philosophy. If two products were of equal quality, wouldn't you rather use the one made by a company that shares your values?
  - LisaG 14y ago
    Strongly agree! blekko's Bill of Rights is a great expression of their values and of why we should all be using blekko. blekko Bill of Rights 1. Search shall be open 2. Search results shall involve people 3. Ranking data shall not be kept secret 4. Web data shall be readily available 5. There is no one-size-fits-all for search 6. Advanced search shall be accessible 7. Search engine tools shall be open to all 8. Search & community go hand-in-hand 9. Spam does not belong in search results 10. Privacy of searchers shall not be violated
    - boyter 14y ago
      Although to be fair, you have to pay to get the SEO/Ranking data now. I'm cool with that if it keeps the service open but it should be pointed out. On a related note www.procog.com has a totally open algorithm.
nell 14y ago
I was at the open data meetup in Mozilla where Common Crawl presented and Blekko's CTO was present. Little did I know that great stuff like this was in the making.

114 points by dwynings 14y ago

If you'd like to see some examples of what you can do with Common Crawl data, here are the winning projects from a code contest held last September: http://commoncrawl.org/announcing-the-winners-of-the-code-contest/ http://commoncrawl.org/announcing-the-winners-of-the-code-co... Some code libraries for using Common Crawl data: https://github.com/commoncrawl/ https://github.com/commoncrawl/ Some clues for getting started: http://commoncrawl.org/get-started/ http://commoncrawl.org/get-started/

chiph 14y ago
This is a pretty cool thing that you're doing, Greg.

LisaG 14y ago

I am part of Common Crawl and I just wanted to say that we are super excited about blekko's donation! This is yet another demonstration how much blekko values openness and transparency.

mtgx 14y ago

Last I checked Blekko was pretty good, better than DuckDuckGo in search relevancy I thought, but that was like a year ago.

jggonz 14y ago
It has gotten even better! =)
NonEUCitizen 14y ago
Does Blekko also NOT track you, like DuckDuckGo ?
- sp332 14y ago
  The terms seem pretty reasonable, which means they are not as extreme as DDG's. https://blekko.com/about/privacy-policy https://blekko.com/about/privacy-policy
COKing 14y ago
Well no one really uses either one. They should merge into one useless web site.

hosay123 14y ago

The point worth note is that it's not an archive of downloaded pages, it's data for the Blekko equivalent of PageRank (i.e. computed relationships, not just the pages). To generate this independently would not only require access to a large crawl, but also robust code and most probably a large cluster to compute it in reasonable time, not to mention legal advice to avoid stepping on Google (et al) patents. If only certain other companies weren't so precious about their publicly derived data. Fabulous donation.

jacquesm 14y ago
> not to mention legal advice to avoid stepping on Google (et al) patents. You may be right, you may not be right, but if this is the equivalent of PageRanked data then you may not be in the clear to use this as is. After all if 'PageRank' went into producing it buying it and using it makes you the beneficiary of patent infringement. Personally I'd say so-su-mi, but I still think that it should be noted that the fact that someone else did the infringing does not put you automatically in the clear when using the end product. http://en.wikipedia.org/wiki/Patent_infringement_under_United_States_law http://en.wikipedia.org/wiki/Patent_infringement_under_Unite... Section indirect infringement.
- AznHisoka 14y ago
  No, this is not the equivalent of PageRank. it could be an approximation however, and they are probably calling it a different name. If that's not legal, then SEOMoz would've been sued a long long time ago (see OpenSiteExplorer, Page Authority, Domain Authority, etc)
  - greglindahl 14y ago
    blekko doesn't compute PageRank, and we don't compute anything similar to it, either. It's highly gamed and less useful than you might think. (The academic equivalent of PageRank for research papers is highly gamed, too, by citation clubs...) By the way, the original PageRank patent is owned and licensed by Stanford University, not by Google.

ssalevan 14y ago

Common Crawl has a really neat mission, as there isn't a whole lot of free and open data out in the world right now and they're trying to change that. With this donation it looks like their commons will be augmented with some great stuff and that can only mean awesome things.

chuhnk 14y ago

I wondered how Google's foothold in search would ever be overcome. I think this might just be the start of something.

iroy 14y ago
Google has had years to tweak and tune things. I think it is practically impossible for anyone to match them, let alone surpass them in index quality. Google needs to open its index and create a search market place. There are millions of domain/location specific apps that can be built around that index.

graue 14y ago

This is great, but I'm confused by the part about avoiding porn: > Common Crawl will use blekko’s metadata to improve its crawl quality, while avoiding webspam, porn, and the influence of excessive SEO (search engine optimization) Why avoid porn? Millions of people deliberately search for porn on the internet every day. It's hardly less worthy of crawling than any other content. The next sentence goes on to suggest that porn is not "useful to humans", which is obviously false. If Common Crawl is indeed filtering out content they determine to be pornographic, I hope they are taking care not to also remove information on sexual and relationship health and LGBT rights, which are often collateral damage of porn-blocking systems. And it would be nice to see an open acknowledgement that filtering is going on - I couldn't find any references to this at commoncrawl.org.

LisaG 14y ago
Graue I am from Common Crawl. We don't filter for porn. A corpus of web data needs to include porn or it wouldn't be a representative sample of the web ;) We do want to enrich our sample of the web with high-value sites and that is where the blekko data will be so incredibly valuable. I really appreciate your mention of LGBT and sexual health sites being collateral damage - we need to draw more attention to that problem. I would love to see someone work with Common Crawl to improve methods of distinguishing. Lisa
- graue 14y ago
  Glad to have brought up the issue. Sounds like you should talk to 'randomstring, above.
jmillikin 14y ago
> The next sentence goes on to suggest that porn is not > "useful to humans", which is obviously false. It's worth noting that porn is, in fact, of use only to humans.
- graue 14y ago
  So are webspam and SEO abuse, but just to a small number of humans and not the ones performing the search. :)

ot 14y ago

This is going to be invaluable for information retrieval researchers. Google, MSR, and Yahoo! have an edge on research over universities because of the large amount of data they collect from the users; all the other institutions are left with either small-size benchmark datasets or synthetic data, which are usually not representative of the actual usage scenarios. I myself had to synthetize a query log from the Wikipedia request logs to test some of my data structures on large-scale data. I expect to see a huge number of papers which will use these data in their experiments in the immediate future. Thanks, Blekko!

robrenaud 14y ago
As far as I can tell, this contribution from Blekko doesn't have any user data/queries in it. As far as I can tell, this is the best resource for publicly available search engine query logs. http://jeffhuang.com/search_query_logs.html http://jeffhuang.com/search_query_logs.html I don't intend to downplay the contribution, having a large collection of spam/porn classified web docs is still a very nice thing to have for researchers.
- wumpus 14y ago
  This is our first donation. We have a lot more we plan on giving, but for user queries, for example, the privacy issues are a lot more difficult to work through. We have no interest in being the next privacy scandal.

kjackson2012 14y ago

I've tried Blekko a couple of times, and thought they were okay, but I don't find them so much better than Google to make me switch. I'm curious, how Blekko can stay in business? Do they get enough traffic and revenue from ads, etc, to maintain some sort of positive cash flow, or are they simply burning through cash from investors?

frederi 14y ago
For me, blekko doesn't have to be "so much better" to make me use it over Google, it just has to be just as good or slightly better because I respect and support blekko's philosophy. If two products were of equal quality, wouldn't you rather use the one made by a company that shares your values?
- LisaG 14y ago
  Strongly agree! blekko's Bill of Rights is a great expression of their values and of why we should all be using blekko. blekko Bill of Rights 1. Search shall be open 2. Search results shall involve people 3. Ranking data shall not be kept secret 4. Web data shall be readily available 5. There is no one-size-fits-all for search 6. Advanced search shall be accessible 7. Search engine tools shall be open to all 8. Search & community go hand-in-hand 9. Spam does not belong in search results 10. Privacy of searchers shall not be violated
  - boyter 14y ago
    Although to be fair, you have to pay to get the SEO/Ranking data now. I'm cool with that if it keeps the service open but it should be pointed out. On a related note www.procog.com has a totally open algorithm.

nell 14y ago

I was at the open data meetup in Mozilla where Common Crawl presented and Blekko's CTO was present. Little did I know that great stuff like this was in the making.