5 ms·

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for re

I dont believe Common Crawl offers a real time search index as its delayed by more than a month (although that could have changed recently). Still useful for research purposes but not that desirable for a search engine that competes with Google, etc.

onion2k 8y ago
Would users notice for many searches? Obviously it wouldn't be useful for news or social media, but for practically everything else a latency of a month would be fine.
- walterbell 8y ago
  Yes. As the Internet Archive shows, a large corpus of valuable content is no longer changing.
- IanCal 8y ago
  It absolutely breaks any use for information produced in the last month. Here's a few things that come to mind: While you say "news" really that covers any information about current-ish events. It's not just "what happened today" but background on things like the Muller report right now. Any technology release, or update. Reviews of any hardware or software. Information about security vulnerabilities. Film reviews. Game reviews. Book reviews. New scientific publications.
  - Retric 8y ago
    Books, games, hardware etc are all sold for several years. What you’re describing is just news on the latest updates representing a small slice of the market. Making the remander far from useless.
  - onion2k 8y ago
    I don't know this for certain but I strongly suspect news (and reviews/criticism, which is editorialised news) doesn't make up a huge percentage of search traffic. People read news sites that align with their preexisting points of view. They don't often go looking for new perspectives. If someone wants to know what's happening they want the filter of their preferred news outlet, if they want a review they want to read or watch their preferred reviewer. They don't want whatever happens to be the top search result. Although, that said, with Google personalising search results the top result is very likely to be the user's preferred site anyway. We can't have people seeing outside their filter bubble after all.
tgbugs 8y ago
For many use cases I would imagine that an index that was a bit delayed might actually be preferred. I'm not entirely sure what you meant to imply by 'research purposes' but many of the use cases I imagine are scholarly use cases where something a more stable would be preferable. That said I seem to recall Henry Thompson telling a story about trying to do a study of the statistics of the net using common crawl. By the time he was done he ended up being less certain of the results, the understanding, and the methodological validity of anything related to trying to measure the internet by looking at a single snapshot of a subset of the link structure. Too hard to understand what you are actually counting. edit: yep here it is https://doi.org/10.1145/3184558.3191636 https://doi.org/10.1145/3184558.3191636
Aeolun 8y ago
Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.
- bogomipz 8y ago
  I'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?
  - jdsully 8y ago
    A site you just learned about probably isn't very important, so you can just watch its month to month change. You don't care if they change that much because they aren't important yet. A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.
amitamb 8y ago
Apart from that Common Crawl respects robots.txt (which makes sense) so many sites you expect to see there are not indexed. Netflix, Facebook LinkedIn and many more. If common-crawl sees serious adoption those sites will modify their robots.txt but it's and chicken/egg problem.
- amelius 8y ago
  There is a simple solution: if companies do not respect do-not-track then why should we respect robots.txt?
  - nostrademons 8y ago
    Because then you end up in an arms race that the little guy usually does not win. There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations).
burtonator 8y ago
This is literally why I created my company: http://www.datastreamer.io/ http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline extortionate pricing and I fundamentally believe that the web MUST remain open. We've also been providing data for very affordable pricing to researchers for more than a decade. Search for us as Spinn3r under Google Scholar (our previous name) and we have hundreds and hundreds of PhDs who have access to our data. We do charge for research usage now but it's very very very affordable. The entire point is that we're trying to enable innovation.
- Sir_Cmpwn 8y ago
  This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.
  - la6470 8y ago
    You are a fool ... it takes money to run servers pay for electricity and cooling and salaries for the people who work for it. Don’t talk like a child.
  - sytelus 8y ago
    You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.
    - 2stop 8y ago
ape4 8y ago
Yup, article says "A search engine needs to keep its index current, meaning it needs to update at least a part of it every minute. This is an important requirement that is not being met by any of the current projects (like Common Crawl) aiming at indexing snapshots of (parts of) the Web."

onion2k 8y ago

Would users notice for many searches? Obviously it wouldn't be useful for news or social media, but for practically everything else a latency of a month would be fine.

walterbell 8y ago
Yes. As the Internet Archive shows, a large corpus of valuable content is no longer changing.
IanCal 8y ago
It absolutely breaks any use for information produced in the last month. Here's a few things that come to mind: While you say "news" really that covers any information about current-ish events. It's not just "what happened today" but background on things like the Muller report right now. Any technology release, or update. Reviews of any hardware or software. Information about security vulnerabilities. Film reviews. Game reviews. Book reviews. New scientific publications.
- Retric 8y ago
  Books, games, hardware etc are all sold for several years. What you’re describing is just news on the latest updates representing a small slice of the market. Making the remander far from useless.
- onion2k 8y ago
  I don't know this for certain but I strongly suspect news (and reviews/criticism, which is editorialised news) doesn't make up a huge percentage of search traffic. People read news sites that align with their preexisting points of view. They don't often go looking for new perspectives. If someone wants to know what's happening they want the filter of their preferred news outlet, if they want a review they want to read or watch their preferred reviewer. They don't want whatever happens to be the top search result. Although, that said, with Google personalising search results the top result is very likely to be the user's preferred site anyway. We can't have people seeing outside their filter bubble after all.

tgbugs 8y ago

For many use cases I would imagine that an index that was a bit delayed might actually be preferred. I'm not entirely sure what you meant to imply by 'research purposes' but many of the use cases I imagine are scholarly use cases where something a more stable would be preferable. That said I seem to recall Henry Thompson telling a story about trying to do a study of the statistics of the net using common crawl. By the time he was done he ended up being less certain of the results, the understanding, and the methodological validity of anything related to trying to measure the internet by looking at a single snapshot of a subset of the link structure. Too hard to understand what you are actually counting. edit: yep here it is https://doi.org/10.1145/3184558.3191636 https://doi.org/10.1145/3184558.3191636

Aeolun 8y ago

Google doesn’t crawl all of the internet very often either. Only sites that have proven to change a lot. So you could presumably supplement commoncrawl with your own more regular crawls.

bogomipz 8y ago
I'm curious how they track and rank a site's "change velocity" without crawling all of the internet all of the time. It almost seems like a catch 22 no? Might you have any insight into how this works? Any suggested reading or links?
- jdsully 8y ago
  A site you just learned about probably isn't very important, so you can just watch its month to month change. You don't care if they change that much because they aren't important yet. A site that gets lots of links quickly (and is therefore important) will likely garner them from sites you are already frequently visiting.

amitamb 8y ago

Apart from that Common Crawl respects robots.txt (which makes sense) so many sites you expect to see there are not indexed. Netflix, Facebook LinkedIn and many more. If common-crawl sees serious adoption those sites will modify their robots.txt but it's and chicken/egg problem.

amelius 8y ago
There is a simple solution: if companies do not respect do-not-track then why should we respect robots.txt?
- nostrademons 8y ago
  Because then you end up in an arms race that the little guy usually does not win. There are a significant number of crawlers out there that don't respect robots.txt. The usual response to them isn't to roll over dead, it's to get CloudFlare (on the technological end) and/or sic the lawyers on them (for CFAA, IP, or ToS violations).

burtonator 8y ago

This is literally why I created my company: http://www.datastreamer.io/ http://www.datastreamer.io/ We've been around for about a decade. IBM watson used us as their social data provider during Jeopardy. We provide data to tons of companies and you're probably using our services - just that it's not obvious where we're used since it's SaaS B2B and not B2C. We're not free but the primary reason we exist is that other vendors charge borderline extortionate pricing and I fundamentally believe that the web MUST remain open. We've also been providing data for very affordable pricing to researchers for more than a decade. Search for us as Spinn3r under Google Scholar (our previous name) and we have hundreds and hundreds of PhDs who have access to our data. We do charge for research usage now but it's very very very affordable. The entire point is that we're trying to enable innovation.

Sir_Cmpwn 8y ago
This doesn't make any sense. You talk about open data but yours is the opposite. You're just another commercial data hoarder, please don't act like you're not.
- la6470 8y ago
  You are a fool ... it takes money to run servers pay for electricity and cooling and salaries for the people who work for it. Don’t talk like a child.
- sytelus 8y ago
  You are mistaking between free and open. You can be open without being free. Maintaining web index is extremely expensive. Imagine storing most of the web on your own servers and serving it. Someone has to pay bills for all those disk space and bandwidth. I don’t think web index would ever be free (unless storage, compute and bandwidth were free) but having at reasonably priced is a very good thing. I would hope these indices are available on AWS, Azure etc where people can just use it with cloud compute and pay per use.
  - 2stop 8y ago

ape4 8y ago

Yup, article says "A search engine needs to keep its index current, meaning it needs to update at least a part of it every minute. This is an important requirement that is not being met by any of the current projects (like Common Crawl) aiming at indexing snapshots of (parts of) the Web."