42 ms·

"In 2010, Google estimated that it had indexed just 0.004% of the internet." I don't believe this. Does anyone else?

by codex 13y ago

agilebyte 13y ago
Infographic with source at the bottom here: http://urlm.co/blog/2010/10/28/the-awesome-size-of-the-internet-infographic/ http://urlm.co/blog/2010/10/28/the-awesome-size-of-the-inter...
- codex 13y ago
  So Google has indexed 200TB? My cat pictures alone are 1TB!
thebooktocome 13y ago
There are some giant internet databases behind a "Disallow: /".
p4bl0 13y ago
This sentence made me bug too. One should not mix up the web and the internet. Google only indexes the web (well, and a part of usenet). Found this: http://webapps.stackexchange.com/questions/11740/how-much-of-the-internet-is-indexed-by-google http://webapps.stackexchange.com/questions/11740/how-much-of... which support the claim in the article.
damoncali 13y ago
A shockingly high percentage of the internet is spam or other junk that is never seen. I don't know about .004%, but I don't doubt that Google doesn't index the absolutely huge volumes of crap out there.
- rz2k 13y ago
  Google may index a lot of the spam as relatively useless since it is often found at top levels. If the number is correct what it is probably missing is all of the content that lies in databases, easily accessible, but not necessarily on the front pages of web sites, or found without entering a query of some sort.
miahi 13y ago
It's probably because Google mostly indexes public-accessible text and images. I have a home server with a few TB of Internet-accessible storage, but it's not public. I can say the data is on the Internet, but cannot be indexed as it's not directly accessible. My cloud backup of that storage is on the Internet but it's not public and it's encrypted, so it cannot be indexed (maybe the NSA is crunching my 4096-bit key right now). Based on the infographic agilebyte provided (sibling post), I contribute to ~10TB of data that cannot be indexed by Google - and that's already 5% of the data Google had indexed by 2010!

"In 2010, Google estimated that it had indexed just 0.004% of the internet." I don't believe this. Does anyone else?

by codex 13y ago

"In 2010, Google estimated that it had indexed just 0.004% of the internet." I don't believe this. Does anyone else?

Infographic with source at the bottom here: http://urlm.co/blog/2010/10/28/the-awesome-size-of-the-internet-infographic/ http://urlm.co/blog/2010/10/28/the-awesome-size-of-the-inter...

codex 13y ago
So Google has indexed 200TB? My cat pictures alone are 1TB!

thebooktocome 13y ago

There are some giant internet databases behind a "Disallow: /".

p4bl0 13y ago

This sentence made me bug too. One should not mix up the web and the internet. Google only indexes the web (well, and a part of usenet). Found this: http://webapps.stackexchange.com/questions/11740/how-much-of-the-internet-is-indexed-by-google http://webapps.stackexchange.com/questions/11740/how-much-of... which support the claim in the article.

damoncali 13y ago

A shockingly high percentage of the internet is spam or other junk that is never seen. I don't know about .004%, but I don't doubt that Google doesn't index the absolutely huge volumes of crap out there.

rz2k 13y ago
Google may index a lot of the spam as relatively useless since it is often found at top levels. If the number is correct what it is probably missing is all of the content that lies in databases, easily accessible, but not necessarily on the front pages of web sites, or found without entering a query of some sort.

miahi 13y ago

It's probably because Google mostly indexes public-accessible text and images. I have a home server with a few TB of Internet-accessible storage, but it's not public. I can say the data is on the Internet, but cannot be indexed as it's not directly accessible. My cloud backup of that storage is on the Internet but it's not public and it's encrypted, so it cannot be indexed (maybe the NSA is crunching my 4096-bit key right now). Based on the infographic agilebyte provided (sibling post), I contribute to ~10TB of data that cannot be indexed by Google - and that's already 5% of the data Google had indexed by 2010!