3 ms·

I was talking about text-only, filtered and deduped content. Most of Google's 100PB is picture and video. Filtering the spam and deduping the content helped Go

I was talking about text-only, filtered and deduped content. Most of Google's 100PB is picture and video. Filtering the spam and deduping the content helped Google reduce the ~50B page index in 2012 to ~<10B today.

deleted 2y ago
[deleted]
the-rc 2y ago
Where are these figures coming from?
Larrikin 2y ago
But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s and all the content you just threw away as irrelevant actually contains information I am looking for. There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext. I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.
- graemep 2y ago
  There is still large opportunity. Most of my searches are for plain text information. > But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content. > There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext. If they want video they will do the same as everyone else and search Youtube. Different niche. > I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description. That is true, but if you are getting bad search results (and the market for other search engines are people who are not happy with Google and Bing results) that does not help much are you are not seeing the information you want anyway.
  - flir 2y ago
    > That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content. Ya know... a search engine that was limited to *.gov, *.edu and country equivalents (*.ac.uk, etc) would actually be pretty useful. Ok, I know you can do something like it with site: modifiers in the search, but if you know from the beginning you're never going to search the commercial internet you can bake that assumption into the design of your search engine in interesting ways. And the spam problem goes away. Hmm.
ipaddr 2y ago
That explains why you were 10 times more likely to find something 15-20 years ago then you are today. They reduced the size by dropping a lot of sites and not crawling as much. We expect google to be at 100PB x 100 with the growth of users and content over that time period. Someone made the decision to prioritize a smaller size over a more complete index and some A/B test was run and turned out well.

deleted 2y ago

[deleted]

the-rc 2y ago

Where are these figures coming from?

Larrikin 2y ago

But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s and all the content you just threw away as irrelevant actually contains information I am looking for. There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext. I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description.

graemep 2y ago
There is still large opportunity. Most of my searches are for plain text information. > But what if I don't want to search Reddit, stack overflow, and blogs from the early 2000s That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content. > There is an entire working generation that never heard a modem sound and has never even made a consideration for making sure their content is accessible in plaintext. If they want video they will do the same as everyone else and search Youtube. Different niche. > I'm sure all the LLM providers are already considering this, but there's so much important information that is locked away in videos and pictures that isn't even obvious from a transcript or description. That is true, but if you are getting bad search results (and the market for other search engines are people who are not happy with Google and Bing results) that does not help much are you are not seeing the information you want anyway.
- flir 2y ago
  > That is a strawman. There are huge numbers of websites (including authoritative ones like governments and universities) and a lot of content. Ya know... a search engine that was limited to *.gov, *.edu and country equivalents (*.ac.uk, etc) would actually be pretty useful. Ok, I know you can do something like it with site: modifiers in the search, but if you know from the beginning you're never going to search the commercial internet you can bake that assumption into the design of your search engine in interesting ways. And the spam problem goes away. Hmm.

ipaddr 2y ago

That explains why you were 10 times more likely to find something 15-20 years ago then you are today. They reduced the size by dropping a lot of sites and not crawling as much. We expect google to be at 100PB x 100 with the growth of users and content over that time period. Someone made the decision to prioritize a smaller size over a more complete index and some A/B test was run and turned out well.