3 ms·
> have access to virtually the entire internet It isn't even close to 1% of the internet, much less virtually the entire internet. According to the latest dump
by krainboltgreene 3y ago
> have access to virtually the entire internet
It isn't even close to 1% of the internet, much less virtually the entire internet. According to the latest dump, Common Crawl has 4.3B pages, but Google in 2016 estimated there are 130T pages. The difference between 130T and 4.3B is about 130T. Even if you narrow it down to Google's searchable text index it's "100's of billions of pages" and roughly 100P compared to CommonCrawl's 400T.
- fspeech 3y ago130T unique pages? That seems highly unlikely as that averages to over 10000 pages for each human being alive. If gp merely wants texts of interest to self as opposed to an accurate snapshot it seems LLMs should be quite capable, one day.
- darby_eight 3y agoIt doesn't seem that hard to believe given how much automatically generated "content" (mostly garbage) there is. I think a more interesting question is how much information there is on the internet, especially after optimal compression. I'm guessing this is a very difficult question to answer, but also much higher than LLMs currently store.
- cypress66 3y agoIs it? Every user profile in every website is a page. Every single tweet is a page.
- fspeech 3y agoTweets count. HN posts also count (actually as high quality texts :). IOT devices reporting status based on the same templates should not qualify as unique pages (count the number of templates if you want). Now if I do a search for some news, many almost verbatim copies would show up. They should only count as one, as we are looking for unique texts!
- whatever1 3y agoThe internet to me and to most of the people is the 10 first search results for the various terms we search for.