2 ms·
15-25TB to provide search for the entire internet archive. This is a pretty small number if you think about it. Your avg undergrad class room would be able matc
by gus7 9y ago
15-25TB to provide search for the entire internet archive. This is a pretty small number if you think about it. Your avg undergrad class room would be able match it if they pooled all their devices together. So why do we need Google and Bing ladies and gents?
If we can get a 100 devs in every city in every corner of the planet to donate a hardrive/cpuand some time, distributed open search could well do to Google what Linux did to Microsoft. I think its going to happen. There is lot of duplication of the above work going on in little silos like Wikipedia, Khan academy, Stackoverflow, Zealdocs etc it just has get hooked up
- btian 9y agoYou know Google is free right? No need to donate resources. And latency will be so horrible if you get 100 devs in every city to donate resources.
- gwern 9y agoI'd love to see fulltext someday on their Wayback Machine WWW corpus, not just their side projects. I've used it heavily in researching various things but the limitation to querying by domain or URL is highly restrictive: I'm certain that many of the pages I've failed to find are present in the Wayback Machine at other URLs, either after moving or being quoted or mirrored. (For example, Google's Blogspot has this amazingly horrible thing of providing the same blog across dozens of national TLDs like 'blog.com', 'blog.co.uk', 'blog.fr' etc - so there may be a full copy of that blog in the WM except it's split across dozens of domains and the Wayback Machine forces you to check every possible permutation of domain name since it can't do a fulltext search.)
- hoschicz 9y agoNote that this search is in everything the Archive backs up except the Web.