3 ms·
Making the index available would be very valuable for scientific research. when I was a googler, we used the index (and google scholar's index) to do all sorts
by dekhn 7y ago
Making the index available would be very valuable for scientific research. when I was a googler, we used the index (and google scholar's index) to do all sorts of interesting science projects (DNA search, gene search, etc) But we couldn't publish the results for various reasons. If the index (or a fragment of it) was available sitting in parquet files (or possibly a better format for indices) you could easily sit around doing spark jobs to extract all sorts of interesting web data.
- sytelus 7y agoThere are tons of projects with serious sizable open source index, for ex, http://commoncrawl.org/ http://commoncrawl.org/. Even crawling 15B pages on AWS is not super expensive on your own using well established OSS tool chain. Bing already provides APIs if you don’t want to do crawling. Web index, while important, is by no means most important part of search engine (which I had say is relevance). The OP article is written by someone who has little clue that search engine involves massive amount of technologies besides index and even relevance - everything from spell correction to recommendations to query rewriting to answers to segment specific searches like images/video/maps/local/product/news/entertainment/events, so on and on and on. This is above and beyond the gigantic infrastructure needed to run all these at scale, speed and cost effectively. All of these needs to work harmoniously with each other and designed keeping in mind weaknesses and strength of each component. For example, index for news must be refreshed almost in real time and relevance needs higher emphasis on location. Search is not free and no one has figured out if someone just can provide index and someone else can provide relevance and everything can just work as efficiently as before (data point: each query consumes 0.3 Watt-hour of energy at Google).