26 ms·
> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is
by wolverine876 5y ago
> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.?
The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"?
The Wayback Machine possibly would need a much larger index than any normal search engine: not only the present websites, but all the historical versions (though I don't know what proportion of the web they've indexed).
- gregsadetsky 5y agoNo you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built, and both would really like to have that (or any) search engine to look through the archive..! :-) As for the index, I agree -- I was trying to guesstimate its size by going from the ~~20PB total Wayback Machine size (which includes all historical versions). Is it 1% of 20PB (for the size of the text content), and then another ~10% of that for the index size? So 20TB...?
- wolverine876 5y agoI may have given the wrong impression: it sounds like a good idea to me. > I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? If we think about use cases, users may often search specific domains. In that case, results ordered by frequency and/or date might be sufficient and even desireable. > I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built It's often the first step! > So 20TB...? That doesn't sound so bad.
- gregsadetsky 5y agoI agree & appreciate your response. I have 0 time for this, but I also can't easily let go ;-) Want to collab on this?
- wolverine876 5y agoWow, great offer and a great project. I also have 0 time and I'm dealing with extra obligations anyway. I'm afraid I need to be disciplined to keep my primary life goals - things that take a decade or more - on course. I am really thrilled by the energy you are showing, though, and hate to contribute anything negative. Please don't stop for me! You made my day.
- gregsadetsky 5y agoThree cheers, thank you for the nice exchange! Happy upcoming Holidays. I'll (obviously) post on HN if anything comes out of this.