3 ms·
No you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting t
by gregsadetsky 5y ago
No you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..?
I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built, and both would really like to have that (or any) search engine to look through the archive..! :-)
As for the index, I agree -- I was trying to guesstimate its size by going from the ~~20PB total Wayback Machine size (which includes all historical versions). Is it 1% of 20PB (for the size of the text content), and then another ~10% of that for the index size? So 20TB...?
- wolverine876 5y agoI may have given the wrong impression: it sounds like a good idea to me. > I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? If we think about use cases, users may often search specific domains. In that case, results ordered by frequency and/or date might be sufficient and even desireable. > I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built It's often the first step! > So 20TB...? That doesn't sound so bad.
- gregsadetsky 5y agoI agree & appreciate your response. I have 0 time for this, but I also can't easily let go ;-) Want to collab on this?
- wolverine876 5y agoWow, great offer and a great project. I also have 0 time and I'm dealing with extra obligations anyway. I'm afraid I need to be disciplined to keep my primary life goals - things that take a decade or more - on course. I am really thrilled by the energy you are showing, though, and hate to contribute anything negative. Please don't stop for me! You made my day.
- gregsadetsky 5y agoThree cheers, thank you for the nice exchange! Happy upcoming Holidays. I'll (obviously) post on HN if anything comes out of this.