4 ms·

A full-text search for the Wayback Machine would be my top feature request. It's not uncommon to lose the URL of a site and for active webpages to not have the

A full-text search for the Wayback Machine would be my top feature request. It's not uncommon to lose the URL of a site and for active webpages to not have the URL of the old website. Plus I'm sure there are many interesting archived webpages I could find with a full-text search. I understand they've tried this or things like it a few times but they haven't ever kept the feature.

gregsadetsky 5y ago
I imagine that the costs to run this would outweight the potential marketing benefits, but it'd be amazing to see Algolia take this project on to benefit everyone. The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately? I imagine that creating (and maintaining, of course) the index would be the most time-consuming part? Is it at all possible to imagine hosting this index... somewhere... and doing sqlite http range-like queries on it..? Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? And have separate docid -> archive.org url mapping?
- dreamcompiler 5y ago
  I'd think you'd want something with at least the features of Lucene, whose index is 20%-30% the size of the data. https://lucene.apache.org/core/features.html https://lucene.apache.org/core/features.html
- wolverine876 5y ago
  > Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"? The Wayback Machine possibly would need a much larger index than any normal search engine: not only the present websites, but all the historical versions (though I don't know what proportion of the web they've indexed).
  - gregsadetsky 5y ago
    No you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built, and both would really like to have that (or any) search engine to look through the archive..! :-) As for the index, I agree -- I was trying to guesstimate its size by going from the ~~20PB total Wayback Machine size (which includes all historical versions). Is it 1% of 20PB (for the size of the text content), and then another ~10% of that for the index size? So 20TB...?
    - wolverine876 5y ago
      I may have given the wrong impression: it sounds like a good idea to me. > I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? If we think about use cases, users may often search specific domains. In that case, results ordered by frequency and/or date might be sufficient and even desireable. > I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built It's often the first step! > So 20TB...? That doesn't sound so bad.
      - gregsadetsky 5y ago
        I agree & appreciate your response. I have 0 time for this, but I also can't easily let go ;-) Want to collab on this?
        wolverine876 5y ago
        Wow, great offer and a great project. I also have 0 time and I'm dealing with extra obligations anyway. I'm afraid I need to be disciplined to keep my primary life goals - things that take a decade or more - on course. I am really thrilled by the energy you are showing, though, and hate to contribute anything negative. Please don't stop for me! You made my day.

gregsadetsky 5y ago

I imagine that the costs to run this would outweight the potential marketing benefits, but it'd be amazing to see Algolia take this project on to benefit everyone. The Wayback Machine's data is ~20PB..? What is approximately the size of the indexable text (i.e. the text content of html pages, sans tags)? And what would the index size be like, approximately? I imagine that creating (and maintaining, of course) the index would be the most time-consuming part? Is it at all possible to imagine hosting this index... somewhere... and doing sqlite http range-like queries on it..? Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? And have separate docid -> archive.org url mapping?

dreamcompiler 5y ago
I'd think you'd want something with at least the features of Lucene, whose index is 20%-30% the size of the data. https://lucene.apache.org/core/features.html https://lucene.apache.org/core/features.html
wolverine876 5y ago
> Would it be enough to have an index consist of a list of found words, and the related "document ids"? i.e. "apple" is in doc ids 1000, 2000, 3000, "banana" is in doc ids 2000, 4000, etc.? The problem, as with any search engine, is the ranking algorithm. Without a sufficent one, the search results are useless. What use is a list of every page in the Wayback Machine containing the word "apple"? The Wayback Machine possibly would need a much larger index than any normal search engine: not only the present websites, but all the historical versions (though I don't know what proportion of the web they've indexed).
- gregsadetsky 5y ago
  No you are right of course. I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built, and both would really like to have that (or any) search engine to look through the archive..! :-) As for the index, I agree -- I was trying to guesstimate its size by going from the ~~20PB total Wayback Machine size (which includes all historical versions). Is it 1% of 20PB (for the size of the text content), and then another ~10% of that for the index size? So 20TB...?
  - wolverine876 5y ago
    I may have given the wrong impression: it sounds like a good idea to me. > I was imagining that a full text Wayback Machine search engine would mostly be useful to look for words unique enough that sifting through a lot of results (even if those were not ranked "well") could still be useful..? If we think about use cases, users may often search specific domains. In that case, results ordered by frequency and/or date might be sufficient and even desireable. > I was very naively "back of the envelope" prototyping a search engine. I both realize that this is not the way that these things are built It's often the first step! > So 20TB...? That doesn't sound so bad.
    - gregsadetsky 5y ago
      I agree & appreciate your response. I have 0 time for this, but I also can't easily let go ;-) Want to collab on this?
      - wolverine876 5y ago
        Wow, great offer and a great project. I also have 0 time and I'm dealing with extra obligations anyway. I'm afraid I need to be disciplined to keep my primary life goals - things that take a decade or more - on course. I am really thrilled by the energy you are showing, though, and hate to contribute anything negative. Please don't stop for me! You made my day.