4 ms·

Lucene is moving away from TF-IDF to BM25 as the default. Pretty similar idea, but tends to performs a better with short content. https://issues.apache.org/jir

• meeper16 11y ago
  Vector Space replacing or being combined with TF-IDF approaches is new way of summarizing and searching for meaning in documents... http://52.11.1.7/TuataraSum/example_context_control-ml2.html http://52.11.1.7/TuataraSum/example_context_control-ml2.html
  • gibrown 11y ago
    Interesting. This basically uses the background word2vec data for the entire Web to provide more information and help with things like disambiguation, synonyms, etc? Am I understanding that correctly? Maybe nit-picky thought, but its not clear to me that the TF-IDF part is what's doing a lot of extra lifting there. Do you know of any good evaluations between using vector space data and other methods for summarization?
    • meeper16 11y ago
      Word2Vec was a fork or based on a more exhuastive vector space approach here https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/12349/word2vec-is-based-on-an-approach-from-lawrence-berkeley-national-lab https://www.kaggle.com/c/word2vec-nlp-tutorial/forums/t/1234... I've compared the summarization to others like OTS http://libots.sourceforge.net/ http://libots.sourceforge.net/ which I believe strictly relies on TF-IDF and it seems better and allows for context to control the summarization. Other similar approaches might be based on Latent Semantic Analysis, Latent Semantic Indexing or LDA.
      • gibrown 11y ago
        Thanks for the links!