3 ms·

I used to work on textual Information Retrieval (which is quite related to search). One of the key contributions of my PhD thesis was a statistical method to ex

I used to work on textual Information Retrieval (which is quite related to search). One of the key contributions of my PhD thesis was a statistical method to extract concepts from texts [0]. Surprisingly enough, with those concepts, Tf-Idf [1] was quite good to extract keywords from documents which allowed us to build document descriptor tables which can eventually be used for document search [2]. We also built a small prototype that allowed us to retrieve results of specific areas of documents where the searched concepts were more "valuable" than on other parts of the document [3]. It is a very interesting subarea of AI/NLP which unfortunately doesn't seem to attract much interest. Since the article also talks a bit about Wikipedia dumps datasets, here's a tool that I've created to build textual corpora: https://github.com/joaoventura/WikiCorpusExtractor https://github.com/joaoventura/WikiCorpusExtractor [0] - http://www.sciencedirect.com/science/article/pii/S1877050912001251 http://www.sciencedirect.com/science/article/pii/S1877050912... [1] - https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://en.wikipedia.org/wiki/Tf%E2%80%93idf [2] - https://link.springer.com/chapter/10.1007/978-3-642-40669-0_42 https://link.springer.com/chapter/10.1007/978-3-642-40669-0_... [3] - https://link.springer.com/chapter/10.1007/978-3-642-40669-0_44 https://link.springer.com/chapter/10.1007/978-3-642-40669-0_...

garysieling 9y ago
How does your research compare to word2vec? That seems to be getting some attention
- jventura 9y ago
  Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space. This seems quite similar to LSA (Latent Semantic Analysis). In LSA, a word is represented by a vector (list / array) of values which represent the number of occurrences of the word in a document. For instance, the list [0, 5, 9, 10] for the word "aeroplane" means than "aeroplane" occurs zero times in D1 (document #1), 5 times in D2, etc. Unlike LSA, word2vec seems to have the vectors values implicit in the neural network weights (?). The interesting thing on these kind of approaches is that words that share similar contexts, tend to occur in the same documents. In other words, the vectors tend to have approximate values. For instance, it is natural that "aeroplane" tends to occur in the same set of documents as "aeroport".. Tf-Idf [2] does something similar, so these vectors are mostly a data representation of single words. As for my research, the basis of it was to be able to quantify how "specific" a word or multiword is. The idea is that the more a word/multiword occurs in different contexts (in my case, near different words) the less specific it is. Being able to identify the specificity of words allowed me to know which words/multiwords were probably concepts and which ones weren't. Then, with these concepts I was able to use Tf-Idf to extract document keywords and later I was able to infer broad relationships between concepts (like in if two words tend to co-occur in the same documents and same parts of documents, they are somewhat related). So, how does word2vec compare to my research? - I prefer simple statistical approaches like the one I did (which is just counting frequencies and dividing numbers, etc.) than the black-box approach of neural networks. I can tell you why my research provides good results (explained above) but if you read a follow-up paper on word2vec [3] this is what you can find in section 4: "Why does this produce good word representations? Good question. We don’t really know". - Also, bag-of-box approaches like word2vec and LSA don't tend to represent multiwords (such as "President of the United States of America", which leaves out lots of information on texts. - Finally, I think there's more possible applications after you know which words/multiwords are informative. With word2vec and LSA you are only representing something similar to frequencies of words in documents, which mostly allows for Information Retrieval-like applications. On the other hand, one of the applications I did with the extracted concepts was to implement a prototype which allowed us to find the definition of a concept on the corpus. The idea was that the definition of a concept was the paragraph where that concept occurred many times but also used other concepts to help the first one to be defined. I don't know how you could do something like that using something like word2vec. But as I said above, I am not that knowledgeable about word2vec, so I may miss something.. [0] - https://en.wikipedia.org/wiki/Word2vec https://en.wikipedia.org/wiki/Word2vec [1] - https://arxiv.org/abs/1301.3781 https://arxiv.org/abs/1301.3781 [2] - https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://en.wikipedia.org/wiki/Tf%E2%80%93idf [3] - https://arxiv.org/pdf/1402.3722.pdf https://arxiv.org/pdf/1402.3722.pdf
paduc 9y ago
Hey Joao, I've been using your WikiCorpusExtractor to work on the wiki dumps for years now for my NLP research. It has provided tremendous value to me. Thank you very much for your work and dedication !

garysieling 9y ago

How does your research compare to word2vec? That seems to be getting some attention

jventura 9y ago
Neural networks are all the rage these days. As my knowledge on NNs is quite limited, what I could grasp from the wikipedia article [0] and the original paper [1] is that word2vec uses a vector to represent words in a n-dimensional space. This seems quite similar to LSA (Latent Semantic Analysis). In LSA, a word is represented by a vector (list / array) of values which represent the number of occurrences of the word in a document. For instance, the list [0, 5, 9, 10] for the word "aeroplane" means than "aeroplane" occurs zero times in D1 (document #1), 5 times in D2, etc. Unlike LSA, word2vec seems to have the vectors values implicit in the neural network weights (?). The interesting thing on these kind of approaches is that words that share similar contexts, tend to occur in the same documents. In other words, the vectors tend to have approximate values. For instance, it is natural that "aeroplane" tends to occur in the same set of documents as "aeroport".. Tf-Idf [2] does something similar, so these vectors are mostly a data representation of single words. As for my research, the basis of it was to be able to quantify how "specific" a word or multiword is. The idea is that the more a word/multiword occurs in different contexts (in my case, near different words) the less specific it is. Being able to identify the specificity of words allowed me to know which words/multiwords were probably concepts and which ones weren't. Then, with these concepts I was able to use Tf-Idf to extract document keywords and later I was able to infer broad relationships between concepts (like in if two words tend to co-occur in the same documents and same parts of documents, they are somewhat related). So, how does word2vec compare to my research? - I prefer simple statistical approaches like the one I did (which is just counting frequencies and dividing numbers, etc.) than the black-box approach of neural networks. I can tell you why my research provides good results (explained above) but if you read a follow-up paper on word2vec [3] this is what you can find in section 4: "Why does this produce good word representations? Good question. We don’t really know". - Also, bag-of-box approaches like word2vec and LSA don't tend to represent multiwords (such as "President of the United States of America", which leaves out lots of information on texts. - Finally, I think there's more possible applications after you know which words/multiwords are informative. With word2vec and LSA you are only representing something similar to frequencies of words in documents, which mostly allows for Information Retrieval-like applications. On the other hand, one of the applications I did with the extracted concepts was to implement a prototype which allowed us to find the definition of a concept on the corpus. The idea was that the definition of a concept was the paragraph where that concept occurred many times but also used other concepts to help the first one to be defined. I don't know how you could do something like that using something like word2vec. But as I said above, I am not that knowledgeable about word2vec, so I may miss something.. [0] - https://en.wikipedia.org/wiki/Word2vec https://en.wikipedia.org/wiki/Word2vec [1] - https://arxiv.org/abs/1301.3781 https://arxiv.org/abs/1301.3781 [2] - https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://en.wikipedia.org/wiki/Tf%E2%80%93idf [3] - https://arxiv.org/pdf/1402.3722.pdf https://arxiv.org/pdf/1402.3722.pdf

paduc 9y ago

Hey Joao, I've been using your WikiCorpusExtractor to work on the wiki dumps for years now for my NLP research. It has provided tremendous value to me. Thank you very much for your work and dedication !