3 ms·
I would suggest to start simple and manually to get some feeling for the problems in the field. No frameworks, no tools, just you and Python! Do a simple exper
by jventura 10y ago
I would suggest to start simple and manually to get some feeling for the problems in the field. No frameworks, no tools, just you and Python!
Do a simple experiment: get some texts, split words between spaces (e.g line.split(" ")) and use a dict to count the frequency of the words. Sort the words by frequency, look at them, and you will eventually reach the same conclusion as in figure 1 of the paper by Luhn when working for IBM in 1958 (http://courses.ischool.berkeley.edu/i256/f06/papers/luhn58.pdf http://courses.ischool.berkeley.edu/i256/f06/papers/luhn58.p...)
There are lots of corpora out there in the wild, but if you need to roll your own from wikipedia texts you can use this tool I did: https://github.com/joaoventura/WikiCorpusExtractor https://github.com/joaoventura/WikiCorpusExtractor
From this experiment, and depending if you like statistics or not, you can play a bit with the numbers. For instance, you can use Tf-Idf (https://en.wikipedia.org/wiki/Tf%E2%80%93idf https://en.wikipedia.org/wiki/Tf%E2%80%93idf) to extract potential keywords from documents. Check the formula, it only uses the frequency of occurrence of words in documents.
Only use tools such as Deep neural networks if you decide later that they are essential for what you need. I did an entire PhD on this area just with Python and playing with frequencies, no frameworks at all (an eg. of my work can be found at http://www.sciencedirect.com/science/article/pii/S1877050912001251 http://www.sciencedirect.com/science/article/pii/S1877050912...).
Good luck!
- navyad 10y agothanks for input. right now i am playing with built-in corpus and text of NLTK library.
- arvinsim 10y agoThanks for posting this. Good to know that using just vanilla Python is as viable for learning as using specialized frameworks.
- jventura 10y agoOh, definitely you don't need much more than vanilla Python for playing around and test things. The only thing that you may need is a good tokenizer (i.e., to know where you should break the words), but the regex in [0] was good enough for what I needed. I was working with texts for three different languages (EN, DE and PT), and that is the reason I recommend statistical approaches, as they tend to be language independent. [0] https://github.com/joaoventura/WikiCorpusExtractor/blob/master/tokenizer.py#L47 https://github.com/joaoventura/WikiCorpusExtractor/blob/mast...
- syllogism 10y agoYou might find these posts interesting: https://explosion.ai/blog/part-of-speech-pos-tagger-in-python https://explosion.ai/blog/part-of-speech-pos-tagger-in-pytho... https://explosion.ai/blog/parsing-english-in-python https://explosion.ai/blog/parsing-english-in-python These days I would say these articles are better indications of solving NLP problems with linear models -- tagging and parsing are less important than they used to be. Here's how I think about doing NLP with current neural network techniques: https://explosion.ai/blog/deep-learning-formula-nlp https://explosion.ai/blog/deep-learning-formula-nlp