4 ms·
We used to do a lot more preprocessing to NLP, like stemming, removing stop words, or even adding grammar information (NP, VP, etc.). Now we just do basic token
by thomasahle 3y ago
We used to do a lot more preprocessing to NLP, like stemming, removing stop words, or even adding grammar information (NP, VP, etc.).
Now we just do basic tokenization. The rest turned out to be irrelevant or even counter productive.
- kbelder 3y agoBut also, that basic tokenization is essential; training it on a raw ascii stream would be much less efficient. There is a sweet spot of processing & abstraction that should be aimed for.