4 ms·
Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-traine
by rpedela 8y ago
Yes, definitely. I have also found sentence segmentation to be important. I recommend using Stanford CoreNLP for sentence segmentation if they have a pre-trained model for your language. In my experience, their pre-trained models are the best and NLTK is a good fallback. I think OpenNLP is the easiest to train a new model if CoreNLP and NLTK don't work.
For training, I wonder if NER can be abused to produce a good sentence segmentation model. My idea is label the first word of every sentence as NEW_SENT and every other word as nothing. Then the model would learn which words start a sentence in a document. I haven't tried it nor know if anyone else has, but I keep meaning to try.