33 ms·

I'm unable to get it to run either. Turning text into tokens is something I've wanted to be able to do in golang before, so it would be super nice if it was eas

I'm unable to get it to run either. Turning text into tokens is something I've wanted to be able to do in golang before, so it would be super nice if it was easy to install with `go get`. I still need to work through some of the other libraries here: http://biosphere.cc/software-engineering/go-machine-learning-nlp-libraries http://biosphere.cc/software-engineering/go-machine-learning... I agree though, seems promising.

mschoch 11y ago
If you just want to segment larger blocks of text into tokens you can try the segment library (it implements the word boundary portion of unicode annex 29): https://github.com/blevesearch/segment https://github.com/blevesearch/segment If you need more manipulation of tokens after segmentation/tokenization, you could look at the analysis sub-package of bleve. Its intended to be able to be used indepenently of the rest of the library. https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve