3 ms·
As of now wink-tokenizer (https://github.com/winkjs/wink-tokenizer https://github.com/winkjs/wink-tokenizer) supports multiple scripts, therefore, it can also t
by sanjayaksaxena1 8y ago
As of now wink-tokenizer (https://github.com/winkjs/wink-tokenizer https://github.com/winkjs/wink-tokenizer) supports multiple scripts, therefore, it can also tokenize sentences in languages like Hindi, Marathi, French, German etc. We are working on extending multi-lingual support to other components including this lemmatizer.