If you go to their first tutorial, "Tutorial 1: Basics", you will see this comment in the code: "# Make a sentence object by passing a whitespace tokenized stri

If you go to their first tutorial, "Tutorial 1: Basics", you will see this comment in the code: "# Make a sentence object by passing a whitespace tokenized string" In that simple example you posted, they already did the tokenization manually, as it's pretty trivial. But yes, in many cases, you have preprocessors that do the tokenization. In some libraries, you actually have a class/object_type for tokens, but it's pretty common to just preprocess and take every space as a token separator. In some contexts and cases, it's possible to see tokens like "social_network", where multiple words are considered a single token. In that first tutorial, they also mention they have a tokenizer if you need it: "In some use cases, you might not have your text already tokenized. For this case, we added a simple tokenizer using the lightweight segtok library." So for your example you would simply run the tokenizer first, then the named entity recognition. EDIT: apparently you can do this directly: "sentence = Sentence('The grass is green.', use_tokenizer=True)"