3 ms·

Can someone explain what the expected behaviour is with punctuation and Named Entity Recognition? Is there an assumption that punctuation is preprocessed in som

Can someone explain what the expected behaviour is with punctuation and Named Entity Recognition? Is there an assumption that punctuation is preprocessed in some form? I'm a noob but it's not what I expect - periods change what is extracted in inconsistent ways. e.g. "I love Berlin." -> "Berlin." "I love Berlin ." -> "Berlin" "George Washington loves Berlin." -> "George Washington" "George Washington loves Berlin ." -> ["George Washington", "Berlin"]

slx26 8y ago
If you go to their first tutorial, "Tutorial 1: Basics", you will see this comment in the code: "# Make a sentence object by passing a whitespace tokenized string" In that simple example you posted, they already did the tokenization manually, as it's pretty trivial. But yes, in many cases, you have preprocessors that do the tokenization. In some libraries, you actually have a class/object_type for tokens, but it's pretty common to just preprocess and take every space as a token separator. In some contexts and cases, it's possible to see tokens like "social_network", where multiple words are considered a single token. In that first tutorial, they also mention they have a tokenizer if you need it: "In some use cases, you might not have your text already tokenized. For this case, we added a simple tokenizer using the lightweight segtok library." So for your example you would simply run the tokenizer first, then the named entity recognition. EDIT: apparently you can do this directly: "sentence = Sentence('The grass is green.', use_tokenizer=True)"
- duncanawoods 8y ago
  Thanks. I assumed it was French style punctuation with a space before the period rather than space tokenization!
  - yorwba 8y ago
    French doesn't put spaces before single-part punctuation like periods or commas.
cjg_ 8y ago
It mentions in the tutorial that you can input space tokenized sentences directly or have it tokenize it for you, see https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_1_BASICS.md#tokenization https://github.com/zalandoresearch/flair/blob/master/resourc....
- duncanawoods 8y ago
  Ah yes, thanks!

slx26 8y ago

If you go to their first tutorial, "Tutorial 1: Basics", you will see this comment in the code: "# Make a sentence object by passing a whitespace tokenized string" In that simple example you posted, they already did the tokenization manually, as it's pretty trivial. But yes, in many cases, you have preprocessors that do the tokenization. In some libraries, you actually have a class/object_type for tokens, but it's pretty common to just preprocess and take every space as a token separator. In some contexts and cases, it's possible to see tokens like "social_network", where multiple words are considered a single token. In that first tutorial, they also mention they have a tokenizer if you need it: "In some use cases, you might not have your text already tokenized. For this case, we added a simple tokenizer using the lightweight segtok library." So for your example you would simply run the tokenizer first, then the named entity recognition. EDIT: apparently you can do this directly: "sentence = Sentence('The grass is green.', use_tokenizer=True)"

duncanawoods 8y ago
Thanks. I assumed it was French style punctuation with a space before the period rather than space tokenization!
- yorwba 8y ago
  French doesn't put spaces before single-part punctuation like periods or commas.

cjg_ 8y ago

It mentions in the tutorial that you can input space tokenized sentences directly or have it tokenize it for you, see https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_1_BASICS.md#tokenization https://github.com/zalandoresearch/flair/blob/master/resourc....

duncanawoods 8y ago
Ah yes, thanks!