4 ms·

Can you expand on emoji tokenization? I don't understand what you mean?

• pcsanwald 9y ago
  the lucene standard tokenizer doesn't offer support for tokenizing emojis: it strips them out. I'm not aware of any lucene tokenizer that does support emojis tokenization. This isn't a huge deal, as lucene is open source and tokenizers can be written, but I'm just highlighting it as a shortcoming that's illustrative of the time and place lucene evolved from.