4 ms·

What's a tokenizer?

• sbergot 4y ago
  It is a processing step in natural language processing: https://www.analyticsvidhya.com/blog/2020/05/what-is-tokenization-nlp/ https://www.analyticsvidhya.com/blog/2020/05/what-is-tokeniz...
• teruakohatu 4y ago
  Example: Split an input into tokens. Tokens: [split, an, input, into, tokens, .]
  • deelly 4y ago
    Example: umbrella Tokens: [umb, rella] ?
    • artifabrian 4y ago
      Could be as well, given there's many varieties of tokenisers, each with different pros and cons. This particular tokenizer is very interesting given that it tries to be best of both worlds (word-level tokenizer and character-level).