4 ms·

What does tokenization (of strings, I guess) do?

• wyldfire 7y ago
  The README [1] shows a great example: The sentence "Hello, y'all! How are you ?" is tokenized into words. Those words are then encoded into integers representative of the words' identity in the model's dictionary. >>> output = tokenizer.encode("Hello, y'all! How are you ?") Encoding(num_tokens=13, attributes=[ids, type_ids, tokens, offsets, attention_mask, special_tokens_mask, overflowing, original_str, normalized_str]) >>> print(output.ids, output.tokens, output.offsets) [101, 7592, 1010, 1061, 1005, 2035, 999, 2129, 2024, 2017, 100, 1029, 102] ['[CLS]', 'hello', ',', 'y', "'", 'all', '!', 'how', 'are', 'you', '[UNK]', '?', '[SEP]'] [(0, 0), (0, 5), (5, 6), (7, 8), (8, 9), (9, 12), (12, 13), (14, 17), (18, 21), (22, 25), (26, 27), (28, 29), (0, 0)] But there's also good detail in the source [2] which says, "A Tokenizer works as a pipeline, it processes some raw text as input and outputs an Encoding. The various steps of the pipeline are: ...." [1] https://github.com/huggingface/tokenizers#quick-examples-using-python https://github.com/huggingface/tokenizers#quick-examples-usi... [2] https://github.com/huggingface/tokenizers/tree/master/tokenizers#what-is-a-tokenizer https://github.com/huggingface/tokenizers/tree/master/tokeni...