3 ms·
Summary of the Tokenizers
- deleted 4y ago[deleted]
- h2odragon 4y agoOne might also do a "stupid" tokenizer [1], live with the large number of variant forms of tokens for a while, and then use statistical methods to find and equate the synonyms. Putting much language knowledge into the tokenizer means the results are indelibly stamped with those assumptions. Similar to networking "dumb vs. smart pipe" arguments. This is another example where its better to save the smarts til later. [1] "anything between runs of whitespace" forex.