3 ms·
Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user
by wolfgarbe 2y ago
Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.