4 ms·
> language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful compon
by espadrine 4y ago
> language transformers don’t operate by chunking text into arbitrary clusters of a fixed number of characters - they tokenize on semantically meaningful components extracted from the text (I.e words - and sometimes then word components)
Don’t they? Most use BPE, which yields arbitrary but common sequence of characters that often are just syllables. Additionally, many use SentencePiece, which randomizes the subdivisions, which sometimes produces single characters instead.
(That said, what you suggest would be an interesting research avenue.)
- jameshart 4y agoBPE isn’t arbitrary, though, is it? It’s based on essentially a pre trained markov model of how likely each byte is to be ‘part’ of the preceding token - ie it uses an existing semantic model for ‘what normal text looks like’ to choose potentially useful tokens (even for nonsensical text, which is why it works on stuff like code) And it does that typically after word splitting has already happened (although yes, looking at sentencepiece, maybe not always… interesting.) Was thinking that a similarly low level approach to tile extraction would be to grab tiles that represent areas of high detail - but then maybe that just amounts to applying some sort of compression to an image before processing…
- fxtentacle 4y agoIn this paper https://arxiv.org/abs/2206.12693 https://arxiv.org/abs/2206.12693 I show that choosing the text snippets deliberately so that your choices minimize overall entropy will also minimize result entropy, meaning it'll increase the accuracy of your speech recognition system (for German speech). In my case, the entropy-optimized tokenization also matched conserved features of the German language.