3 ms·
> If we don't automatically pair "anti" or "ism" into a single token before handing it off to the model, then the attention heads on the lower layers in the mo
by amelius 2y ago
> If we don't automatically pair "anti" or "ism" into a single token before handing it off to the model, then the attention heads on the lower layers in the model have to do the same work.
What I mean is an extra neural network that comes before the input of the llm, which converts characters (or simple 1-hot vectors which correspond to charactes) into tokens (or whatever it is you would call the internal representation of the network). The advantage would be a more unified way of representing the llm, and I guess one downside would be that you'd get a lot of replication in the NN, but perhaps these parts can be merged (have shared weights).