4 ms·
Does anyone know why Mistral use a 17 bit (131k) vocabulary? I'm sure it's more efficient at encoding text but each token doesn't fit into a 16 bit register whi
by barbegal 2y ago
Does anyone know why Mistral use a 17 bit (131k) vocabulary? I'm sure it's more efficient at encoding text but each token doesn't fit into a 16 bit register which must make it more inefficient computationally?
- cpldcpu 2y agoThe tokens are immediately transformed into embeddings (very large vectors), so the 17 bit values are not used for any computation.