2 ms·
An interesting read on alternative tokenization methods. Questions: 1. What's the goal of entropy based byte token grouping as tokenization? Is this tokenizat
by RandyOrion 2y ago
An interesting read on alternative tokenization methods.
Questions:
1. What's the goal of entropy based byte token grouping as tokenization? Is this tokenization method best suited for the goal?
2. What about simply using byte level sequence to sequence autoencoder with down sampling for tokenization?