3 ms·2D Tokenization for Large Language Models1 points by bubble_boi 3y agobubble_boi 3y agoThis article is about how we process text before passing it to large language models, the problems with this approach, and an alternative solution.