3 ms·
No? In his recent tutorial, Karpathy showed just how much complexity there is in the tokenizer. This technology has been years in the making with many small ad
by gessha 2y ago
No? In his recent tutorial, Karpathy showed just how much complexity there is in the tokenizer.
This technology has been years in the making with many small advances pushing the performance ever so slightly. There’s been theoretical and engineering advances that contributed to where we are today. And we need many more to get the technology to an actually usable level instead of the current word spaghetti that we get.
Also, the post is generally about neural networks and not just LMs.
When making design decisions about an ML system you shouldn’t just choose the attention hammer and hammer away. There’s a lot of design constraints you need to consider which is why I made the original reply.
- whimsicalism 2y agoAre there micro-optimizations that eke out small advancements? Yes, absolutely - the modern tokenizer is a good example of that. Is the core of the technology that complex? No. You could get very far with a naive tokenizer that just tokenized by words and replaced unknown words with <unk>. This is extremely simple to implement and I've trained transformers like this. It (of course) makes a perplexity difference but the core of the technology is not changed and is quite simple. Most of the complexity is in the hardware, not the software innovations. > And we need many more to get the technology to an actually usable level instead of the current word spaghetti that we get. I think the current technology is useable. > you shouldn’t just choose the attention hammer and hammer away It's a good first choice of hammer, tbph.