4 ms·
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp
by ftxbro 3y ago
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
- spywaregorilla 3y agoI don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.
- ftxbro 3y ago> "I don't think that is intuitive at all." That's exactly the point. Every intuition is always on the side of feature engineering.
- spywaregorilla 3y agoNot really.
- whimsicalism 3y agoYou would be wrong at the scales we are talking about. The whole point is that it is unintuitive.
- yorwba 3y agoIf you replace a tokenizer with 5 bytes per token on average by a byte-level representation, you now need 5 times as much memory and (depending on the specifics of the attention mechanism) 11 to 25 times as much compute. At the scales we're talking about, that's quite a hefty price to pay, and it doesn't even take into account that you might need more layers to replace the processing that was implicitly done by the tokenizer.
- sp332 3y agoMost current implementations can't count syllables at all, so it would get you at least that far.
- nsinreal 3y agoYes, but how much extra layers and computing power do you need? Of course, phonetic tokens are awkward idea, but there is a reason why word "human" is encoded as only one token.
- jcims 3y agoI think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not. To me it’s like changing the periodic table, at the macroscopic scale it may or may not make a difference.
- zecg 3y ago> I think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not. The crazy thing is it's already solved. YC should just spin a subreddit.ycombinator.com for each one, there's a nice search that works and some very nice apps for reading. What the reddit shareholders are about to buy is incredibly fragile and the management is fucking with it so much it's obvious they only care about the payola.
- Der_Einzige 3y agoIf we don't fix up issues caused by the tokenizers, than techniques which literally remove superfluous computation (i.e. through filters of the LLM probability distribution) are useful as a stop-gap. Switching to bytes is the ultimate fix, but for the interim, if you want reliable rhyming with an LLM, you need filter-assisted decoding: https://paperswithcode.com/paper/most-language-models-can-be-poets-too-an-ai https://paperswithcode.com/paper/most-language-models-can-be... and replicas post about this work: https://replicate.com/blog/turn-your-llm-into-a-poet https://replicate.com/blog/turn-your-llm-into-a-poet