4 ms·
Nice work! > If it can do this in 13kb, it makes me wonder what it could do with more bytes. Maybe I misunderstand, but is this not just the first baby steps
by underscoring 3y ago
Nice work!
> If it can do this in 13kb, it makes me wonder what it could do with more bytes.
Maybe I misunderstand, but is this not just the first baby steps of an LLM written in JS? "what it could do with more bytes" is surely "GPT2 in javascript"?
- refulgentis 3y agoAuthor had LLM help them make a tree of words, and the algo choose which node we're at and offers children as completions. It's clever and cute but, not even close to an LLM.
- cchance 3y agoI mean it's not far off from a super low quant LLM with limited params, like a 1bit quant LLM with low params XD
- refulgentis 3y agoIt's very far off, like "not even wrong" in the Pauli sense of the phrase. There's a lot of abstractions one can have for this stuff, I think you're looking at that "text predictor" is one of them? If you roll with that, then you're in a position where you're saying GPT-2 class LLMs were very close in 1960, because at the end of the day, it's just a dictionary lookup with a string key and a value of list<string> completions. That confuses instead of illuminates.
- Legend2440 3y agoThe trouble with decision trees for language modeling is that they overfit really hard. They don't do the magical generalization that makes LLMs interesting.