4 ms·
>> NOT an LLM. its a small ar transforme Super cool project! Though, aren't most modern LLM's ar transformers internally?
by pj_mukh 2mo ago
>> NOT an LLM. its a small ar transforme
Super cool project! Though, aren't most modern LLM's ar transformers internally?
- dnautics 2mo agoIt's an slm
- bkaae 2mo agoThere is no language in the training of this, so there is no l.
- _ache_ 2mo agoIn computer science, that is technically a language. A formal language if you want to look it up on Wikipedia.
- FridgeSeal 2mo agoTechnically correct, but not in a functionally useful way. The “L” in LLM’s generally refers to human-language specifically. You’d expect to feed it…human text. Nitpicking that the human text also constitutes a mathematical language is like, correct, but so general as to be unhelpful.
- dfdydx 2mo agoNot all transformers are _language_ models - the sequences of tokens don't have to be sequences of words.
- perfmode 2mo agoIn this case, what are the tokens?
- evilmathkid 2mo ago9 color tokens + 4 special tokens (start, end, newline, inp_out_sep)
- shsshs 2mo ago[flagged]
- Forgeties79 2mo agoThis is incredibly pedantic if you ask me.
- deleted 2mo ago[deleted]
- kgwgk 2mo ago> The distinction is meaningless IMO. Not when discussing whether it's an LLM. The second L in LLM does have a meaning.
- shsshs 2mo ago[flagged]
- kgwgk 2mo agoYou're the first! Congratulations.
- fc417fc802 2mo agoI'd argue that meaningful sequences of symbols constitute a language. This example doesn't use a human language but it does use a language IMO (at least AFAIU).
- dfdydx 2mo agoAre number sequences a language?
- fc417fc802 2mo agoAre rectangles squares? I can express a passage written in english as a sequence of numbers (ie tokens).
- m4nu3l 1mo agoAlso, not all language models are transformers. You can have language models based on diffusion models or state-space models, or any other model that can be used to model sequences (so all of them, as sequences are just functions). Transformers are just the ones that are most common and successful today.
- HNDevsSuck2 2mo ago[dead]
- dpoloncsak 2mo agoPretty sure it's one of those "All squares are rectangles but not all rectangles are squares" situations. Transformers are what really started the LLM Boom, and seem to be crucial to the technology. They also have other applications, such as what OP created
- calebkaiser 2mo agoIt gets extremely blurry, because people commonly refer to any model that uses a component associated with the Transformer architecture as a Transformer (i.e. using some kind of QKV-esque attention mechanism). I think it's easier to think of it like this: A large language model is just what it says--a very large statistical model trained for language tasks. This covers the spectrum of GPT-style models, but also those hard to classify ones, like Liquid's "Liquid Foundation Models", which can get up to 24 billion parameters and use grouped query attention, but are closely related to state-space models as well: https://huggingface.co/LiquidAI/LFM2-24B-A2B https://huggingface.co/LiquidAI/LFM2-24B-A2B Also, as others have pointed out, a Transformer isn't inherently a language model. So really they're sort of two different axes, one classifying the model size and task, the other referring to a specific architecture.
- porridgeraisin 2mo agoIn my unpopular opinion, it wasn't transformers or attention, but pretraining on language data that kicked off the LLM Boom. Alec radford in his little jupyter notebook trained a very small non-transformer to predict simply the next-character on Amazon reviews. He noticed emergence of a neuron which when toggled controlled the sentiment of the remaining text. This is published as the sentiment neuron paper. Till then, transformers were being used primarily for stuff like translation and such and no one was even pretraining at scale, even tho transformers and attention existed. Openai and google if you count T5 persisting with pretrained generative models was what led to the LLM boom. Yes they used transformers, but that's just one IMO minor aspect.
- razodactyl 1mo agoWasn't it an LSTM neuron that changed sign and magnitude over a stretch of text? I remember seeing color coding against the current sentiment which was neat.