5 ms·

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies. Why is 'London in UK' "known" but 'London in France' isnt? Just because 'UK' much more frequently occurs in the dataset. The algorithm isnt doing anything other than aligning computation to hardware; the computation isnt doing anything interesting. The value comes from the conditional probability structure in the data. -- that comes from people arranging words usefully, because they're communicating information with one another

sirsinsalot 3y ago
It isn't some kind of Markov chain situation. Attention cross-links the abstract meaning of words, subtle implications based on context and so on. So, "mat" follows "the cat sat on the" where we understand the entire worldview of the dataset used for training; not just the next-word probability based on one or more previous words ... it's based on all previous meaning probability, and those meaning probablility and so on.
fellendrone 3y ago
> Why does, 'mat' follow from 'the cat sat on the ...' You're confidently incorrect by oversimplifying all LLMs to a base model performing a completion from a trivial context of 5 words. This is tantamount to a straw man. Not only do few people use untuned base models, it completely ignores in-context learning that allows the model to build complex semantic structures from the relationships learnt from its training data. Unlike base models, instruct and chat fine-tuning teaches models to 'reason' (or rather, perform semantic calculations in abstract latent spaces) with their "conditional probability structure", as you call it, to varying extents. The model must learn to use its 'facts', understand semantics, and perform abstractions in order to follow arbitrary instructions. You're also confabulating the training metric of "predicting tokens" with the mechanisms required to satisfy this metric for complex instructions. It's like saying "animals are just performing survival of the fittest". While technically correct, complex behaviours evolve to satisfy this 'survival' metric. You could argue they're "just stitching together phrases", but then you would be varying degrees of wrong: For one, this assumes phrases are compressed into semantically addressable units, which is already a form of abstraction ripe for allowing reasoning beyond 'stochastic parroting'. For two, it's well known that the first layers perform basic structural analysis such as grammar, and later layers perform increasing levels of abstract processing. For three, it shows a lack of understanding in how transformers perform semantic computation in-context from the relationships learnt by the feed-forward layers. If you're genuinely interested in understanding the computation model of transformers and how attention can perform semantic computation, take a look here: https://srush.github.io/raspy/ https://srush.github.io/raspy/ For a practical example of 'understanding' (to use the term loosely), give an instruct/chat tuned model the text of an article and ask it something like "What questions should this article answer, but doesn't?" This requires not just extracting phrases from a source, but understanding the context of the article on several levels, then reasoning about what the context is not asserting. Even comparatively simple 4x7B MoE models are able to do this effectively.
astrange 3y ago
LLMs don't work on words, they work on sequences of subword tokens. "It doesn't actually do anything" is a common explanation that's clearly a form of cope, because you can't even explain why it can form complete words, let alone complete sentences.
fspeech 3y ago
There are an infinite number of distributions that can fit the training data well (e.g., one that completely memorize the data and therefore replicate the frequencies). The trick is to find the distributions that generalize well, and here the NN architecture is critical.
raindear 3y ago
But why do transformers perform better than older language models including other neural language models.
nerdponx 3y ago
I think you're downplaying the importance of the attention/transformer architecture here. If it was "just" a matter of throwing compute at probabilities, then we wouldn't need any special architecture at all. P(next_word|previous_words) is ridiculously hard to estimate in a way that is actually useful. Remember how bad text generation used to be before GPT? There is innovation in discovering an architecture that makes it possible to learn P(next_word|previous_words), in addition to the computing techniques and hardware improvements required to make it work.
- JeremyNT 3y ago
  > There is innovation in discovering an architecture that makes it possible to learn P(next_word|previous_words), in addition to the computing techniques and hardware improvements required to make it work. Isn't that essentially what mjburgess said in the parent post? > LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data... The algorithm isnt doing anything other than aligning computation to hardware
  - nerdponx 3y ago
    Not really, and no. Torch and CUDA align computation to hardware. If it were just a matter of doing that, we would be fine with fully-connected MLP. And maybe that would work with orders of magnitude more data and compute than we currently throw at these models. But we are already pushing the cutting edge of those things to get useful results out of the specialized architecture. Choosing the right NN architecture is like feature engineering: the exact details don't matter that much, but getting the right overall structure can be the difference between learning a working model and failing to learn a working model, from the same source data with the same information content. Clearly our choice of inductive bias matters, and the transformer architecture is clearly an improvement over other designs. Surely you wouldn't argue that a CNN is "just" aligning computation to hardware, right? Transformers are clearly showing themselves as a reliably effective model architecture for text in the same way that CNNs are reliably effective for images.
    - 3y ago
seydor 3y ago
People specifically would like to know what the attention calculations add to this learning of the distribution
- ffwd 3y ago
  Just speculating but I think attention enables differentiation of semantic concepts for a word or sentence within a particular context. Like for any total set of training data you have a lesser number of semantic concepts (like let's say you have 10000 words, then it might contain 2000 semantic concepts, and those concepts are defined by the sentence structure and surrounding words, which is why they have a particular meaning), and then attention allows to differentiate those different contexts at different levels (words/etc). Also the fact you can do this attention at runtime/inference means you can generate the context from the prompt, which enables the flexibility of variable prompt/variable output but you lose the precision of giving an exact prompt and getting an exact answer
  - ffwd 3y ago
    I'm not one to whine about downvotes but I just have to say, it's a bad feeling when I can't even respond to the negative feedback because there is no accompanying comment. Did I misinterpret something? Did you? Who will ever know when there is no information. :L
forrestthewoods 3y ago
I find this take super weak sauce and shallow. This recent $10,000 challenge is super super interesting imho. https://twitter.com/VictorTaelin/status/1778100581837480178 https://twitter.com/VictorTaelin/status/1778100581837480178 State of the art models are doing more than “just” predicting the probability of the next symbol.
- mjburgess 3y ago
  You underestimate the properties of the sequential-conditional structure of human communication. Consider how a clever 6yo could fake being a physicist with access to a library of physics textbooks and a shredder. All the work is done for them. You'd need to be a physicist to spot them faking it. Of course, LLMs are in a much better position than having shredded physics textbooks -- they have shreddings of all books. So you actually have to try to expose this process, rather than just gullibly prompt using confirmation bias. It's trivial to show they work this way, both formally and practically. The issue is, practically, gullible people aren't trying.
  - astrange 3y ago
    You can program algorithms into transformer networks, up to the limit of how many computations you get. https://srush.github.io/raspy/ https://srush.github.io/raspy/ Are you going to do computer reductionism too and say computers can't do arithmetic, they just run electricity through silicon?
    - mjburgess 3y ago
      Sure, electronic computers largely don't implement arithmetic. The value of electronic computers derives from the fact they can "differentially activate" electronic devices (graphics cards -> lcd screens, etc.). If they werent electronic, they'd be basically useless -- since electricity is essentially the only "power transmission" force which we can reliably use. I do think much of the metaphorical language we use around these devices completely mystifies them. These abstractions we use fail all the time, and reveal themselves as deceptions. We ought, often, get back to reality. The reality of NNs, implemented on electronic devices, is that they can accept digitally encoded electronic signals and output likewise, where the variations in output signals are models of conditional probability structures This, as a an analogy for learning, for animals, for people, etc. is madness; provably so. It's a convenient pseudoscience, for an era obsessed with the power of electricity to power devices of automation, rather than steam. Were we in the steam age, The Brain would be hydaulic. The gullible in the intelligensia are always obsessed by the baubles and trinkets of elite appeal.
nextaccountic 3y ago
> Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies. What about cases that are not present in the dataset? The model must be doing something besides storing raw probabilities to avoid overfitting and enable generalization (imagine that you could have a very performant model - when it works - but it sometimes would spew "Invalid input, this was not in the dataset so I don't have a conditional probability and I will bail out")
albertzeyer 3y ago
You are more speaking about n-gram models here. NNs do far more than that. Or if you just want to say that NNs are used as a statistical model here: Well, yea, but that doesn't really tell you anything. Everything can be a statistical model. E.g., you could also say "this is exactly the way the human brain works", but it doesn't really tell you anything how it really works.
- mjburgess 3y ago
  My description is true of any statistical learning algorithm. The thing that people are looking to for answers, the NN itself, does not have them. That's like looking to Newton's compass to understand his general law of gravitation. The reason that LLMs trained on the internet and every ebook has the structure of human communication is because the dataset has that structure. Why does the data have that structure? this requires science, there is no explanation "in the compass". NNs are statistical models trained on data -- drawing analogies to animals is a mystification that causes people's ability to think clearly he to jump out the window. No one compares stock price models to the human brain; no banking regulator says, "well your volatility estimates were off because your machines had the wrong thoughts". This is pseudoscience. Animals are not statistical learning algorithms, so the reason that's uninformative is because it's false. Animals are in direct causal contact with the world and uncover its structure through interventional action and counterfactual reasoning. The structure of animal bodies, and the general learning strategies are well-known, and having nothing to do with LLMs/NNs. The reason that I know "The cup is in my hand" is not because P("The cup is in my hand"|HistoricalTexts) > P(not "The cup is in my hand"|HistoricalTexts)
  - Demlolomot 3y ago
    If learning in real life over 5-20 years shows the same result as a LLM being trained by billions of tokens, than yes it can be compared. And there are a lot of people out there who do not a lot of reasoning. After all optical illusions exist, our brain generalizes. The same thing happens with words like the riddle about the doctor operating on a child were we discover that the doctor is actually a female. And while llms only use text, we can already see how multimodal models become better, architecture gets better and hardware too.
michaelt 3y ago
That's not really an explanation that tells people all that much, though. I can explain that car engines 'just' convert gasoline into forward motion. But if a the person hearing the explanation is hoping to learn what a cam belt or a gearbox is, or why cars are more reliable now than they were in the 1970s, or what premium gas is for, or whether helicopter engines work on the same principle - they're going to need a more detailed explanation.
- mjburgess 3y ago
  It explains the LLM/NN. If you want to explain why it emits words in a certain order you need to explain how reality generated the dataset, ie., you need to explain how people communicate (and so on). There is no mystery why an NN trained on the night sky would generate nightsky-like photos; the mystery is why those photos have those patterns... solving that is called astrophysics. Why do people, in reasoning through physics problems, write symbols in a certain order? Well, explain physics, reasoning, mathematical notation, and so on. The ordering of the symbols gives rise to a certain utility of immitating that order -- but it isnt explained by that order. That's circular: "LLMs generate text in the order they do, because that's the order of the text they were given"
  - michaelt 3y ago
    That leaves loads of stuff unexplained. If the LLM is capable of rewording the MIT license into a set of hard-hitting rap battle lyrics, but the training dataset didn't contain any examples of anyone doing that, is the LLM therefore capable of producing output beyond the limits of its training data set? Is an LLM inherently constrained to mediocrity? If an LLM were writing a novel, does its design force it to produce cliche characters and predictable plotlines? If applied in science, are they inherently incapable of advancing the boundaries of human knowledge? Why transformers instead of, say, LSTMs? Must attention be multi-headed? Why can't the model have a simpler architecture, allowing such implementation details to emerge from the training data? Must they be so big that leading performance is only in the hands of multi-billion-dollar corporations? What's going on with language handling? Are facts learned in an abstract enough way that they can cross language barriers? Should a model produce different statements of fact when questioned in different languages? Does France need a French-language LLM? Is it reasonable to expect models to perform basic arithmetic accurately? What about summarising long documents? Why is it that I can ask questions with misspellings, but get answers with largely correct spelling? If misspellings were in the training data, why aren't they in the output? Does the cleverness that stops LLMs from learning misspellings from the training data also stop them from learning other common mistakes? If LLMs can be trained to be polite despite having examples of impoliteness in their training data, can they also be trained to not be racist, despite having examples of racism in their training data? Can a model learn a fact that is very rarely present in the training data - like an interesting result in an obscure academic paper? Or must a fact be widely known and oft-repeated in order to be learned? Merely saying "it predicts the next word" doesn't really explain much at all.
IanCal 3y ago
This is wrong, or at least a simplification to the point of removing any value. > NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). They are trained to maximise this, yes. > Their weights are a model of this distribution. That doesn't really follow, but let's leave that. > Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies. Here's the rub. If how you describe them is all they're doing then a sequence of never-before-seen words would have no valid response. All words would be equally likely. It would mean that a single brand new word would result in absolute gibberish following it as there's nothing to go on. Let's try: Input: I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now have. Result: You now have two kjsdhlisrnj. I would wager a solid amount that kjsdhlisrnj never appears in the input data. If it does pick another one, it doesn't matter. So we are learning something more general than the frequencies of sequences of tokens. I always end up pointing to this but OthelloGPT is very interesting https://thegradient.pub/othello/ https://thegradient.pub/othello/ While it's trained on sequences of moves, what it does is more than just "sequence a,b,c is followed by d most often"
- pas 3y ago
  how does it work underneath? "kjsdhlisrnj" is in the context, it gets tokenized, and now when the LLM is asked to predict/generate next-token sequences somehow "kjsdhlisrnj" is there too. it learns patterns. okay sure, they ger encoded somehow, but during infernce how does this lead to application of a recalled pattern on the right token(s)? also, can it invent new words?
  - empath-nirvana 3y ago
    Yes. This is one of the first things people did with chatgpt: can you help me invent a language for a fictional setting with talking frogs. Give me a sample sentence that frogs might say in their language, along with a translation. ChatGPT Creating a language for a fictional setting can be a fun and imaginative task! For your setting with talking frogs, we'll consider the environment and characteristics that might influence their language. Here’s a brief outline of the linguistic features and a sample sentence: Linguistic Features of the Frog Language Phonetics: Since frogs are known for their croaks and ribbits, their language might include a lot of guttural and clicking sounds, which can be represented in writing with characters like "k", "g", "q", and clicks like "!" or "ǂ". Syntax: We might imagine that frogs are direct and to the point, so their sentence structure could be subject-verb-object (SVO), similar to English, for straightforward communication. Vocabulary: Words would likely relate to their daily life—water, flies, lily pads, weather changes, predators, etc. Grammar: Simplified grammar with minimal use of tenses and conjugations to reflect an emphasis on the present moment, which is typical for many animalistic languages in fiction. Sample Language: Froglan Froglii (hello) Lok (water) Krik (eat) Zraak (fly - the insect) !quu (danger) ǂogg (jump) Sample Sentence in Froglan "Froglii! Lok krik zraak ǂogg." Translation "Hello! Jump eat fly water." This sentence, structured in a somewhat English syntax for ease of understanding, would literally mean that a frog is greeting another and then mentioning that it will jump towards water to eat flies. The sentence structure and vocabulary can be further refined and expanded based on how deeply you want to dive into the language creation!

sirsinsalot 3y ago

It isn't some kind of Markov chain situation. Attention cross-links the abstract meaning of words, subtle implications based on context and so on. So, "mat" follows "the cat sat on the" where we understand the entire worldview of the dataset used for training; not just the next-word probability based on one or more previous words ... it's based on all previous meaning probability, and those meaning probablility and so on.

fellendrone 3y ago

> Why does, 'mat' follow from 'the cat sat on the ...' You're confidently incorrect by oversimplifying all LLMs to a base model performing a completion from a trivial context of 5 words. This is tantamount to a straw man. Not only do few people use untuned base models, it completely ignores in-context learning that allows the model to build complex semantic structures from the relationships learnt from its training data. Unlike base models, instruct and chat fine-tuning teaches models to 'reason' (or rather, perform semantic calculations in abstract latent spaces) with their "conditional probability structure", as you call it, to varying extents. The model must learn to use its 'facts', understand semantics, and perform abstractions in order to follow arbitrary instructions. You're also confabulating the training metric of "predicting tokens" with the mechanisms required to satisfy this metric for complex instructions. It's like saying "animals are just performing survival of the fittest". While technically correct, complex behaviours evolve to satisfy this 'survival' metric. You could argue they're "just stitching together phrases", but then you would be varying degrees of wrong: For one, this assumes phrases are compressed into semantically addressable units, which is already a form of abstraction ripe for allowing reasoning beyond 'stochastic parroting'. For two, it's well known that the first layers perform basic structural analysis such as grammar, and later layers perform increasing levels of abstract processing. For three, it shows a lack of understanding in how transformers perform semantic computation in-context from the relationships learnt by the feed-forward layers. If you're genuinely interested in understanding the computation model of transformers and how attention can perform semantic computation, take a look here: https://srush.github.io/raspy/ https://srush.github.io/raspy/ For a practical example of 'understanding' (to use the term loosely), give an instruct/chat tuned model the text of an article and ask it something like "What questions should this article answer, but doesn't?" This requires not just extracting phrases from a source, but understanding the context of the article on several levels, then reasoning about what the context is not asserting. Even comparatively simple 4x7B MoE models are able to do this effectively.

astrange 3y ago

LLMs don't work on words, they work on sequences of subword tokens. "It doesn't actually do anything" is a common explanation that's clearly a form of cope, because you can't even explain why it can form complete words, let alone complete sentences.

fspeech 3y ago

There are an infinite number of distributions that can fit the training data well (e.g., one that completely memorize the data and therefore replicate the frequencies). The trick is to find the distributions that generalize well, and here the NN architecture is critical.

raindear 3y ago

But why do transformers perform better than older language models including other neural language models.

nerdponx 3y ago

I think you're downplaying the importance of the attention/transformer architecture here. If it was "just" a matter of throwing compute at probabilities, then we wouldn't need any special architecture at all. P(next_word|previous_words) is ridiculously hard to estimate in a way that is actually useful. Remember how bad text generation used to be before GPT? There is innovation in discovering an architecture that makes it possible to learn P(next_word|previous_words), in addition to the computing techniques and hardware improvements required to make it work.

JeremyNT 3y ago
> There is innovation in discovering an architecture that makes it possible to learn P(next_word|previous_words), in addition to the computing techniques and hardware improvements required to make it work. Isn't that essentially what mjburgess said in the parent post? > LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data... The algorithm isnt doing anything other than aligning computation to hardware
- nerdponx 3y ago
  Not really, and no. Torch and CUDA align computation to hardware. If it were just a matter of doing that, we would be fine with fully-connected MLP. And maybe that would work with orders of magnitude more data and compute than we currently throw at these models. But we are already pushing the cutting edge of those things to get useful results out of the specialized architecture. Choosing the right NN architecture is like feature engineering: the exact details don't matter that much, but getting the right overall structure can be the difference between learning a working model and failing to learn a working model, from the same source data with the same information content. Clearly our choice of inductive bias matters, and the transformer architecture is clearly an improvement over other designs. Surely you wouldn't argue that a CNN is "just" aligning computation to hardware, right? Transformers are clearly showing themselves as a reliably effective model architecture for text in the same way that CNNs are reliably effective for images.
  - 3y ago

seydor 3y ago

People specifically would like to know what the attention calculations add to this learning of the distribution

ffwd 3y ago
Just speculating but I think attention enables differentiation of semantic concepts for a word or sentence within a particular context. Like for any total set of training data you have a lesser number of semantic concepts (like let's say you have 10000 words, then it might contain 2000 semantic concepts, and those concepts are defined by the sentence structure and surrounding words, which is why they have a particular meaning), and then attention allows to differentiate those different contexts at different levels (words/etc). Also the fact you can do this attention at runtime/inference means you can generate the context from the prompt, which enables the flexibility of variable prompt/variable output but you lose the precision of giving an exact prompt and getting an exact answer
- ffwd 3y ago
  I'm not one to whine about downvotes but I just have to say, it's a bad feeling when I can't even respond to the negative feedback because there is no accompanying comment. Did I misinterpret something? Did you? Who will ever know when there is no information. :L

forrestthewoods 3y ago

I find this take super weak sauce and shallow. This recent $10,000 challenge is super super interesting imho. https://twitter.com/VictorTaelin/status/1778100581837480178 https://twitter.com/VictorTaelin/status/1778100581837480178 State of the art models are doing more than “just” predicting the probability of the next symbol.

mjburgess 3y ago
You underestimate the properties of the sequential-conditional structure of human communication. Consider how a clever 6yo could fake being a physicist with access to a library of physics textbooks and a shredder. All the work is done for them. You'd need to be a physicist to spot them faking it. Of course, LLMs are in a much better position than having shredded physics textbooks -- they have shreddings of all books. So you actually have to try to expose this process, rather than just gullibly prompt using confirmation bias. It's trivial to show they work this way, both formally and practically. The issue is, practically, gullible people aren't trying.
- astrange 3y ago
  You can program algorithms into transformer networks, up to the limit of how many computations you get. https://srush.github.io/raspy/ https://srush.github.io/raspy/ Are you going to do computer reductionism too and say computers can't do arithmetic, they just run electricity through silicon?
  - mjburgess 3y ago
    Sure, electronic computers largely don't implement arithmetic. The value of electronic computers derives from the fact they can "differentially activate" electronic devices (graphics cards -> lcd screens, etc.). If they werent electronic, they'd be basically useless -- since electricity is essentially the only "power transmission" force which we can reliably use. I do think much of the metaphorical language we use around these devices completely mystifies them. These abstractions we use fail all the time, and reveal themselves as deceptions. We ought, often, get back to reality. The reality of NNs, implemented on electronic devices, is that they can accept digitally encoded electronic signals and output likewise, where the variations in output signals are models of conditional probability structures This, as a an analogy for learning, for animals, for people, etc. is madness; provably so. It's a convenient pseudoscience, for an era obsessed with the power of electricity to power devices of automation, rather than steam. Were we in the steam age, The Brain would be hydaulic. The gullible in the intelligensia are always obsessed by the baubles and trinkets of elite appeal.

nextaccountic 3y ago

> Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies. What about cases that are not present in the dataset? The model must be doing something besides storing raw probabilities to avoid overfitting and enable generalization (imagine that you could have a very performant model - when it works - but it sometimes would spew "Invalid input, this was not in the dataset so I don't have a conditional probability and I will bail out")

albertzeyer 3y ago

You are more speaking about n-gram models here. NNs do far more than that. Or if you just want to say that NNs are used as a statistical model here: Well, yea, but that doesn't really tell you anything. Everything can be a statistical model. E.g., you could also say "this is exactly the way the human brain works", but it doesn't really tell you anything how it really works.

mjburgess 3y ago
My description is true of any statistical learning algorithm. The thing that people are looking to for answers, the NN itself, does not have them. That's like looking to Newton's compass to understand his general law of gravitation. The reason that LLMs trained on the internet and every ebook has the structure of human communication is because the dataset has that structure. Why does the data have that structure? this requires science, there is no explanation "in the compass". NNs are statistical models trained on data -- drawing analogies to animals is a mystification that causes people's ability to think clearly he to jump out the window. No one compares stock price models to the human brain; no banking regulator says, "well your volatility estimates were off because your machines had the wrong thoughts". This is pseudoscience. Animals are not statistical learning algorithms, so the reason that's uninformative is because it's false. Animals are in direct causal contact with the world and uncover its structure through interventional action and counterfactual reasoning. The structure of animal bodies, and the general learning strategies are well-known, and having nothing to do with LLMs/NNs. The reason that I know "The cup is in my hand" is not because P("The cup is in my hand"|HistoricalTexts) > P(not "The cup is in my hand"|HistoricalTexts)
- Demlolomot 3y ago
  If learning in real life over 5-20 years shows the same result as a LLM being trained by billions of tokens, than yes it can be compared. And there are a lot of people out there who do not a lot of reasoning. After all optical illusions exist, our brain generalizes. The same thing happens with words like the riddle about the doctor operating on a child were we discover that the doctor is actually a female. And while llms only use text, we can already see how multimodal models become better, architecture gets better and hardware too.

michaelt 3y ago

That's not really an explanation that tells people all that much, though. I can explain that car engines 'just' convert gasoline into forward motion. But if a the person hearing the explanation is hoping to learn what a cam belt or a gearbox is, or why cars are more reliable now than they were in the 1970s, or what premium gas is for, or whether helicopter engines work on the same principle - they're going to need a more detailed explanation.

mjburgess 3y ago
It explains the LLM/NN. If you want to explain why it emits words in a certain order you need to explain how reality generated the dataset, ie., you need to explain how people communicate (and so on). There is no mystery why an NN trained on the night sky would generate nightsky-like photos; the mystery is why those photos have those patterns... solving that is called astrophysics. Why do people, in reasoning through physics problems, write symbols in a certain order? Well, explain physics, reasoning, mathematical notation, and so on. The ordering of the symbols gives rise to a certain utility of immitating that order -- but it isnt explained by that order. That's circular: "LLMs generate text in the order they do, because that's the order of the text they were given"
- michaelt 3y ago
  That leaves loads of stuff unexplained. If the LLM is capable of rewording the MIT license into a set of hard-hitting rap battle lyrics, but the training dataset didn't contain any examples of anyone doing that, is the LLM therefore capable of producing output beyond the limits of its training data set? Is an LLM inherently constrained to mediocrity? If an LLM were writing a novel, does its design force it to produce cliche characters and predictable plotlines? If applied in science, are they inherently incapable of advancing the boundaries of human knowledge? Why transformers instead of, say, LSTMs? Must attention be multi-headed? Why can't the model have a simpler architecture, allowing such implementation details to emerge from the training data? Must they be so big that leading performance is only in the hands of multi-billion-dollar corporations? What's going on with language handling? Are facts learned in an abstract enough way that they can cross language barriers? Should a model produce different statements of fact when questioned in different languages? Does France need a French-language LLM? Is it reasonable to expect models to perform basic arithmetic accurately? What about summarising long documents? Why is it that I can ask questions with misspellings, but get answers with largely correct spelling? If misspellings were in the training data, why aren't they in the output? Does the cleverness that stops LLMs from learning misspellings from the training data also stop them from learning other common mistakes? If LLMs can be trained to be polite despite having examples of impoliteness in their training data, can they also be trained to not be racist, despite having examples of racism in their training data? Can a model learn a fact that is very rarely present in the training data - like an interesting result in an obscure academic paper? Or must a fact be widely known and oft-repeated in order to be learned? Merely saying "it predicts the next word" doesn't really explain much at all.

IanCal 3y ago

This is wrong, or at least a simplification to the point of removing any value. > NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). They are trained to maximise this, yes. > Their weights are a model of this distribution. That doesn't really follow, but let's leave that. > Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model of those frequencies. Here's the rub. If how you describe them is all they're doing then a sequence of never-before-seen words would have no valid response. All words would be equally likely. It would mean that a single brand new word would result in absolute gibberish following it as there's nothing to go on. Let's try: Input: I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now have. Result: You now have two kjsdhlisrnj. I would wager a solid amount that kjsdhlisrnj never appears in the input data. If it does pick another one, it doesn't matter. So we are learning something more general than the frequencies of sequences of tokens. I always end up pointing to this but OthelloGPT is very interesting https://thegradient.pub/othello/ https://thegradient.pub/othello/ While it's trained on sequences of moves, what it does is more than just "sequence a,b,c is followed by d most often"

pas 3y ago
how does it work underneath? "kjsdhlisrnj" is in the context, it gets tokenized, and now when the LLM is asked to predict/generate next-token sequences somehow "kjsdhlisrnj" is there too. it learns patterns. okay sure, they ger encoded somehow, but during infernce how does this lead to application of a recalled pattern on the right token(s)? also, can it invent new words?
- empath-nirvana 3y ago
  Yes. This is one of the first things people did with chatgpt: can you help me invent a language for a fictional setting with talking frogs. Give me a sample sentence that frogs might say in their language, along with a translation. ChatGPT Creating a language for a fictional setting can be a fun and imaginative task! For your setting with talking frogs, we'll consider the environment and characteristics that might influence their language. Here’s a brief outline of the linguistic features and a sample sentence: Linguistic Features of the Frog Language Phonetics: Since frogs are known for their croaks and ribbits, their language might include a lot of guttural and clicking sounds, which can be represented in writing with characters like "k", "g", "q", and clicks like "!" or "ǂ". Syntax: We might imagine that frogs are direct and to the point, so their sentence structure could be subject-verb-object (SVO), similar to English, for straightforward communication. Vocabulary: Words would likely relate to their daily life—water, flies, lily pads, weather changes, predators, etc. Grammar: Simplified grammar with minimal use of tenses and conjugations to reflect an emphasis on the present moment, which is typical for many animalistic languages in fiction. Sample Language: Froglan Froglii (hello) Lok (water) Krik (eat) Zraak (fly - the insect) !quu (danger) ǂogg (jump) Sample Sentence in Froglan "Froglii! Lok krik zraak ǂogg." Translation "Hello! Jump eat fly water." This sentence, structured in a somewhat English syntax for ease of understanding, would literally mean that a frog is greeting another and then mentioning that it will jump towards water to eat flies. The sentence structure and vocabulary can be further refined and expanded based on how deeply you want to dive into the language creation!