18 ms·

It doesn’t have a grasp of bridging at all. It tagged a few of your terms as important based on its training data, then did a weighted search for words to respo

It doesn’t have a grasp of bridging at all. It tagged a few of your terms as important based on its training data, then did a weighted search for words to respond, then as it built the response, continued applying importance-weighting to its “own words” as it did further searches, to avoid writing gibberish or going off some weird direction. It’s still not much more than Markov chains, just with some clever anti-nonsense filtering and importance-weighting. There’s no “understanding”, nor anything particularly close to it. It’s impressive we’ve accomplished so much with something that is so thoroughly, entirely stupid, in fact. They are useful tools, for sure.

jodrellblank 2y ago
> "It tagged a few of your terms as important based on its training data, then did a weighted search for words to respond, then as it built the response, continued applying importance-weighting to its “own words” as it did further searches, to avoid writing gibberish or going off some weird direction." what specifically would you or I do different, apart from having less training data? > "There’s no “understanding”, nor anything particularly close to it" any evidence for this claim? It explained, it responded in context to a followup question, it gave other relevant examples, by what measure does it "not understand" but I "do understand"?
- vundercind 2y ago
  I’ll be a lot more open to arguments that the technology described in the Transformer Paper becomes capable of things like understanding when you throw enough data and computing power at it when the state of the art moves beyond, “yeah, but don’t human brains just randomly guess at the right thing all the time?” I dunno, do they do more than that? Seems like it to me. Does a Chinese Room “understand”? I say no, but hey, maybe it does. If I laboriously do the math by hand, taking care never to actually know the informational content of any of the input or output myself, does my scratch paper understand your questions? If the output’s just as good as ChatGPT? Where’s the part that understands?
  - DougN7 2y ago
    The scratch paper is a fantastic analogy. I’m going to use it.
  - fragmede 2y ago
    If we open up a human skull, what would we find? A clump of grey matter, a wet piece of meat. As we dissect it further and zoomed in, what would we find? Neurons. Zoom in futher and we get a mess of chemicals. As we look at the pile of cells that make up the brain, we have the same fundamental question: where's the part that understands? Yet somehow, from the connections of all of those cells, and neurotransmitters, there's consciousness and something there that does understand (and think and reason and love). If, instead of LLM architecture, on more powerful computers than we have now, we simulated all of those neurons and their connections, would we have a computer that understands? If we then did those computations on scratch paper, where would the piece that understands be on that piece of paper? The sum of a thing's parts can be greater than the individual parts. Whether or not ChatGPT understands is a whole big question, but we'll have no more luck dissecting LLMs to find out if it does than if we dissected a human brain.
    - vundercind 2y ago
      You can simulate a brain to some degree, maybe, but you won’t get the whole thing short of just… using an actual brain, that is physically identical to the one you’re simulating, down to at least the quantum level (and who knows what else?). It’s not at all certain that this can be reduced to math. The calculations for operating an LLM definitely can be reduced to math. No reduction needed, in fact—they are math. This isn’t an argument (to my mind, anyway) against even the possibility of machine whatever-you-like (consciousness, understanding, whatever) but against the idea of equivalence because we could simulate either one—in fact, we can’t simulate one. The other, essentially is already simulation, no further steps needed. What you’re getting at (if I may attempt to present your argument) is that we could reduce either to its components and make it look ridiculous that it might be doing anything particularly advanced. However, in fact we definitely can reproduce exactly what one of them does with a bunch of thick books of lookup tables and some formulas that we could mechanically follow by hand, and it might even be possible to do so in practice, not just hypothetically (at significant, but not impossible, expense) while we do not know we can do that for a human brain, short of just using exactly the brain that we want to “simulate”.
- consteval 2y ago
  > what specifically would you or I do different, apart from having less training data? Create things that are not in the training data. I could coin a new phrase that uses "bridge", I doubt an LLM can or would have any motivation to. Because LLMs understand data and patterns, but they don't understand the meaning behind them. They can't be imaginative. For example, an LLM can recite Shakespeare and can even write new works in Shakespeare's style. But it cannot be a new Shakespeare, the is one level above it. An image generator might be able to make a painting is Picasso's style, but it cannot be a new Picasso - that is, create a style never before seen and explain why they did it and what emotions it conveys.
  - jodrellblank 2y ago
    Me: Hi, can you use the concept "bridge" in a never-seen-before sentence, where it still makes sense? Please make the sentence very unusual, but still coherent. Perhaps a couple of examples? ChatGPT: Sure! Here are a couple of unusual but coherent sentences using the concept of "bridge": The stars whispered their ancient stories across the sky, building a bridge of forgotten time between my dreams and the universe’s memories. Each thought she left unspoken laid a ghostly bridge between us, invisible but weighty, made of all the words that could have been. Both sentences play with abstract ideas, pushing the concept of a bridge beyond its typical use but maintaining coherence through metaphor and vivid imagery. ---- It's impossible for me to know everything in the training set but there are no Google results for "building a bridge of forgotten time" or "laid a ghostly bridge between us". If you told me those and said you made them up and didn't copy them from somewhere, I'd have to either find the source you copied or believe you.
    - vundercind 2y ago
      Novelty is not the same as understanding. I think it’s telling that one of the two probably works at least as well with something that’s the opposite of a bridge: gulf, chasm, wall. It just picked a pattern for an analogy and stuck your word in it, mad-libs style. Like, the fact that it’s kinda the opposite of what me would expect makes it a bit poetic in that one’s tempted to pursue what it might mean, but of course the LLM didn’t mean anything (though we still might find meaning). Its patterns encoded two items and “between” being likely to occur, anyway.
empath75 2y ago
You do not understand how GPT models work. It does more than 'importance weighting' and an absolutely massive amount of knowledge about the world is encoded in those weights. --- I asked ChatGPT to help you better understand how it works: There are a few common misconceptions in that comment regarding how large language models (LLMs) like GPT-4 actually work, so let's clarify those: Markov Chain Comparison: LLMs are not based on Markov chains, though they might seem similar at a high level due to their ability to predict the next word in a sequence. Markov chains rely on simple probabilistic transitions between states, often based on a very limited "memory" of previous states (e.g., the previous word or two). LLMs, on the other hand, use a transformer architecture, which allows them to consider long-term dependencies and relationships in text. This means they can account for the context of many preceding words, sentences, or even paragraphs when generating responses. No "Understanding": While it’s true that LLMs do not have consciousness, self-awareness, or human-like understanding, the term “understanding” can be misleading. They operate by modeling patterns in language, but in a highly sophisticated way. LLMs capture a deep representation of the relationships between words, sentences, and broader concepts through billions of parameters, giving them a kind of statistical "understanding" of language. This enables them to generate coherent and contextually appropriate responses, even if it’s not the same as human comprehension. Importance Weighting and Search: LLMs do not search through predefined sets of phrases or apply “importance-weighting” to words in the way described. They generate text dynamically by using the probabilities derived from the training data they’ve seen. The model calculates probabilities for each possible next word in the sequence, taking into account the entire context (not just key terms), and selects the next word based on these probabilities. This process is not about tagging words as important but about predicting the next most likely word or phrase given the context. Not Just "Anti-Nonsense Filtering": The quality of LLM output doesn’t arise from filtering out nonsense but from the underlying model’s ability to capture the complexity of human language through its learned representations. While there's a certain degree of training that discourages incoherent outputs, the coherent responses you see are mostly due to the model's training on vast, high-quality datasets. "Thoroughly Stupid": It's more accurate to say that LLMs are highly specialized in a particular domain: the patterns of human language. They excel at generating contextually relevant responses based on their training data. While they lack human-style cognition, calling them "stupid" overlooks the complexity of what they achieve within their domain. In summary, LLMs use advanced neural networks to predict and generate language, capturing sophisticated patterns across large datasets. They don't "understand" in a human sense, but their ability to model language goes far beyond simple mechanisms like Markov chains or weighted searches.

jodrellblank 2y ago

> "It tagged a few of your terms as important based on its training data, then did a weighted search for words to respond, then as it built the response, continued applying importance-weighting to its “own words” as it did further searches, to avoid writing gibberish or going off some weird direction." what specifically would you or I do different, apart from having less training data? > "There’s no “understanding”, nor anything particularly close to it" any evidence for this claim? It explained, it responded in context to a followup question, it gave other relevant examples, by what measure does it "not understand" but I "do understand"?

vundercind 2y ago
I’ll be a lot more open to arguments that the technology described in the Transformer Paper becomes capable of things like understanding when you throw enough data and computing power at it when the state of the art moves beyond, “yeah, but don’t human brains just randomly guess at the right thing all the time?” I dunno, do they do more than that? Seems like it to me. Does a Chinese Room “understand”? I say no, but hey, maybe it does. If I laboriously do the math by hand, taking care never to actually know the informational content of any of the input or output myself, does my scratch paper understand your questions? If the output’s just as good as ChatGPT? Where’s the part that understands?
- DougN7 2y ago
  The scratch paper is a fantastic analogy. I’m going to use it.
- fragmede 2y ago
  If we open up a human skull, what would we find? A clump of grey matter, a wet piece of meat. As we dissect it further and zoomed in, what would we find? Neurons. Zoom in futher and we get a mess of chemicals. As we look at the pile of cells that make up the brain, we have the same fundamental question: where's the part that understands? Yet somehow, from the connections of all of those cells, and neurotransmitters, there's consciousness and something there that does understand (and think and reason and love). If, instead of LLM architecture, on more powerful computers than we have now, we simulated all of those neurons and their connections, would we have a computer that understands? If we then did those computations on scratch paper, where would the piece that understands be on that piece of paper? The sum of a thing's parts can be greater than the individual parts. Whether or not ChatGPT understands is a whole big question, but we'll have no more luck dissecting LLMs to find out if it does than if we dissected a human brain.
  - vundercind 2y ago
    You can simulate a brain to some degree, maybe, but you won’t get the whole thing short of just… using an actual brain, that is physically identical to the one you’re simulating, down to at least the quantum level (and who knows what else?). It’s not at all certain that this can be reduced to math. The calculations for operating an LLM definitely can be reduced to math. No reduction needed, in fact—they are math. This isn’t an argument (to my mind, anyway) against even the possibility of machine whatever-you-like (consciousness, understanding, whatever) but against the idea of equivalence because we could simulate either one—in fact, we can’t simulate one. The other, essentially is already simulation, no further steps needed. What you’re getting at (if I may attempt to present your argument) is that we could reduce either to its components and make it look ridiculous that it might be doing anything particularly advanced. However, in fact we definitely can reproduce exactly what one of them does with a bunch of thick books of lookup tables and some formulas that we could mechanically follow by hand, and it might even be possible to do so in practice, not just hypothetically (at significant, but not impossible, expense) while we do not know we can do that for a human brain, short of just using exactly the brain that we want to “simulate”.
consteval 2y ago
> what specifically would you or I do different, apart from having less training data? Create things that are not in the training data. I could coin a new phrase that uses "bridge", I doubt an LLM can or would have any motivation to. Because LLMs understand data and patterns, but they don't understand the meaning behind them. They can't be imaginative. For example, an LLM can recite Shakespeare and can even write new works in Shakespeare's style. But it cannot be a new Shakespeare, the is one level above it. An image generator might be able to make a painting is Picasso's style, but it cannot be a new Picasso - that is, create a style never before seen and explain why they did it and what emotions it conveys.
- jodrellblank 2y ago
  Me: Hi, can you use the concept "bridge" in a never-seen-before sentence, where it still makes sense? Please make the sentence very unusual, but still coherent. Perhaps a couple of examples? ChatGPT: Sure! Here are a couple of unusual but coherent sentences using the concept of "bridge": The stars whispered their ancient stories across the sky, building a bridge of forgotten time between my dreams and the universe’s memories. Each thought she left unspoken laid a ghostly bridge between us, invisible but weighty, made of all the words that could have been. Both sentences play with abstract ideas, pushing the concept of a bridge beyond its typical use but maintaining coherence through metaphor and vivid imagery. ---- It's impossible for me to know everything in the training set but there are no Google results for "building a bridge of forgotten time" or "laid a ghostly bridge between us". If you told me those and said you made them up and didn't copy them from somewhere, I'd have to either find the source you copied or believe you.
  - vundercind 2y ago
    Novelty is not the same as understanding. I think it’s telling that one of the two probably works at least as well with something that’s the opposite of a bridge: gulf, chasm, wall. It just picked a pattern for an analogy and stuck your word in it, mad-libs style. Like, the fact that it’s kinda the opposite of what me would expect makes it a bit poetic in that one’s tempted to pursue what it might mean, but of course the LLM didn’t mean anything (though we still might find meaning). Its patterns encoded two items and “between” being likely to occur, anyway.

empath75 2y ago

You do not understand how GPT models work. It does more than 'importance weighting' and an absolutely massive amount of knowledge about the world is encoded in those weights. --- I asked ChatGPT to help you better understand how it works: There are a few common misconceptions in that comment regarding how large language models (LLMs) like GPT-4 actually work, so let's clarify those: Markov Chain Comparison: LLMs are not based on Markov chains, though they might seem similar at a high level due to their ability to predict the next word in a sequence. Markov chains rely on simple probabilistic transitions between states, often based on a very limited "memory" of previous states (e.g., the previous word or two). LLMs, on the other hand, use a transformer architecture, which allows them to consider long-term dependencies and relationships in text. This means they can account for the context of many preceding words, sentences, or even paragraphs when generating responses. No "Understanding": While it’s true that LLMs do not have consciousness, self-awareness, or human-like understanding, the term “understanding” can be misleading. They operate by modeling patterns in language, but in a highly sophisticated way. LLMs capture a deep representation of the relationships between words, sentences, and broader concepts through billions of parameters, giving them a kind of statistical "understanding" of language. This enables them to generate coherent and contextually appropriate responses, even if it’s not the same as human comprehension. Importance Weighting and Search: LLMs do not search through predefined sets of phrases or apply “importance-weighting” to words in the way described. They generate text dynamically by using the probabilities derived from the training data they’ve seen. The model calculates probabilities for each possible next word in the sequence, taking into account the entire context (not just key terms), and selects the next word based on these probabilities. This process is not about tagging words as important but about predicting the next most likely word or phrase given the context. Not Just "Anti-Nonsense Filtering": The quality of LLM output doesn’t arise from filtering out nonsense but from the underlying model’s ability to capture the complexity of human language through its learned representations. While there's a certain degree of training that discourages incoherent outputs, the coherent responses you see are mostly due to the model's training on vast, high-quality datasets. "Thoroughly Stupid": It's more accurate to say that LLMs are highly specialized in a particular domain: the patterns of human language. They excel at generating contextually relevant responses based on their training data. While they lack human-style cognition, calling them "stupid" overlooks the complexity of what they achieve within their domain. In summary, LLMs use advanced neural networks to predict and generate language, capturing sophisticated patterns across large datasets. They don't "understand" in a human sense, but their ability to model language goes far beyond simple mechanisms like Markov chains or weighted searches.