4 ms·

>> If your argument is simply that symbols and numbers can't contain meaning because they are symbols and numbers, then you've worked your way into a tautologic

>> If your argument is simply that symbols and numbers can't contain meaning because they are symbols and numbers, then you've worked your way into a tautological box that I, nor anyone else, will be able to free you from. Too many assumptions. >> Do you have some competing idea of how it works? It would behoove you to share it with the thread. Yes, I do. It's a language model. It is trained to model the text in a corpus. There's no mystery there and there's no reason to assume it's doing anything else than what it was designed to do. It's a machine. People made it. It does what it was made to do. Other explanations of its behaviour are fantasies. And computers aren't brains. Brains may be computers, or they may not, but you can't look at brains and draw conclusions about computers. For the same reason that you can't look at planes and draw conclusions about birds. >> The use of backpropagation and gradient descent allows us to search the space of all possible models starting from a randomized model, and find a model that does this by nudging our model to be a little more capable at it over and over again. If you haven't already, you should try implementing backpropagation, or even just gradient descent/hill climb, and observe its behaviour for a while. I suspect that, if you haven't done that, the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do. Anyway this is getting a little too long-winded for me and I don't have time to continue the conversation. I must respectfully bow out. Last word is yours and thanks for the interaction.

throw8889 4y ago
[flagged]
- YeGoblynQueenne 4y ago
  Look, this is the internet. I'm sure you know what crazy things happen on the internet. In view of that, are you sure it's such a great idea going around telling people that you are "`stalking'" them, quotes or no quotes? Could you think for a moment how some people might take your comment, including myself? Is that really how you want to interact with people online?
  - throw8889 4y ago
    [flagged]
Mike_12345 4y ago
> the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do. I'm not seeing any "rosy clouds" in his description. Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. We all know that GPT is not conscious, but it doesn't need to be. It seems obvious that it is recognizing abstract patterns and relations in the data that match our metaphorical comprehension of words and phrases, because it was designed and trained to do that, on human language that is filled with metaphorical relations. This is incredibly sophisticated mechanical pattern recognition with many layers. > Other explanations of its behaviour are fantasies. GPT is a type of deep neural network and the explanations I've seen in this thread have been along those lines.
- YeGoblynQueenne 4y ago
  >> Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. Not at all. Gradient optimisation "searches" for a set of parameter values that optimise (maximise or minimise) the gradient of a function. In machine learning, the function is some measure of error over a set of training data, that is to be minimised. There isn't anything in gradient optimisation that has to do with "meaning" at all. What I think you're saying is that by training on a lot of text, a language model somehow learns to minimise its error on the meaning of the text. If you squint a bit, you'll see there's an obvious gap, there, between "text" and "meaning", and it seems to me you're trying to close that gap by alluding to how humans close it. But that's just comparing apples and oranges. As Yann LeCun pointed out recently (there was a video of a discussion with Andrew Ng posted on HN a few days ago) for a human to "train" on all the text that large language models have trained on, the human would have to read several books a day for some thousands of years. So whatever humans can do to learn how to bridge the gap between text and meaning, that's not how language models do it, and drawing an analogy between the two obscures more than it elucidates. >> This is incredibly sophisticated mechanical pattern recognition with many layers. It's incredibly complex. I don't know what "sophisticated" means. Is a jumbo jet "sophisticated"? I'd say it's a very complex machine, but that doesn't mean it has abilities other than the ones it was manufactured to have. Complexity is not magick, you know. We have maths for it and all. Those are even foundational maths of computer science. If we started imagining "meaning" everytime complexity cropped up in computer systems, we'd be up to our knees in it.

throw8889 4y ago

[flagged]

YeGoblynQueenne 4y ago
Look, this is the internet. I'm sure you know what crazy things happen on the internet. In view of that, are you sure it's such a great idea going around telling people that you are "`stalking'" them, quotes or no quotes? Could you think for a moment how some people might take your comment, including myself? Is that really how you want to interact with people online?
- throw8889 4y ago
  [flagged]

Mike_12345 4y ago

> the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do. I'm not seeing any "rosy clouds" in his description. Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. We all know that GPT is not conscious, but it doesn't need to be. It seems obvious that it is recognizing abstract patterns and relations in the data that match our metaphorical comprehension of words and phrases, because it was designed and trained to do that, on human language that is filled with metaphorical relations. This is incredibly sophisticated mechanical pattern recognition with many layers. > Other explanations of its behaviour are fantasies. GPT is a type of deep neural network and the explanations I've seen in this thread have been along those lines.

YeGoblynQueenne 4y ago
>> Gradient descent is a search in model space for one that best models "meaning" relations in the data, among other things. Not at all. Gradient optimisation "searches" for a set of parameter values that optimise (maximise or minimise) the gradient of a function. In machine learning, the function is some measure of error over a set of training data, that is to be minimised. There isn't anything in gradient optimisation that has to do with "meaning" at all. What I think you're saying is that by training on a lot of text, a language model somehow learns to minimise its error on the meaning of the text. If you squint a bit, you'll see there's an obvious gap, there, between "text" and "meaning", and it seems to me you're trying to close that gap by alluding to how humans close it. But that's just comparing apples and oranges. As Yann LeCun pointed out recently (there was a video of a discussion with Andrew Ng posted on HN a few days ago) for a human to "train" on all the text that large language models have trained on, the human would have to read several books a day for some thousands of years. So whatever humans can do to learn how to bridge the gap between text and meaning, that's not how language models do it, and drawing an analogy between the two obscures more than it elucidates. >> This is incredibly sophisticated mechanical pattern recognition with many layers. It's incredibly complex. I don't know what "sophisticated" means. Is a jumbo jet "sophisticated"? I'd say it's a very complex machine, but that doesn't mean it has abilities other than the ones it was manufactured to have. Complexity is not magick, you know. We have maths for it and all. Those are even foundational maths of computer science. If we started imagining "meaning" everytime complexity cropped up in computer systems, we'd be up to our knees in it.