4 ms·

> It was an action of thinking about something in a logical, sensible way If only it was so simple! Empirically, you streamed a short text into a black box sys

> It was an action of thinking about something in a logical, sensible way If only it was so simple! Empirically, you streamed a short text into a black box system and received a short text out. The output text was related and coherent, and -- after the fact -- you assessed it to be logically, sensibly applicable given the input. On your way to using that to exemplify "reasoning" you took for granted that there was "thinking", and that the "thinking" was both "logical" and "sensible". Any number of systems might deliver the empirical result you measured, and many of them would involve none of that. LLM's are sophisticated and highly capable, but the discussion about what language to use for what they do isn't nearly so simple as you suggest.

wbl 2y ago
And how do I know you think?
- deleted 2y ago
  [deleted]
- swatcoder 2y ago
  Are you sure you do? You're just responding to a block of text on an internet forum with trivial signup flow, known to be populated by bots of greater or less sophistication, as everywhere else on the internet.
- d4mi3n 2y ago
  We don't! We've gotten into philosophy, which is always a blast to ponder over. Aside from measuring brain activity and making observations, all we really know is that awareness seems to be an emergent property of our biology. That said, we _do_ know how probabilistic models work and we do know that a calculator doesn't think or have awareness in the way we consider ourselves to.
  - NateEag 2y ago
    > we do know that a calculator doesn't think or have awareness in the way we consider ourselves to. If only this were true. Some people take panpsychism seriously, and while it may sound ludicrous at first blush, it isn't actually unreasonable. Hard materialist reductionism may be true but no one's really given an irrefutable explanation for the oddity which is consciousness. It's tempting to say "Things that don't respond to stimulus are obviously not self-aware", but see locked-in syndrome for a falsification of that claim.
d4mi3n 2y ago
I agree with this sentiment and would remind everyone that LLMs are probabilistic models. Anything that isn't in the training data set will not be produced as an output. If you squint really hard, you could kinda say an LLM is just a fancy compression/decompression scheme. That said, in addition to anthropomorphizing something that sounds like a person, I think a fascinating thing about these discussions are the variety of unstated opinions on what reasoning, thought, or awareness is. This whole topic would be a lot simpler to classify if we had a better understanding of how _we_ are able to reason, think, and be aware to the level of understanding we have of the math behind an LLM.
- famouswaffles 2y ago
  >that LLMs are probabilistic models. And the brain isn't ? How do you think you have such a seamlessly continuous view of reality ? The brain is very probabilistic. >Anything that isn't in the training data set will not be produced as an output. This is not a restriction of probabilistic models. And it's certainly not a restriction of SOTA LLMs you can test today. >be aware to the level of understanding we have of the math behind an LLM. We have very little understanding of the meaning of the computations in large ANNs
- mannykannot 2y ago
  > Anything that isn't in the training data set will not be produced as an output. Thinking back to some of the examples I have seen, it feels as though this is not so. In particular, I'm thinking of a series where ChatGPT was prompted to invent a toy language with a simple grammar and then translate sentences between that language and English [1]. It seems implausible that the outputs produced here were in the training data set. Furthermore, given that LLMs are probabilistic models and produce their output stochastically, it does not seem surprising that they might produce output not in their training data. I agree that we do not have a good understanding of what reasoning, thought, or awareness is. One of the questions I have been pondering lately is whether, when a possible solution or answer to some question pops into my mind, it is the result of an unconscious LLM-like process, but that can't be all of it; for one thing, I - unlike LLMs - have a limited ability to assess my ideas for plausibility without being prompted to do so. [1] https://maximumeffort.substack.com/p/i-taught-chatgpt-to-invent-a-language https://maximumeffort.substack.com/p/i-taught-chatgpt-to-inv...
famouswaffles 2y ago
>Any number of systems might deliver the empirical result you measured Like what ? What exactly would give similar empirical results over the wide number of tests LLMs have been subjected to ? >and many of them would involve none of that. Oh? How would you know that? Did I miss the breakthrough Intellisense-o-meter that can measure the intelligence flowing through a system ? I guess I'm just not sure what point you're trying to make here. Empirically assessing the result after the fact is how we determine intelligence and reasoning in.. anything humans included. How do you determine a piece of metal is real gold ? By comparing the results of a series of tests against the properties of gold you have outlined. The metal is a black box. You don't need to understand every interaction occuring in your tests to determine whether it is gold or not.
- swatcoder 2y ago
  > >Any number of systems might deliver the empirical result you measured > Like what ? What exactly would give similar empirical results over the wide number of tests... The GP gave a single trivial example and my reply was highlighting how that example didn't inform the question at hand in any way. That example might be just as well be exhibited by a lookup table, a lucky random output, a modified markov chain, a Chinese Box, a trivial small language model trained on the right corpus, etc. That certain LLM's or chatbots might also be able to deliver on more sophisticated examples that those systems cannot is not what they or I were talking about here. It was a discussion about obviousness from trivial samples, and about the deeper semantic dependencies hidden the GP's definition. Such trivial samples don't hold up to scrutiny as their definition recurses down through concepts that are not at all demonstrated in these trivial cases. Their attempt at rhetoric just fails. > Empirically assessing the result after the fact is how we determine intelligence and reasoning in.. anything humans included. No. We project intelligence onto humans inductively by identifying our own experience as intelligence and assuming that other things that are almost identical to us are very likely to be working in the same way. In recent decades, there's been growing acceptance that other systems approximately like us (animals of certain kinds) exhibit it to various degrees as well, but even this itself was a formally rejected concept during the early 20th century when "intelligence" first became treated as something measurable and quantifiable at all. The concept of intelligence is a wholly cultural one and its definition and applicability moves around over time. Yet there are currently very few people who would apply the word to a lookup table, a random output, a Chinese Room, a modified markov chain, etc and a smattering-but-growing number of people who are comfortable applying it to LLM's and chatbots as we see them today. As this happens and its use expands, the sense of the word changes, as it has been doing for hundreds of years. At this point, we mostly can't yet rely on chatbots to fulfill roles that require interesting human intelligence. They can pass some benchmark tests to greater or less degrees, but also happen to be excellent benchmark optimizers (that's the essence of their design), so it remains hard to know what that really means. If and when they become reliable substitutes for human intelligence in general tasks, or become interestingly independent in the way we see certain animals, the prior-yet-modern senses of intelligence will be easier to apply. But we're not there yet.

wbl 2y ago

And how do I know you think?

deleted 2y ago
[deleted]
swatcoder 2y ago
Are you sure you do? You're just responding to a block of text on an internet forum with trivial signup flow, known to be populated by bots of greater or less sophistication, as everywhere else on the internet.
d4mi3n 2y ago
We don't! We've gotten into philosophy, which is always a blast to ponder over. Aside from measuring brain activity and making observations, all we really know is that awareness seems to be an emergent property of our biology. That said, we _do_ know how probabilistic models work and we do know that a calculator doesn't think or have awareness in the way we consider ourselves to.
- NateEag 2y ago
  > we do know that a calculator doesn't think or have awareness in the way we consider ourselves to. If only this were true. Some people take panpsychism seriously, and while it may sound ludicrous at first blush, it isn't actually unreasonable. Hard materialist reductionism may be true but no one's really given an irrefutable explanation for the oddity which is consciousness. It's tempting to say "Things that don't respond to stimulus are obviously not self-aware", but see locked-in syndrome for a falsification of that claim.

d4mi3n 2y ago

I agree with this sentiment and would remind everyone that LLMs are probabilistic models. Anything that isn't in the training data set will not be produced as an output. If you squint really hard, you could kinda say an LLM is just a fancy compression/decompression scheme. That said, in addition to anthropomorphizing something that sounds like a person, I think a fascinating thing about these discussions are the variety of unstated opinions on what reasoning, thought, or awareness is. This whole topic would be a lot simpler to classify if we had a better understanding of how _we_ are able to reason, think, and be aware to the level of understanding we have of the math behind an LLM.

famouswaffles 2y ago
>that LLMs are probabilistic models. And the brain isn't ? How do you think you have such a seamlessly continuous view of reality ? The brain is very probabilistic. >Anything that isn't in the training data set will not be produced as an output. This is not a restriction of probabilistic models. And it's certainly not a restriction of SOTA LLMs you can test today. >be aware to the level of understanding we have of the math behind an LLM. We have very little understanding of the meaning of the computations in large ANNs
mannykannot 2y ago
> Anything that isn't in the training data set will not be produced as an output. Thinking back to some of the examples I have seen, it feels as though this is not so. In particular, I'm thinking of a series where ChatGPT was prompted to invent a toy language with a simple grammar and then translate sentences between that language and English [1]. It seems implausible that the outputs produced here were in the training data set. Furthermore, given that LLMs are probabilistic models and produce their output stochastically, it does not seem surprising that they might produce output not in their training data. I agree that we do not have a good understanding of what reasoning, thought, or awareness is. One of the questions I have been pondering lately is whether, when a possible solution or answer to some question pops into my mind, it is the result of an unconscious LLM-like process, but that can't be all of it; for one thing, I - unlike LLMs - have a limited ability to assess my ideas for plausibility without being prompted to do so. [1] https://maximumeffort.substack.com/p/i-taught-chatgpt-to-invent-a-language https://maximumeffort.substack.com/p/i-taught-chatgpt-to-inv...

famouswaffles 2y ago

>Any number of systems might deliver the empirical result you measured Like what ? What exactly would give similar empirical results over the wide number of tests LLMs have been subjected to ? >and many of them would involve none of that. Oh? How would you know that? Did I miss the breakthrough Intellisense-o-meter that can measure the intelligence flowing through a system ? I guess I'm just not sure what point you're trying to make here. Empirically assessing the result after the fact is how we determine intelligence and reasoning in.. anything humans included. How do you determine a piece of metal is real gold ? By comparing the results of a series of tests against the properties of gold you have outlined. The metal is a black box. You don't need to understand every interaction occuring in your tests to determine whether it is gold or not.

swatcoder 2y ago
> >Any number of systems might deliver the empirical result you measured > Like what ? What exactly would give similar empirical results over the wide number of tests... The GP gave a single trivial example and my reply was highlighting how that example didn't inform the question at hand in any way. That example might be just as well be exhibited by a lookup table, a lucky random output, a modified markov chain, a Chinese Box, a trivial small language model trained on the right corpus, etc. That certain LLM's or chatbots might also be able to deliver on more sophisticated examples that those systems cannot is not what they or I were talking about here. It was a discussion about obviousness from trivial samples, and about the deeper semantic dependencies hidden the GP's definition. Such trivial samples don't hold up to scrutiny as their definition recurses down through concepts that are not at all demonstrated in these trivial cases. Their attempt at rhetoric just fails. > Empirically assessing the result after the fact is how we determine intelligence and reasoning in.. anything humans included. No. We project intelligence onto humans inductively by identifying our own experience as intelligence and assuming that other things that are almost identical to us are very likely to be working in the same way. In recent decades, there's been growing acceptance that other systems approximately like us (animals of certain kinds) exhibit it to various degrees as well, but even this itself was a formally rejected concept during the early 20th century when "intelligence" first became treated as something measurable and quantifiable at all. The concept of intelligence is a wholly cultural one and its definition and applicability moves around over time. Yet there are currently very few people who would apply the word to a lookup table, a random output, a Chinese Room, a modified markov chain, etc and a smattering-but-growing number of people who are comfortable applying it to LLM's and chatbots as we see them today. As this happens and its use expands, the sense of the word changes, as it has been doing for hundreds of years. At this point, we mostly can't yet rely on chatbots to fulfill roles that require interesting human intelligence. They can pass some benchmark tests to greater or less degrees, but also happen to be excellent benchmark optimizers (that's the essence of their design), so it remains hard to know what that really means. If and when they become reliable substitutes for human intelligence in general tasks, or become interestingly independent in the way we see certain animals, the prior-yet-modern senses of intelligence will be easier to apply. But we're not there yet.