7 ms·

After reading the paper I’m struck by the lack of any discussion of awareness. Cognition requires at its basis awareness, which due to its entirely non verbal a

After reading the paper I’m struck by the lack of any discussion of awareness. Cognition requires at its basis awareness, which due to its entirely non verbal and unconstructed basis, is profoundly difficult to describe, measure, quantify, or label. This makes it to my mind impossible to train a model to be aware, let alone for humans to concretely describe it or evaluate it. Philosophy, especially Buddhism, has tried for thousands of years and psychology has all but abandoned attempting so. Hence papers like this that define AGI on psychometric dimensions that have the advantage of being easily measured but the disadvantage of being incomplete. My father is an emeritus professor of psychometrics and he agrees this is the biggest hurdle to AGI - that our ability to measure the dimensions of intelligence is woefully insufficient to the task of replicating intelligence. We scratch the surface and his opinion is language is sufficient to capture the knowledge of man, but not the spark of awareness required to be intelligent. This isn’t meant to be a mystical statement that it’s magic that makes humans intelligent or some exotic process impossible to compute. But that the nature of our mind is not observable in its entirety to us sufficient that the current learned reinforcement techniques can’t achieve it. Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. This process is unobservable and not measurable, and the only way we have to do so is through imperfect verbalizations that hint out some vague outline of a subconscious mind. But without being able to train a model on that subconscious process, one that can’t be expressed in language with any meaningful sufficiency, how will language models demonstrate it? Their very nature of autoregressive inference prohibits such a process from emerging at any scale. We might very well be able to fake it to an extent that it fools us, but awareness isn’t there - and I’d assert that awareness is all you need.

stevenjgarner 1y ago
I so completely agree. In virtually every conversation I have heard about AI, it only every talks about one of the multiple intelligences as theorized in Howard Gardner's book Frames of Mind: The Theory of Multiple Intelligences (1983)[1] There is little discussion of how AI will enhance (or destroy) our emotional intelligence, or our naturalistic, intrapersonal or interpersonal intelligences. Most religions, spiritual practices and even forms of meditation highlight the value of transcending mind and having awareness be present in the body. The way AGI is described, it would seem transcendence may be treated as a malfunction or bug. [1] https://en.wikipedia.org/wiki/Theory_of_multiple_intelligences https://en.wikipedia.org/wiki/Theory_of_multiple_intelligenc...
friendzis 1y ago
> Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. I do not have any even remotely practical definition for this, but this has to somehow involve the system being in a closed loop. It has to "run" in a sense that an operating system runs. Even if there is nothing coming on certain inputs it still has to run. And probably hallucinate (hehe) like humans do in an absence of a signal or infer patterns where there are none, yet be able to self-reflect that it is in fact a hallucination
colordrops 1y ago
Does general intelligence require awareness though? I think you are talking about consciousness, not intelligence. Though to be frank consciousness and intelligence are not well defined terms either.
newusertoday 1y ago
agreed. There is no way to tell if someone is aware or not we rely on brain activity to say someone is alive or not there is no way to tell someone or something is conscious currently.
diamond559 1y ago
Yep, computers execute code, they are tools. Humans have the capacity to spontaneously generate new thoughts out of nothing, solve problems never before solved and not just by brute force number crunching.
exe34 1y ago
> We might very well be able to fake it to an extent that it fools us, but awareness isn’t there we only need to fake it to the point it's undistinguishable from the carbon based one. faking is all you need.
sbergot 1y ago
There is no way to measure awareness. We can only know we are aware ourselves. For all we know trees or rocks might have awareness. Or I could be the only being aware of itself in the universe. We have no way to prove anything about it. Therefore it is not a useful descriptor of intelligence (be it human, animal or artificial).
- john_minsk 1y ago
  Agreed. Everything that looks like intelligence to ME is intelligent. My measurement of outside intelligence is limited by my intelligence. So I can understand when something is stupider compared to me. For example, industrial machine vs human worker, human worker is infinitely more intelligent compared to machine, because this human worker can do all kinds of interesting stuff. this metaphorical "human worker" did everything around from laying a brick to launching a man to the Moon. .... Imagine Super-future, where humanity created nanobots and they ate everything around. And now instead of Earth there is just a cloud of them. These nanonobots were clever and could adapt, and they had all the knowledge that humans had and even more(as they were eating earth a swarm was running global science experiments to understand as much as possible before the energy ends). Once they ate the last bite of our Earth(an important note here: they left an optimal amount of matter to keep running experiments. Humans were kept in a controlled state and were studied to increase Swarm's intelligence), they launched next stage. A project, grand architect named "Optimise Energy capture from the Sun". Nanobots re-created the most efficient ways of capturing the Sun energy - ancient plants, which swarm studied for centuries. Swarm added some upgrades on top of what nature came up with, but it was still built on top of what nature figured by itself. A perfect plant to capture the Sun's energy. All of them a perfect copy of itself + adaptive movements based on their geolocation and time(which makes all of them unique). For plants nanobots needed water, so they created efficient oceans to feed the plants. They added clouds and rains as transport mechanism between oceans and plants... etc etc. One night the human, which you already know by the name "Ivan the Liberator"(back then everyone called him just Ivan), didn't sleep on his usual hour. Suddenly all the lights went off and he saw a spark on the horizon. Horizon, that was strongly prohibited to approach. He took his rifle, jumped on a truck and raced to the shore - closest point to the spark vector. Once he approached - there was no horizon or water. A wall of dark glass-like material, edges barely noticeable. Just 30 cm wide. On the left and on the right from a 30 cm wide wall - an image as real as his hands - of a water and sky. At the top of the wall - a hole. He used his gun to hit the wall with the light - and it wasn't very thick, but once he hit - it regenerated very quickly. But once he hit a black wall - it shattered and he saw a different world - world of plants. He stepped into the forest, but these plants, were behaving differently. This part of the swarm wasn't supposed to face the human, so these nanobots never saw one and didn't have optimised instructions on what to do in that case. They started reporting new values back to the main computer and performing default behaviour until the updated software arrived from an intelligence center of the Swarm. A human was observing a strange thing - plants were smoothly flowing around him to keep a safe distance, like water steps away from your hands in a pond. "That's different" thought Ivan, extended his hand in a friendly gesture and said - Nice to meet you. I'm Ivan. .... In this story a human sees a forest with plants and has no clue that it is a swarm of intelligence far greater than him. To him it looks repetitive simple action that doesn't look random -> let's test how intelligent outside entity is -> If entity wants to show its intelligence - it answers to communication -> If entity wants to hide its intelligence - it pretends to be not intelligent. If Swarm decides to show you that it is intelligent - it can show you that it is intelligent up to your level. It won't be able to explain everything that it knows or understands to you, because you will be limited by your hardware. The limit for the Swarm is only computation power it can get.
munksbeer 1y ago
Does any of that argument really matter? And frankly, this statement: >This makes it to my mind impossible to train a model to be aware feels wrong. If you're arguing that human's are aware, then it is apparent that it is possible to train something to be aware. Nature doesn't have any formal definition of intelligence, or awareness, yet here we are. From a practical perspective, it might be implausibly difficult to recreate that on computers, but theoretically, no reason why not.
- raducu 1y ago
  > Does any of that argument really matter? And frankly, this statement. My definition of a complete AGI is: an AI that can read JIRA tickets, talk with non-programmers and do all my job and get me and all/most software engineers fired and proves sustainable. But in general, it's an AI that can do any remote-work just as good as humans.
- 2muchcoffeeman 1y ago
  Have we shown what the human brain does at a “hardware” level? Or are you just assuming that the basic building block of a computer is that same as the basic building block of a human brain?
  - munksbeer 1y ago
    I'm not assuming anything, I thought my post made that clear.
  - laterium 1y ago
    Basic building blocks are atoms. So, yes same. If you mean cells vs transistors, sure they're different. But we don't have to demonstrate anything to know that nature already made conscious intelligent AGI without it itself not understanding anything. Therefore AGI can be created without knowing what consciousness is. It happened at least once.
ZoomZoomZoom 1y ago
We don't want awareness because it begets individuals by means of agency and we'd need to give them rights. This is industry's nightmare scenario. People want autonomy, self-learning, consistent memory and perhaps individuality (in the discernability/quirkiness sense), but still morally unencumbered slaves.
- Fargren 1y ago
  Any definition of AGI that doesn't include awareness is wrongly co-opting the term, in my opinion. I do think some people are doing that, on purpose. That way they can get people who are passionate about actual-AGI to jump on board on working with/for unaware-AGI.
luisml77 1y ago
Awareness is just continuous propagation of the neural network, be that artificial or biological. The reason thoughts just "appear" is because the brain is continuously propagating signal through the neural network. LLMs also do this during their decoding phase, where they reason continuously with every token that they generate. There is no difference here. Then you say "we don't think most of the times using language exclusively" , but neither do LLMs. What most people fail to realise is that in between each token being generated, black magic is happening in between the transformer layers. The same type of magic you describe. High dimensional. Based on complex concepts. Merging of ideas. Fusion of vectors to form a combined concept. Smart compression. Application of abstract rules. An LLM does all of these things, and more, and you can prove this by how complex their output is. Or, you can read studies by Anthropic on interpretability, and how LLMs do math underneath the transformer layers. How they manipulate information. AGI is not here with LLMs, but its not because they lack reasoning ability. It's due to something different. Here is what I think is truly missing: continuous learning, long term memory, and infinite and efficient context/operation. All of these are tied together deeply, and thus I believe we are but a simple breakthrough away from AGI.
- 10weirdfishes 1y ago
  There are very significant differences between biological and artificial neural networks. Artificial neural networks are mathematical attempts to replicating how the brain’s neurons work. They are not and were never meant to be 1 to 1 replications. There is the difference in scale, where the “parameters” of human neural networks absolutely dwarf the current LLMs we have today. There is also the fact that they are materially different. The underlying biology and cell structure affects biological neural networks in ways that artificial neural networks just simply dont have access to. The idea of awareness being propagations through the NN is an interesting concept though. I wonder if this idea be proven through monitoring the electrical signals within the brain.
  - luisml77 1y ago
    People like to focus on the differences between the brain and artificial neural networks. I myself believe the only thing that truly matters is that you can form complex functions with the common neuron element. This is achieved via linking lots them together, and by each having a property known as non-linearity. These two things ensure that with neurons you can just about approximate any linear or non-linear function or behaviour. This means you can simulate inside your network pretty much any reality within this universe, its causation and the effects. The deeper your network the more complex the reality you can "understand". Understand just means simulate and run inputs to get outputs in a way that matches the real phenomenon. When someone is said to be "smart", it means they possess a set of rules and functions that can very accurately predict a reality. You mention scale, and while its true the number of neuron elements the brain has is larger than any LLM, its also true the brain is more sparse, meaning much less of the neurons are active at the same time. For a more fair comparison, you can also remove the motor cortex from the discussion, and talk just about the networks that reason. I believe the scale is comparable. In essence, I think it doesn't matter that the brain has a whole bunch of chemistry added into it that artificial neural networks don't. The underlying deep non-linear function mapping capability is the same, and I believe this depth is, in both cases, comparable.
wwizo 1y ago
Language is one of communication contracts. LLModels leverage these contracts to communicate data structures (shapes) that emerge when evaluating input. They are so good at prediction that when you give them a clue of a shape they will create something passable, and they keep getting better with training. I hear there's work being done on getting the world models out, distilling the 'cortex-core' (aka the thinking without data), to perhaps see if they're capable of more, but so far we're looking at holograms of wishful thinking that increase in resolution, but still lack any essence. This begs a question - can true intelligence even be artificial?
bnreed 1y ago
Can't "awareness" in both examples be approximated by a random seed generator? Both the human mind and autoregressive model just need any initial thought to iterate and improve off of, influenced by unique design + experienced priors.
throwaway314155 1y ago
> Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. Not to pile on, but isn't this actually a distinct example of _lack_ of awareness? As in, our brains have sparks of creativity without understanding the inception of those sparks? Perhaps I'm conflating some definition of "aware" with another definition of "awareness"?
- polytely 1y ago
  I think OPs example the awareness refers to the thing that becomes aware of the thoughts bubbling up from the subconscious
antonvs 1y ago
> Cognition requires at its basis awareness This seems like an unsupported assertion. LLMs already exhibit good functional understanding of and ability in many domains, and so it's not at all clear that they require any more "awareness" (are you referring to consciousness?) than they already have. > the spark of awareness required to be intelligent. Again, this seems like an assumption - that there's some quality of awareness (again, consciousness?), that LLMs don't have, that they need in order to be "intelligent". But why do you believe that? > We’ve all had sudden insights without deliberation or thought. Highly doubtful. What you mean is, "without conscious thought". Your conscious awareness of your cognition is not the entirety of your cognition. It's worth reading a bit of Dennett's work about this - he's good at pointing out the biases we tend to have about these kinds of issues. > We might very well be able to fake it to an extent that it fools us This leads to claiming that there are unobservable, undetectable differences. Which there may be - we might succeed in building LLMs that meet whatever the prevailing arbitrary definition of intelligence is, but that don't possess consciousness. At that point, though, how meaningful is it to say they're not intelligent because they're not conscious? They would be functionally intelligent. Arguably, they already are, in many significant ways.
tim333 1y ago
Awareness doesn't seem that hard for AI systems though - if you look at the screen on a self driving Tesla you can see if it's aware of pedestrians, cyclists etc. because it draws boxes around them on the screen as it becomes aware of them. I guess by 'AGI' most people mean human level or above so I guess you'd want human level awareness which Teslas and the like don't have yet.
fireflash38 1y ago
I'd argue the biggest issue with concretely defining intelligence is that any attempts end up falling in two buckets: 1. "Too" Broad, which raises uncomfortable questions about non-human intelligence and how we as humans treat them (see: whales, elephants, octopuses/cephalopods) 2. Too narrow, which again raises very uncomfortable issues about who and who does not qualify as human, and what we do with them. Put in other words, it's more an issue of ethics and morals than it is definitional.
sega_sai 1y ago
Anything that is not measurable (i.e. awareness, consciousness) is not very useful in practice as a metric. I don't think there is even an agreed definition what consciousness is, partially because it is not really observable outside of our own mind. Therefore I think it makes perfect sense that awareness is not discussed in the paper.
- sojournerc 1y ago
  Consciousness is observable in others! Our communication and empathy and indeed language depend on the awareness that others share our perceived reality but not our mind. As gp says, this is hard to describe or quantify, but that doesn't mean it's not a necessary trait for general intelligence. https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind
  - johntb86 1y ago
    But LLMs have been measured to have some theory of mind abilities at least as strong as humans: https://www.nature.com/articles/s41562-024-01882-z https://www.nature.com/articles/s41562-024-01882-z . At this point you either need to accept that either LLMs are already conscious, or that it's easy enough to fake being conscious that it's practically impossible to test for - philosophical zombies are possible. It doesn't seem to me that LLMs are conscious, so consciousness isn't really observable to others.
    - sojournerc 1y ago
      That's still using language. My dog has theory of mind in the real world where things actually exist. Also, those results don't look as strong to me as you suggest. I do not accept that an LLM is conscious nor could I ever unless I can have a theory of mind for it... Which is impossible given that it's a stochastic parrot without awareness of the things my five senses and my soul feel in reality.
laterium 1y ago
Because LLMs don't have this special quality that you call "awareness", then they cannot have "cognition", neither of which you defined? This is along the lines of "There must be something special with my mind that LLMs don't have, I can just feel it" special pleading whether you call it awareness, consciousness, qualia etc. As long as you cannot define it clearly or even show that you yourself have this quality, I think the burden of proof is on you to show why this has any real world implications rather than just being word play. We can build thinking, reasoning machines just fine without waiting for philosophers to finally answer what consciousness is.

stevenjgarner 1y ago

I so completely agree. In virtually every conversation I have heard about AI, it only every talks about one of the multiple intelligences as theorized in Howard Gardner's book Frames of Mind: The Theory of Multiple Intelligences (1983)[1] There is little discussion of how AI will enhance (or destroy) our emotional intelligence, or our naturalistic, intrapersonal or interpersonal intelligences. Most religions, spiritual practices and even forms of meditation highlight the value of transcending mind and having awareness be present in the body. The way AGI is described, it would seem transcendence may be treated as a malfunction or bug. [1] https://en.wikipedia.org/wiki/Theory_of_multiple_intelligences https://en.wikipedia.org/wiki/Theory_of_multiple_intelligenc...

friendzis 1y ago

> Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. I do not have any even remotely practical definition for this, but this has to somehow involve the system being in a closed loop. It has to "run" in a sense that an operating system runs. Even if there is nothing coming on certain inputs it still has to run. And probably hallucinate (hehe) like humans do in an absence of a signal or infer patterns where there are none, yet be able to self-reflect that it is in fact a hallucination

colordrops 1y ago

Does general intelligence require awareness though? I think you are talking about consciousness, not intelligence. Though to be frank consciousness and intelligence are not well defined terms either.

newusertoday 1y ago

agreed. There is no way to tell if someone is aware or not we rely on brain activity to say someone is alive or not there is no way to tell someone or something is conscious currently.

diamond559 1y ago

Yep, computers execute code, they are tools. Humans have the capacity to spontaneously generate new thoughts out of nothing, solve problems never before solved and not just by brute force number crunching.

exe34 1y ago

> We might very well be able to fake it to an extent that it fools us, but awareness isn’t there we only need to fake it to the point it's undistinguishable from the carbon based one. faking is all you need.

sbergot 1y ago

There is no way to measure awareness. We can only know we are aware ourselves. For all we know trees or rocks might have awareness. Or I could be the only being aware of itself in the universe. We have no way to prove anything about it. Therefore it is not a useful descriptor of intelligence (be it human, animal or artificial).

john_minsk 1y ago
Agreed. Everything that looks like intelligence to ME is intelligent. My measurement of outside intelligence is limited by my intelligence. So I can understand when something is stupider compared to me. For example, industrial machine vs human worker, human worker is infinitely more intelligent compared to machine, because this human worker can do all kinds of interesting stuff. this metaphorical "human worker" did everything around from laying a brick to launching a man to the Moon. .... Imagine Super-future, where humanity created nanobots and they ate everything around. And now instead of Earth there is just a cloud of them. These nanonobots were clever and could adapt, and they had all the knowledge that humans had and even more(as they were eating earth a swarm was running global science experiments to understand as much as possible before the energy ends). Once they ate the last bite of our Earth(an important note here: they left an optimal amount of matter to keep running experiments. Humans were kept in a controlled state and were studied to increase Swarm's intelligence), they launched next stage. A project, grand architect named "Optimise Energy capture from the Sun". Nanobots re-created the most efficient ways of capturing the Sun energy - ancient plants, which swarm studied for centuries. Swarm added some upgrades on top of what nature came up with, but it was still built on top of what nature figured by itself. A perfect plant to capture the Sun's energy. All of them a perfect copy of itself + adaptive movements based on their geolocation and time(which makes all of them unique). For plants nanobots needed water, so they created efficient oceans to feed the plants. They added clouds and rains as transport mechanism between oceans and plants... etc etc. One night the human, which you already know by the name "Ivan the Liberator"(back then everyone called him just Ivan), didn't sleep on his usual hour. Suddenly all the lights went off and he saw a spark on the horizon. Horizon, that was strongly prohibited to approach. He took his rifle, jumped on a truck and raced to the shore - closest point to the spark vector. Once he approached - there was no horizon or water. A wall of dark glass-like material, edges barely noticeable. Just 30 cm wide. On the left and on the right from a 30 cm wide wall - an image as real as his hands - of a water and sky. At the top of the wall - a hole. He used his gun to hit the wall with the light - and it wasn't very thick, but once he hit - it regenerated very quickly. But once he hit a black wall - it shattered and he saw a different world - world of plants. He stepped into the forest, but these plants, were behaving differently. This part of the swarm wasn't supposed to face the human, so these nanobots never saw one and didn't have optimised instructions on what to do in that case. They started reporting new values back to the main computer and performing default behaviour until the updated software arrived from an intelligence center of the Swarm. A human was observing a strange thing - plants were smoothly flowing around him to keep a safe distance, like water steps away from your hands in a pond. "That's different" thought Ivan, extended his hand in a friendly gesture and said - Nice to meet you. I'm Ivan. .... In this story a human sees a forest with plants and has no clue that it is a swarm of intelligence far greater than him. To him it looks repetitive simple action that doesn't look random -> let's test how intelligent outside entity is -> If entity wants to show its intelligence - it answers to communication -> If entity wants to hide its intelligence - it pretends to be not intelligent. If Swarm decides to show you that it is intelligent - it can show you that it is intelligent up to your level. It won't be able to explain everything that it knows or understands to you, because you will be limited by your hardware. The limit for the Swarm is only computation power it can get.

munksbeer 1y ago

Does any of that argument really matter? And frankly, this statement: >This makes it to my mind impossible to train a model to be aware feels wrong. If you're arguing that human's are aware, then it is apparent that it is possible to train something to be aware. Nature doesn't have any formal definition of intelligence, or awareness, yet here we are. From a practical perspective, it might be implausibly difficult to recreate that on computers, but theoretically, no reason why not.

raducu 1y ago
> Does any of that argument really matter? And frankly, this statement. My definition of a complete AGI is: an AI that can read JIRA tickets, talk with non-programmers and do all my job and get me and all/most software engineers fired and proves sustainable. But in general, it's an AI that can do any remote-work just as good as humans.
2muchcoffeeman 1y ago
Have we shown what the human brain does at a “hardware” level? Or are you just assuming that the basic building block of a computer is that same as the basic building block of a human brain?
- munksbeer 1y ago
  I'm not assuming anything, I thought my post made that clear.
- laterium 1y ago
  Basic building blocks are atoms. So, yes same. If you mean cells vs transistors, sure they're different. But we don't have to demonstrate anything to know that nature already made conscious intelligent AGI without it itself not understanding anything. Therefore AGI can be created without knowing what consciousness is. It happened at least once.

ZoomZoomZoom 1y ago

We don't want awareness because it begets individuals by means of agency and we'd need to give them rights. This is industry's nightmare scenario. People want autonomy, self-learning, consistent memory and perhaps individuality (in the discernability/quirkiness sense), but still morally unencumbered slaves.

Fargren 1y ago
Any definition of AGI that doesn't include awareness is wrongly co-opting the term, in my opinion. I do think some people are doing that, on purpose. That way they can get people who are passionate about actual-AGI to jump on board on working with/for unaware-AGI.

luisml77 1y ago

Awareness is just continuous propagation of the neural network, be that artificial or biological. The reason thoughts just "appear" is because the brain is continuously propagating signal through the neural network. LLMs also do this during their decoding phase, where they reason continuously with every token that they generate. There is no difference here. Then you say "we don't think most of the times using language exclusively" , but neither do LLMs. What most people fail to realise is that in between each token being generated, black magic is happening in between the transformer layers. The same type of magic you describe. High dimensional. Based on complex concepts. Merging of ideas. Fusion of vectors to form a combined concept. Smart compression. Application of abstract rules. An LLM does all of these things, and more, and you can prove this by how complex their output is. Or, you can read studies by Anthropic on interpretability, and how LLMs do math underneath the transformer layers. How they manipulate information. AGI is not here with LLMs, but its not because they lack reasoning ability. It's due to something different. Here is what I think is truly missing: continuous learning, long term memory, and infinite and efficient context/operation. All of these are tied together deeply, and thus I believe we are but a simple breakthrough away from AGI.

10weirdfishes 1y ago
There are very significant differences between biological and artificial neural networks. Artificial neural networks are mathematical attempts to replicating how the brain’s neurons work. They are not and were never meant to be 1 to 1 replications. There is the difference in scale, where the “parameters” of human neural networks absolutely dwarf the current LLMs we have today. There is also the fact that they are materially different. The underlying biology and cell structure affects biological neural networks in ways that artificial neural networks just simply dont have access to. The idea of awareness being propagations through the NN is an interesting concept though. I wonder if this idea be proven through monitoring the electrical signals within the brain.
- luisml77 1y ago
  People like to focus on the differences between the brain and artificial neural networks. I myself believe the only thing that truly matters is that you can form complex functions with the common neuron element. This is achieved via linking lots them together, and by each having a property known as non-linearity. These two things ensure that with neurons you can just about approximate any linear or non-linear function or behaviour. This means you can simulate inside your network pretty much any reality within this universe, its causation and the effects. The deeper your network the more complex the reality you can "understand". Understand just means simulate and run inputs to get outputs in a way that matches the real phenomenon. When someone is said to be "smart", it means they possess a set of rules and functions that can very accurately predict a reality. You mention scale, and while its true the number of neuron elements the brain has is larger than any LLM, its also true the brain is more sparse, meaning much less of the neurons are active at the same time. For a more fair comparison, you can also remove the motor cortex from the discussion, and talk just about the networks that reason. I believe the scale is comparable. In essence, I think it doesn't matter that the brain has a whole bunch of chemistry added into it that artificial neural networks don't. The underlying deep non-linear function mapping capability is the same, and I believe this depth is, in both cases, comparable.

wwizo 1y ago

Language is one of communication contracts. LLModels leverage these contracts to communicate data structures (shapes) that emerge when evaluating input. They are so good at prediction that when you give them a clue of a shape they will create something passable, and they keep getting better with training. I hear there's work being done on getting the world models out, distilling the 'cortex-core' (aka the thinking without data), to perhaps see if they're capable of more, but so far we're looking at holograms of wishful thinking that increase in resolution, but still lack any essence. This begs a question - can true intelligence even be artificial?

bnreed 1y ago

Can't "awareness" in both examples be approximated by a random seed generator? Both the human mind and autoregressive model just need any initial thought to iterate and improve off of, influenced by unique design + experienced priors.

throwaway314155 1y ago

> Try this exercise. Do not think and let your mind clear. Ideas will surface. By what process did they surface? Or clear your mind entirely then try to perform some complex task. You will be able to. How did you do this without thought? We’ve all had sudden insights without deliberation or thought. Where did these come from? By what process did you arrive at them? Most of the things we do or think are not deliberative and definitely not structured with language. Not to pile on, but isn't this actually a distinct example of _lack_ of awareness? As in, our brains have sparks of creativity without understanding the inception of those sparks? Perhaps I'm conflating some definition of "aware" with another definition of "awareness"?

polytely 1y ago
I think OPs example the awareness refers to the thing that becomes aware of the thoughts bubbling up from the subconscious

antonvs 1y ago

> Cognition requires at its basis awareness This seems like an unsupported assertion. LLMs already exhibit good functional understanding of and ability in many domains, and so it's not at all clear that they require any more "awareness" (are you referring to consciousness?) than they already have. > the spark of awareness required to be intelligent. Again, this seems like an assumption - that there's some quality of awareness (again, consciousness?), that LLMs don't have, that they need in order to be "intelligent". But why do you believe that? > We’ve all had sudden insights without deliberation or thought. Highly doubtful. What you mean is, "without conscious thought". Your conscious awareness of your cognition is not the entirety of your cognition. It's worth reading a bit of Dennett's work about this - he's good at pointing out the biases we tend to have about these kinds of issues. > We might very well be able to fake it to an extent that it fools us This leads to claiming that there are unobservable, undetectable differences. Which there may be - we might succeed in building LLMs that meet whatever the prevailing arbitrary definition of intelligence is, but that don't possess consciousness. At that point, though, how meaningful is it to say they're not intelligent because they're not conscious? They would be functionally intelligent. Arguably, they already are, in many significant ways.

tim333 1y ago

Awareness doesn't seem that hard for AI systems though - if you look at the screen on a self driving Tesla you can see if it's aware of pedestrians, cyclists etc. because it draws boxes around them on the screen as it becomes aware of them. I guess by 'AGI' most people mean human level or above so I guess you'd want human level awareness which Teslas and the like don't have yet.

fireflash38 1y ago

I'd argue the biggest issue with concretely defining intelligence is that any attempts end up falling in two buckets: 1. "Too" Broad, which raises uncomfortable questions about non-human intelligence and how we as humans treat them (see: whales, elephants, octopuses/cephalopods) 2. Too narrow, which again raises very uncomfortable issues about who and who does not qualify as human, and what we do with them. Put in other words, it's more an issue of ethics and morals than it is definitional.

sega_sai 1y ago

Anything that is not measurable (i.e. awareness, consciousness) is not very useful in practice as a metric. I don't think there is even an agreed definition what consciousness is, partially because it is not really observable outside of our own mind. Therefore I think it makes perfect sense that awareness is not discussed in the paper.

sojournerc 1y ago
Consciousness is observable in others! Our communication and empathy and indeed language depend on the awareness that others share our perceived reality but not our mind. As gp says, this is hard to describe or quantify, but that doesn't mean it's not a necessary trait for general intelligence. https://en.wikipedia.org/wiki/Theory_of_mind https://en.wikipedia.org/wiki/Theory_of_mind
- johntb86 1y ago
  But LLMs have been measured to have some theory of mind abilities at least as strong as humans: https://www.nature.com/articles/s41562-024-01882-z https://www.nature.com/articles/s41562-024-01882-z . At this point you either need to accept that either LLMs are already conscious, or that it's easy enough to fake being conscious that it's practically impossible to test for - philosophical zombies are possible. It doesn't seem to me that LLMs are conscious, so consciousness isn't really observable to others.
  - sojournerc 1y ago
    That's still using language. My dog has theory of mind in the real world where things actually exist. Also, those results don't look as strong to me as you suggest. I do not accept that an LLM is conscious nor could I ever unless I can have a theory of mind for it... Which is impossible given that it's a stochastic parrot without awareness of the things my five senses and my soul feel in reality.

laterium 1y ago

Because LLMs don't have this special quality that you call "awareness", then they cannot have "cognition", neither of which you defined? This is along the lines of "There must be something special with my mind that LLMs don't have, I can just feel it" special pleading whether you call it awareness, consciousness, qualia etc. As long as you cannot define it clearly or even show that you yourself have this quality, I think the burden of proof is on you to show why this has any real world implications rather than just being word play. We can build thinking, reasoning machines just fine without waiting for philosophers to finally answer what consciousness is.