6 ms·

Well, I think because we know how the code is written, in the sense that humans quite literally wrote the code for it - it's definitely not thinking, and it is

Well, I think because we know how the code is written, in the sense that humans quite literally wrote the code for it - it's definitely not thinking, and it is literally doing what we asked, based on the data we gave it. It is specifically executing code we thought of. The output of course, we had no flying idea it would work this well. But it is not sentient. It has no idea of a self or anything like that. If it makes people believe that it does, it is because we have written so much lore about it in the training data.

famouswaffles 11mo ago
We do not write the code that makes it do what it does. We write the code that trains it to figure out how to do what it does. There's a big difference.
- abakker 11mo ago
  and then the code to give it context. AFAIU, there is a lot of post training "setup" in the context and variables to get the trained model to "behave as we instruct it to" Am I wrong about this?
- almosthere 11mo ago
  The code that builds the models and performance inference from it is code we have written. The data in the model is obviously the big trick. But what I'm saying is that if you run inference, that alone does not give it super-powers over your computer. You can write some agentic framework where it WOULD have power over your computer, but that's not what I'm referring to. It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model.
  - famouswaffles 11mo ago
    You cannot say, 'we know it's not thinking because we wrote the code' when the inference 'code' we wrote amounts to, 'Hey, just do whatever you figured out during training okay'. 'Power over your computer', all that is orthogonal to the point. A human brain without a functioning body would still be thinking.
    - almosthere 11mo ago
      Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage. I permit you, that yes, the data in the model is a LOT more cool, but some team could by hand, given billions of years (well probably at least 1 Octillion years), reproduce that model and save it to a disk. Again, no different than data stored in JIRA at that point. So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking".
      - famouswaffles 11mo ago
        >Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage. Obviously, it is different or else we would just use JIRA and a database to replace GPT. Models very obviously do NOT store training data in the weights in the way you are imagining. >So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking". Thinking is by all appearances substrate independent. The moment we created computers, we created another substrate that could, in the future think.
        almosthere 11mo ago
        But LLMs are effectively a very complex if/else if tree: if the user types "hi" respond with "hi" or "bye" or "..." you get the point. It's basically storing the most probably following words (tokens) given the current point and its history. That's not a brain and it's not thinking. It's similar to JIRA because it's stored information and there are if statements (admins can do this, users can do that). Yes it is more complex, but it's nowhere near the complexity of the human or bird brain that does not use clocks, does not have "turing machines inside", or any of the other complete junk other people posted in this thread. The information in Jira is just less complex, but it's in the same vein of the data in an LLM, just 10^100 times more complex. Just because something is complex does not mean it thinks.
  - gf000 11mo ago
    > It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model. Sure, and humans are just biochemical reactions moving muscles as their interface with the physical word. I think the model of operation is not a good criticism, but please see my reply to the root comment in this thread where I detail my thoughts a bit.
  - hackinthebochs 11mo ago
    This is a bad take. We didn't write the model, we wrote an algorithm that searches the space of models that conform to some high level constraints as specified by the stacked transformer architecture. But stacked transformers are a very general computational paradigm. The training aspect converges the parameters to a specific model that well reproduces the training data. But the computational circuits the model picks out are discovered, not programmed. The emergent structures realize new computational dynamics that we are mostly blind to. We are not the programmers of these models, rather we are their incubators. As far as sentience is concerned, we can't say they aren't sentient because we don't know the computational structures these models realize, nor do we know the computational structures required for sentience.
    - almosthere 11mo ago
      However there is another big problem, this would require a blob of data in a file to be labelled as "alive" even if it's on a disk in a garbage dump with no cpu or gpu anywhere near it. The inference software that would normally read from that file is also not alive, as it's literally very concise code that we wrote to traverse through that file. So if the disk isn't alive, the file on it isn't alive, the inference software is not alive - then what are you saying is alive and thinking?
      - goatlover 11mo ago
        A similar point was made by Jaron Lanier in his paper, "You can't argue with a Zombie".
      - hackinthebochs 11mo ago
        This is an overly reductive view of a fully trained LLM. You have identified the pieces, but you miss the whole. The inference code is like a circuit builder, it represents the high level matmuls and the potential paths for dataflow. The data blob as the fully converged model configures this circuit builder in the sense of specifying the exact pathways information flows through the system. But this isn't some inert formalism, this is an active, potent causal structure realized by the base computational substrate that is influencing and being influenced by the world. If anything is conscious here, it would be this structure. If the computational theory of mind is true, then there are some specific information dynamics that realize consciousness. Whether or not LLM training finds these structures is an open question.
- mbesto 11mo ago
  I think the discrepancy is this: 1. We trained it on a fraction of the world's information (e.g. text and media that is explicitly online) 2. It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online (which may or may not be different to the experiences humans have in every day life)
  - nix0n 11mo ago
    > It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online This is going to be a huge problem. Most people assume computers are unbiased and rational, and increasing use of AI will lead to more and larger decisions being made by AI.
  - aryehof 11mo ago
    I see this a lot in what LLMs know and promote in terms of software architecture. All seem biased to recent buzzwords and approaches. Discussions will include the same hand-waving of DDD, event-sourcing and hexagonal services, i.e. the current fashion. Nothing of worth apparently preceded them. I fear that we are condemned to a future where there is no new novel progress, but just a regurgitation of those current fashion and biases.
mentos 11mo ago
What’s crazy to me is the mechanism of pleasure or pain. I can understand that with enough complexity we can give rise to sentience but what does it take to achieve sensation?
- spicyusername 11mo ago
  A body
  - mentos 11mo ago
    I’d say it’s possible to experience mental anguish/worry without the body participating. Solely a cognitive pain from consternation.
    - AndrewKemendo 11mo ago
      You can’t cognate without a body - the brain and body is a material system tightly coupled
      - vidarh 11mo ago
        Ignoring that "cognate" isn't a verb, we have basis for making any claim about the necessity of that coupling.
  - exe34 11mo ago
    How does a body know what's going on? Would you say it has any input devices?
- kbrkbr 11mo ago
  Can you tell me how you understand that? Because I sincerely do not. I have frankly no idea how sentience arises from non sentience. But it's a topic that really interests me.
  - mentos 11mo ago
Llamamoe 11mo ago
This is probably true. But the truth is we have absolutely no idea what sentience is and what gives rise to it. We cannot identify why humans have it rather than just being complex biological machines, or whether and why other animals do. We have no idea what the rules or, nevermind how and why they would or wouldn't apply to AI.
marstall 11mo ago
Unless the idea of us having a thinking self is just something that comes out of our mouth, an artifact of language. In which case we are not that different - in the end we all came from mere atoms, after all!
mirekrusin 11mo ago
Now convince us that you’re sentient and not just regurgitating what you’ve heard and seen in your life.
- deleted 11mo ago
  [deleted]
- embedding-shape 11mo ago
  By what definition of "sentience"? Wikipedia claims "Sentience is the ability to experience feelings and sensations" as an opening statement, which I think would be trivial depending again on your definition of "experience" and "sensations". Can a LLM hooked up to sensor events be considered to "experience sensations"? I could see arguments both ways for that.
  - vidarh 11mo ago
    I have no way of measuring whether or not you experience feelings and sensations, or are just regurgitating statements to convince me of that. The only basis I have for assuming you are sentient according to that definition is trust in your self-reports.
    - embedding-shape 11mo ago
      I'm fairly sure we can measure human "sensation" as in detect physiological activity in the body in someone who is under anesthesia yet the body reacts in different ways to touch or pain. The "feelings" part is probably harder though.
      - vidarh 11mo ago
        We can measure the physiological activity, but not whether it gives rise to the same sensations that we experience ourselves. We can reasonably project and guess that they are the same, but we can not know. In practical terms it does not matter - it is reasonable for us to act as if others do experience the same we do. But if we are to talk about the nature of conscience and sentience it does matter that the only basis we have for knowing about other sentient beings is their self-reported experience.
dist-epoch 11mo ago
Your brain is just following the laws of chemistry. So where is your thinking found in a bunch of chemical reactions?
PaulDavisThe1st 11mo ago
> But it is not sentient. It has no idea of a self or anything like that. Who stated that sentience or sense of self is a part of thinking?
gf000 11mo ago
Well, unless you believe in some spiritual, non-physical aspect of consciousness, we could probably agree that human intelligence is Turing-complete (with a slightly sloppy use of terms). So any other Turing-complete model can emulate it, including a computer. We can even randomly generate Turing machines, as they are just data. Now imagine we are extremely lucky and happen to end up with a super-intelligent program which through the mediums it can communicate (it could be simply text-based but a 2D video with audio is no different for my perspective) can't be differentiated from a human being. Would you consider it sentient? Now replace the random generation with, say, a back propagation algorithm. If it's sufficiently large, don't you think it's indifferent from the former case - that is, novel qualities could emerge? With that said, I don't think that current LLMs are anywhere close to this category, but I just don't think this your reasoning is sound.
- myrmidon 11mo ago
  > Would you consider it sentient? Absolutely. If you simulated a human brain by the atom, would you think the resulting construct would NOT be? What would be missing? I think consciousness is simply an emergent property of our nervous system, but in order to express itself "language" is obviously needed and thus requires lots of complexity (more than what we typically see in animals or computer systems until recently).
  - prmph 11mo ago
    > If you simulated a human brain by the atom, That is what we don't know is possible. You don't even know what physics or particles are as yet undiscovered. And from what we even know currently, atoms are too coarse to form the basis of such "cloning" And, my viewpoint is that, even if this were possible, just because you simulated a brain atom by atom, does not mean you have a consciousness. If it is the arrangement of matter that gives rise to consciousness, then would that new consciousness be the same person or not? If you have a basis for answering that question, let's hear it.
    - gf000 11mo ago
      Well, if you were to magically make an exact replica of a person, wouldn't it be conscious and at time 0 be the same person? But later on, he would get different experiences and become a different person no longer identical to the first. In extension, I would argue that magically "translating" a person to another medium (e.g. a chip) would still make for the same person, initially. Though the word "magic" does a lot of work here.
kakapo5672 11mo ago
It's not accurate to say we "wrote the code for it". AI isn't built like normal software. Nowhere inside an AI will you find lines of code that say If X Then Y, and so on. Rather, these models are literally grown during the training phase. And all the intelligence emerges from that growth. That's what makes them a black box and extremely difficult to penetrate. No one can say exactly how they work inside for a given problem.

famouswaffles 11mo ago

We do not write the code that makes it do what it does. We write the code that trains it to figure out how to do what it does. There's a big difference.

abakker 11mo ago
and then the code to give it context. AFAIU, there is a lot of post training "setup" in the context and variables to get the trained model to "behave as we instruct it to" Am I wrong about this?
almosthere 11mo ago
The code that builds the models and performance inference from it is code we have written. The data in the model is obviously the big trick. But what I'm saying is that if you run inference, that alone does not give it super-powers over your computer. You can write some agentic framework where it WOULD have power over your computer, but that's not what I'm referring to. It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model.
- famouswaffles 11mo ago
  You cannot say, 'we know it's not thinking because we wrote the code' when the inference 'code' we wrote amounts to, 'Hey, just do whatever you figured out during training okay'. 'Power over your computer', all that is orthogonal to the point. A human brain without a functioning body would still be thinking.
  - almosthere 11mo ago
    Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage. I permit you, that yes, the data in the model is a LOT more cool, but some team could by hand, given billions of years (well probably at least 1 Octillion years), reproduce that model and save it to a disk. Again, no different than data stored in JIRA at that point. So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking".
    - famouswaffles 11mo ago
      >Well, a model by itself with data that emits a bunch of human written words is literally no different than what JIRA does when it reads a database table and shits it out to a screen, except maybe a lot more GPU usage. Obviously, it is different or else we would just use JIRA and a database to replace GPT. Models very obviously do NOT store training data in the weights in the way you are imagining. >So basically if you have that stance you'd have to agree that when we FIRST invented computers, we created intelligence that is "thinking". Thinking is by all appearances substrate independent. The moment we created computers, we created another substrate that could, in the future think.
      - almosthere 11mo ago
        But LLMs are effectively a very complex if/else if tree: if the user types "hi" respond with "hi" or "bye" or "..." you get the point. It's basically storing the most probably following words (tokens) given the current point and its history. That's not a brain and it's not thinking. It's similar to JIRA because it's stored information and there are if statements (admins can do this, users can do that). Yes it is more complex, but it's nowhere near the complexity of the human or bird brain that does not use clocks, does not have "turing machines inside", or any of the other complete junk other people posted in this thread. The information in Jira is just less complex, but it's in the same vein of the data in an LLM, just 10^100 times more complex. Just because something is complex does not mean it thinks.
- gf000 11mo ago
  > It's not a living thing inside the computer, it's just the inference building text token by token using probabilities based on the pre-computed model. Sure, and humans are just biochemical reactions moving muscles as their interface with the physical word. I think the model of operation is not a good criticism, but please see my reply to the root comment in this thread where I detail my thoughts a bit.
- hackinthebochs 11mo ago
  This is a bad take. We didn't write the model, we wrote an algorithm that searches the space of models that conform to some high level constraints as specified by the stacked transformer architecture. But stacked transformers are a very general computational paradigm. The training aspect converges the parameters to a specific model that well reproduces the training data. But the computational circuits the model picks out are discovered, not programmed. The emergent structures realize new computational dynamics that we are mostly blind to. We are not the programmers of these models, rather we are their incubators. As far as sentience is concerned, we can't say they aren't sentient because we don't know the computational structures these models realize, nor do we know the computational structures required for sentience.
  - almosthere 11mo ago
    However there is another big problem, this would require a blob of data in a file to be labelled as "alive" even if it's on a disk in a garbage dump with no cpu or gpu anywhere near it. The inference software that would normally read from that file is also not alive, as it's literally very concise code that we wrote to traverse through that file. So if the disk isn't alive, the file on it isn't alive, the inference software is not alive - then what are you saying is alive and thinking?
    - goatlover 11mo ago
      A similar point was made by Jaron Lanier in his paper, "You can't argue with a Zombie".
    - hackinthebochs 11mo ago
      This is an overly reductive view of a fully trained LLM. You have identified the pieces, but you miss the whole. The inference code is like a circuit builder, it represents the high level matmuls and the potential paths for dataflow. The data blob as the fully converged model configures this circuit builder in the sense of specifying the exact pathways information flows through the system. But this isn't some inert formalism, this is an active, potent causal structure realized by the base computational substrate that is influencing and being influenced by the world. If anything is conscious here, it would be this structure. If the computational theory of mind is true, then there are some specific information dynamics that realize consciousness. Whether or not LLM training finds these structures is an open question.
mbesto 11mo ago
I think the discrepancy is this: 1. We trained it on a fraction of the world's information (e.g. text and media that is explicitly online) 2. It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online (which may or may not be different to the experiences humans have in every day life)
- nix0n 11mo ago
  > It carries all of the biases us humans have and worse the biases that are present in the information we chose to explicitly share online This is going to be a huge problem. Most people assume computers are unbiased and rational, and increasing use of AI will lead to more and larger decisions being made by AI.
- aryehof 11mo ago
  I see this a lot in what LLMs know and promote in terms of software architecture. All seem biased to recent buzzwords and approaches. Discussions will include the same hand-waving of DDD, event-sourcing and hexagonal services, i.e. the current fashion. Nothing of worth apparently preceded them. I fear that we are condemned to a future where there is no new novel progress, but just a regurgitation of those current fashion and biases.

mentos 11mo ago

What’s crazy to me is the mechanism of pleasure or pain. I can understand that with enough complexity we can give rise to sentience but what does it take to achieve sensation?

spicyusername 11mo ago
A body
- mentos 11mo ago
  I’d say it’s possible to experience mental anguish/worry without the body participating. Solely a cognitive pain from consternation.
  - AndrewKemendo 11mo ago
    You can’t cognate without a body - the brain and body is a material system tightly coupled
    - vidarh 11mo ago
      Ignoring that "cognate" isn't a verb, we have basis for making any claim about the necessity of that coupling.
- exe34 11mo ago
  How does a body know what's going on? Would you say it has any input devices?
kbrkbr 11mo ago
Can you tell me how you understand that? Because I sincerely do not. I have frankly no idea how sentience arises from non sentience. But it's a topic that really interests me.
- mentos 11mo ago

Llamamoe 11mo ago

This is probably true. But the truth is we have absolutely no idea what sentience is and what gives rise to it. We cannot identify why humans have it rather than just being complex biological machines, or whether and why other animals do. We have no idea what the rules or, nevermind how and why they would or wouldn't apply to AI.

marstall 11mo ago

Unless the idea of us having a thinking self is just something that comes out of our mouth, an artifact of language. In which case we are not that different - in the end we all came from mere atoms, after all!

mirekrusin 11mo ago

Now convince us that you’re sentient and not just regurgitating what you’ve heard and seen in your life.

deleted 11mo ago
[deleted]
embedding-shape 11mo ago
By what definition of "sentience"? Wikipedia claims "Sentience is the ability to experience feelings and sensations" as an opening statement, which I think would be trivial depending again on your definition of "experience" and "sensations". Can a LLM hooked up to sensor events be considered to "experience sensations"? I could see arguments both ways for that.
- vidarh 11mo ago
  I have no way of measuring whether or not you experience feelings and sensations, or are just regurgitating statements to convince me of that. The only basis I have for assuming you are sentient according to that definition is trust in your self-reports.
  - embedding-shape 11mo ago
    I'm fairly sure we can measure human "sensation" as in detect physiological activity in the body in someone who is under anesthesia yet the body reacts in different ways to touch or pain. The "feelings" part is probably harder though.
    - vidarh 11mo ago
      We can measure the physiological activity, but not whether it gives rise to the same sensations that we experience ourselves. We can reasonably project and guess that they are the same, but we can not know. In practical terms it does not matter - it is reasonable for us to act as if others do experience the same we do. But if we are to talk about the nature of conscience and sentience it does matter that the only basis we have for knowing about other sentient beings is their self-reported experience.

dist-epoch 11mo ago

Your brain is just following the laws of chemistry. So where is your thinking found in a bunch of chemical reactions?

PaulDavisThe1st 11mo ago

> But it is not sentient. It has no idea of a self or anything like that. Who stated that sentience or sense of self is a part of thinking?

gf000 11mo ago

Well, unless you believe in some spiritual, non-physical aspect of consciousness, we could probably agree that human intelligence is Turing-complete (with a slightly sloppy use of terms). So any other Turing-complete model can emulate it, including a computer. We can even randomly generate Turing machines, as they are just data. Now imagine we are extremely lucky and happen to end up with a super-intelligent program which through the mediums it can communicate (it could be simply text-based but a 2D video with audio is no different for my perspective) can't be differentiated from a human being. Would you consider it sentient? Now replace the random generation with, say, a back propagation algorithm. If it's sufficiently large, don't you think it's indifferent from the former case - that is, novel qualities could emerge? With that said, I don't think that current LLMs are anywhere close to this category, but I just don't think this your reasoning is sound.

myrmidon 11mo ago
> Would you consider it sentient? Absolutely. If you simulated a human brain by the atom, would you think the resulting construct would NOT be? What would be missing? I think consciousness is simply an emergent property of our nervous system, but in order to express itself "language" is obviously needed and thus requires lots of complexity (more than what we typically see in animals or computer systems until recently).
- prmph 11mo ago
  > If you simulated a human brain by the atom, That is what we don't know is possible. You don't even know what physics or particles are as yet undiscovered. And from what we even know currently, atoms are too coarse to form the basis of such "cloning" And, my viewpoint is that, even if this were possible, just because you simulated a brain atom by atom, does not mean you have a consciousness. If it is the arrangement of matter that gives rise to consciousness, then would that new consciousness be the same person or not? If you have a basis for answering that question, let's hear it.
  - gf000 11mo ago
    Well, if you were to magically make an exact replica of a person, wouldn't it be conscious and at time 0 be the same person? But later on, he would get different experiences and become a different person no longer identical to the first. In extension, I would argue that magically "translating" a person to another medium (e.g. a chip) would still make for the same person, initially. Though the word "magic" does a lot of work here.

kakapo5672 11mo ago

It's not accurate to say we "wrote the code for it". AI isn't built like normal software. Nowhere inside an AI will you find lines of code that say If X Then Y, and so on. Rather, these models are literally grown during the training phase. And all the intelligence emerges from that growth. That's what makes them a black box and extremely difficult to penetrate. No one can say exactly how they work inside for a given problem.