4 ms·

conscious, kŏn′shəs, adjective -- Characterized by or having an awareness of one's environment and one's own existence, sensations, and thoughts. synonym: aware

conscious, kŏn′shəs, adjective -- Characterized by or having an awareness of one's environment and one's own existence, sensations, and thoughts. synonym: aware. Self-attention seems to be at least a proxy for "awareness of ... one's own existence." If that closed loop is the thing that converts sensibility into sentience, then maybe it's the source of LLM's leverage too. Is this language comprehension algorithm a sort of consciousness algorithm?

dlkf 3y ago
It’s debatable to what degree ”attention” in LLMs relates to ”attention” in psychology. See Cosma Shalizi’s note on this http://bactra.org/notebooks/nn-attention-and-transformers.html http://bactra.org/notebooks/nn-attention-and-transformers.ht...
sk11001 3y ago
ML attention is nothing like human attention. I think it’s madness to attempt to map concepts from one field we barely understand to another field we also barely understand just because they use overlapping language.
- jampekka 3y ago
  Having done some research into human attention, I have to agree with Hommel et al: No one knows what attention is [1]. In current ANNs "attention" is quite well defined: how to weigh some variables based on other variables. But anthropomorphizing such concepts indeed muddies things more than it clarifies. Including calling interconnected summation units with non-linear transformations "neural networks". But such (wrong) intuition pumping terminology does attract, well, attention, so they get adopted. [1] https://link.springer.com/article/10.3758/s13414-019-01846-w https://link.springer.com/article/10.3758/s13414-019-01846-w
ben_w 3y ago
Careful, depending on who you ask there's 40 different definitions of the term. Any given mind, natural or artificial, may well pass some of these without passing all of them.
kmeisthax 3y ago
No. Self-attention is more akin to kernel smoothing[0] on memorized training data that spits out a weighted probability graph. As for consciousness, LLMs are not particularly well aware of their own strengths and limitations, at least not unless you finetune them to know what they are and aren't good at. They also don't have sensors, so awareness of any environment is not possible. If you trained a neural network with an attention mechanism using data obtained from, say, robotics sensors; then it might be able to at least have environmental awareness. The problem is that current LLM training approaches rely on large amounts of training data - easy to obtain for text, nonexistent for sensor input. I suspect awareness of one's own existence, sensations, and thoughts would additionally require some kind of continuous weight update[1], but I have no proof for that yet. [0] https://en.wikipedia.org/wiki/Kernel_smoother https://en.wikipedia.org/wiki/Kernel_smoother [1] Neural network weights are almost always trained in one big run, occasionally updated with fine-tuning, and almost never modified during usage of the model. All of ChatGPT's ability to learn from prior input comes from in-context learning which does not modify weights. This is also why it tends to forget during long conversations.

dlkf 3y ago

It’s debatable to what degree ”attention” in LLMs relates to ”attention” in psychology. See Cosma Shalizi’s note on this http://bactra.org/notebooks/nn-attention-and-transformers.html http://bactra.org/notebooks/nn-attention-and-transformers.ht...

sk11001 3y ago

ML attention is nothing like human attention. I think it’s madness to attempt to map concepts from one field we barely understand to another field we also barely understand just because they use overlapping language.

jampekka 3y ago
Having done some research into human attention, I have to agree with Hommel et al: No one knows what attention is [1]. In current ANNs "attention" is quite well defined: how to weigh some variables based on other variables. But anthropomorphizing such concepts indeed muddies things more than it clarifies. Including calling interconnected summation units with non-linear transformations "neural networks". But such (wrong) intuition pumping terminology does attract, well, attention, so they get adopted. [1] https://link.springer.com/article/10.3758/s13414-019-01846-w https://link.springer.com/article/10.3758/s13414-019-01846-w

ben_w 3y ago

Careful, depending on who you ask there's 40 different definitions of the term. Any given mind, natural or artificial, may well pass some of these without passing all of them.

kmeisthax 3y ago

No. Self-attention is more akin to kernel smoothing[0] on memorized training data that spits out a weighted probability graph. As for consciousness, LLMs are not particularly well aware of their own strengths and limitations, at least not unless you finetune them to know what they are and aren't good at. They also don't have sensors, so awareness of any environment is not possible. If you trained a neural network with an attention mechanism using data obtained from, say, robotics sensors; then it might be able to at least have environmental awareness. The problem is that current LLM training approaches rely on large amounts of training data - easy to obtain for text, nonexistent for sensor input. I suspect awareness of one's own existence, sensations, and thoughts would additionally require some kind of continuous weight update[1], but I have no proof for that yet. [0] https://en.wikipedia.org/wiki/Kernel_smoother https://en.wikipedia.org/wiki/Kernel_smoother [1] Neural network weights are almost always trained in one big run, occasionally updated with fine-tuning, and almost never modified during usage of the model. All of ChatGPT's ability to learn from prior input comes from in-context learning which does not modify weights. This is also why it tends to forget during long conversations.