4 ms·

The author's definition of "strong AI" is incredibly weak. > Theory of Mind: These AI systems are capable of understanding mental states and can simulate the b

by rcme 3y ago

The author's definition of "strong AI" is incredibly weak. > Theory of Mind: These AI systems are capable of understanding mental states and can simulate the behavior of other agents. GPT models can do this. > Self-Aware Why is that a requirement? > These AI systems are capable of learning any intellectual task that a human can. Again, GPT models appear to do this (at least for text-based tasks). That being said, I think GPT models do have some limitations that prevent them for learning any intellectual task a human can do. I wrote a blog post a few days ago that explored trying to "teach" ChatGPT to perform binary addition [0]. My conclusion was that, when ChatGPT appears to solve a logical problem, that solution is just a regurgitation of a low-entropy example generalized from its training data. But the model fails to apply even basic logical rules as entropy increases, indicating that GPT models have a fundamental limit of applying logic to high entropy problems. 0: https://nobuteru.substack.com/p/can-chatgpt-learn-to-add https://nobuteru.substack.com/p/can-chatgpt-learn-to-add

shagie 3y ago
On the adding part - have you looked at how it tokenizes that string? https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer 10011111110100011100011011110010 00111010100101000100000101111011 is tokenized as: [100][1111][11][101][0001][11][0001][101][11][100][10] [00][11][101][01][001][01][0001][0000][01][01][11][101][1] It might be more trainable if the the bits where space delimited so that only the tokens [ 1] and [ 0] were in use. As it is, it doesn't really understand the contents of the tokens and manipulating [405, 1157, 8784, 486, 8298, 486, 18005, 2388, 486, 486, 1157, 8784, 16] is not easily trained - especially if there is a different bit sequence. Note also the misalignments of the tokens for bit length that can make it more difficult to work with.
- rcme 3y ago
  I disagree that tokenization is important to consider. But just to test it out, I tried adding spaces between each bit and verified that each bit is tokenized separately: Let's add the two binary numbers provided, limiting the work on each bit to a single line: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 -------------------------------------------------------------- 1 0 1 1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 0 1 So, the result of the addition is the binary number 10110110101001010111111001111101. The answer is still wrong. Think about it like this: even when tokenized into 1-5 digit binary chunks, there are still just a few tokens to consider. If ChatGPT could actually apply the rules of addition to a general, high-entropy example, it would be trivial to use these tokens to derive the correct answer. A human could easily do it.
  - sdenton4 3y ago
    Perhaps also consider that you (the human) are using 2-dimensional visual processing to help you solve the problem by collating your work. You also almost certainly work /backwards/ from right to left. The problem is simplified by scratch space and the human sensorium... (It would be interesting to see how multimodal models handle the problem, actually, given a picture of the addition problem.) The problem given to the LLM is more like: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 = ? Try writing the answer entirely from left-to-right without losing your place... It'll be error prone for humans, to say the least. It's why we invented our particular by-hand addition algorithms in the first place: these algorithms are co-adapted to the human sensorium. Luckily, we can ask any LLM worth its salt to write the single line of python code needed to solve the problem and it'll do just fine... edit: Thinking about this more - and trying to explicitly check the intermediate steps - I'm guessing that the LLM is getting lost in the inputs. The attention mechanism is going to need to find the correct pair of digits at each step, and combine them with some info about the carry variable. I'm guessing this is hard to do - there's a positional embedding used to attach some location info to each token, but in a long run of homogenous text, I would guess that it's hard to find a particular digit and end up with some off-by-one errors in the lookup. (and this is very similar to how we would expect humans to fail at the single-line version of the problem.)
    - rcme 3y ago
      I wouldn't expect a human to fail at the single line version. It's pretty easy to do. Write a list, at each position n in the list, compute the sum of the nth least-significant bit and the carry from the line above. You can easily get the nth least significant bit by counting from the right side of each number. The end result is the list printed backwards.
      - shagie 3y ago
        Consider the amount of work that you need to do to reverse a word in GPT. https://twitter.com/npew/status/1525900868305833984?lang=en# https://twitter.com/npew/status/1525900868305833984?lang=en# Note that part of the trick that they did to get it to work better is putting spaces around each letter to enforce a particular tokenization. Trying to do the same with mathematical objects rather than language objects would be even harder because GPT doesn't "know" math.
moffkalast 3y ago
> > Self-Aware > Why is that a requirement? Not to mention that this cannot be proven at all, even for humans.
sva_ 3y ago
I don't understand how you can write a whole piece on Transformers failing to do arithmetics without mentioning the tokenizer used in GPT, which makes it more or less impossible for the model. The reason is imo probably more along the lines of that the model can't see the actual numbers you're giving it. When you provide GPT3 with the number "10011111110100011100011011110010", it actually sees [3064, 26259, 1157, 8784, 18005, 1157, 18005, 8784, 1157, 3064, 940] [see 0]. The number isn't even split up into even parts. Even if it knew the Peano axioms, it wouldn't have a chance because it can't 'see' the actual numbers that you're feeding it. Hence it can only estimate these operations based on combination pairs it has seen in training data, and it seems pretty obvious why this can't be a robust basis for doing arithmetic. 0. https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- valine 3y ago
  It also completely ignores the developments around GPT tool use. If you’re evaluating the AI field as a whole, why are you restricting your analysis to the raw output of GPTs? Arguably the most impressive display of intelligence exhibited by these models is their ability to use tools like calculators to augment their own abilities. It doesn’t matter if the model can’t do three digit multiplication without access to a calculator. Most people can’t mentally multiply large numbers either.
  - rcme 3y ago
    Most people can do 32-bit binary addition using a pen and paper. And the point of using addition as an illustrative example is that the rules of addition are extremely simple, yet a binary addition problem can easily be scaled to N bits of entropy. This means it's very easy to show that the system fails to apply simple logic rules in such cases.
    - carrolldunham 3y ago
      It didn't fail logic, it failed to read your string. You didn't even notice that the terms being added in the 'show working' version don't match the given strings. And you are elsewhere talking about adding spaces to make the digits 'tokens'... you have no idea what you are talking about at all.
the_gipsy 3y ago
> > Self-Aware > Why is that a requirement? And what does self-aware even mean? A lot of animals pass the mirror test.
carrolldunham 3y ago
the linked blog is yet another instance of making false inferences from the false assumption of the model working at a letter/symbol level. it cannot see letters. it cannot see the 1 or 0 in 11010 either. (note that a request to simply reverse a word would illustrate this usually, but some glue code has been added to do it)
sdenton4 3y ago
Arithmetic is a known-bad example because of how the tokenization works. Similar to calling a human unintelligent because we can't recognize bat calls. Additionally 'can't do x, and is therefore a lookup table' is a very poor logical leap. Are humans who can't add just lookup tables as well?
- rcme 3y ago
  I would say that if you can't teach a human to add, then they might be a lookup table. Also, the tokenization is fine, as shown in other examples. Imagine we didn't call it arithmetic and instead used symbols a and b, s.t. a + a = a, a + b = b, and b + b = a w/ carry b. Binary addition really is just very basic logical rules on symbols.
mschuster91 3y ago
> Why is that a requirement? Some sort of self-awareness is required for an AI system interacting with the real world physically - i.e. via controlling a robot or a vehicle - safely and to react appropriately for situations it has not been taught for. > But the model fails to apply even basic logical rules as entropy increases, indicating that GPT models have a fundamental limit of applying logic to high entropy problems. IIRC, the problem with current GPT AI models is tokenization. Basically, words are long and somewhat visually distinct with clear separation, whereas numbers are far harder to "convert" into tokens the model can use. In the end I believe that this problem will eventually be solved by applying enough compute power, as will the problem with "self awareness" - run a GPT model in a continuous loop that allows self training. The real and actually very hard problem IMO is adversarial training and inputs. In humans, the limits of a human brain in the speed of interaction and the number of other actors one brain can interact with act as a natural hard performance ceiling as well as a quantity barrier for too much adversarial input (you can only get fed bullshit 24 person-hours a day), and yet we're seeing adversarial inputs (aka propaganda) be very effective to the tune that 40% of Americans believe that the 2020 election was stolen or manipulated [1]. An AI system that can interact with hundreds of thousands of people simultaneously? Microsoft proved how fast that can go sideways with Tay 2016, when 4chan and other trolls managed to turn the bot into a raging Nazi in not even a day worth of effort [2]. Now imagine you have an AI running airspace travel control and someone convinces it that a plane got hijacked by terrorists or to lead planes onto a collision course... [1] https://www.newsweek.com/40-americans-think-2020-election-stolen-days-before-midterms-1756218 https://www.newsweek.com/40-americans-think-2020-election-st... [2] https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)

by rcme 3y ago

shagie 3y ago

On the adding part - have you looked at how it tokenizes that string? https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer 10011111110100011100011011110010 00111010100101000100000101111011 is tokenized as: [100][1111][11][101][0001][11][0001][101][11][100][10] [00][11][101][01][001][01][0001][0000][01][01][11][101][1] It might be more trainable if the the bits where space delimited so that only the tokens [ 1] and [ 0] were in use. As it is, it doesn't really understand the contents of the tokens and manipulating [405, 1157, 8784, 486, 8298, 486, 18005, 2388, 486, 486, 1157, 8784, 16] is not easily trained - especially if there is a different bit sequence. Note also the misalignments of the tokens for bit length that can make it more difficult to work with.

rcme 3y ago
I disagree that tokenization is important to consider. But just to test it out, I tried adding spaces between each bit and verified that each bit is tokenized separately: Let's add the two binary numbers provided, limiting the work on each bit to a single line: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 -------------------------------------------------------------- 1 0 1 1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 0 1 So, the result of the addition is the binary number 10110110101001010111111001111101. The answer is still wrong. Think about it like this: even when tokenized into 1-5 digit binary chunks, there are still just a few tokens to consider. If ChatGPT could actually apply the rules of addition to a general, high-entropy example, it would be trivial to use these tokens to derive the correct answer. A human could easily do it.
- sdenton4 3y ago
  Perhaps also consider that you (the human) are using 2-dimensional visual processing to help you solve the problem by collating your work. You also almost certainly work /backwards/ from right to left. The problem is simplified by scratch space and the human sensorium... (It would be interesting to see how multimodal models handle the problem, actually, given a picture of the addition problem.) The problem given to the LLM is more like: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 = ? Try writing the answer entirely from left-to-right without losing your place... It'll be error prone for humans, to say the least. It's why we invented our particular by-hand addition algorithms in the first place: these algorithms are co-adapted to the human sensorium. Luckily, we can ask any LLM worth its salt to write the single line of python code needed to solve the problem and it'll do just fine... edit: Thinking about this more - and trying to explicitly check the intermediate steps - I'm guessing that the LLM is getting lost in the inputs. The attention mechanism is going to need to find the correct pair of digits at each step, and combine them with some info about the carry variable. I'm guessing this is hard to do - there's a positional embedding used to attach some location info to each token, but in a long run of homogenous text, I would guess that it's hard to find a particular digit and end up with some off-by-one errors in the lookup. (and this is very similar to how we would expect humans to fail at the single-line version of the problem.)
  - rcme 3y ago
    I wouldn't expect a human to fail at the single line version. It's pretty easy to do. Write a list, at each position n in the list, compute the sum of the nth least-significant bit and the carry from the line above. You can easily get the nth least significant bit by counting from the right side of each number. The end result is the list printed backwards.
    - shagie 3y ago
      Consider the amount of work that you need to do to reverse a word in GPT. https://twitter.com/npew/status/1525900868305833984?lang=en# https://twitter.com/npew/status/1525900868305833984?lang=en# Note that part of the trick that they did to get it to work better is putting spaces around each letter to enforce a particular tokenization. Trying to do the same with mathematical objects rather than language objects would be even harder because GPT doesn't "know" math.

moffkalast 3y ago

> > Self-Aware > Why is that a requirement? Not to mention that this cannot be proven at all, even for humans.

sva_ 3y ago

I don't understand how you can write a whole piece on Transformers failing to do arithmetics without mentioning the tokenizer used in GPT, which makes it more or less impossible for the model. The reason is imo probably more along the lines of that the model can't see the actual numbers you're giving it. When you provide GPT3 with the number "10011111110100011100011011110010", it actually sees [3064, 26259, 1157, 8784, 18005, 1157, 18005, 8784, 1157, 3064, 940] [see 0]. The number isn't even split up into even parts. Even if it knew the Peano axioms, it wouldn't have a chance because it can't 'see' the actual numbers that you're feeding it. Hence it can only estimate these operations based on combination pairs it has seen in training data, and it seems pretty obvious why this can't be a robust basis for doing arithmetic. 0. https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer

valine 3y ago
It also completely ignores the developments around GPT tool use. If you’re evaluating the AI field as a whole, why are you restricting your analysis to the raw output of GPTs? Arguably the most impressive display of intelligence exhibited by these models is their ability to use tools like calculators to augment their own abilities. It doesn’t matter if the model can’t do three digit multiplication without access to a calculator. Most people can’t mentally multiply large numbers either.
- rcme 3y ago
  Most people can do 32-bit binary addition using a pen and paper. And the point of using addition as an illustrative example is that the rules of addition are extremely simple, yet a binary addition problem can easily be scaled to N bits of entropy. This means it's very easy to show that the system fails to apply simple logic rules in such cases.
  - carrolldunham 3y ago
    It didn't fail logic, it failed to read your string. You didn't even notice that the terms being added in the 'show working' version don't match the given strings. And you are elsewhere talking about adding spaces to make the digits 'tokens'... you have no idea what you are talking about at all.

the_gipsy 3y ago

> > Self-Aware > Why is that a requirement? And what does self-aware even mean? A lot of animals pass the mirror test.

carrolldunham 3y ago

the linked blog is yet another instance of making false inferences from the false assumption of the model working at a letter/symbol level. it cannot see letters. it cannot see the 1 or 0 in 11010 either. (note that a request to simply reverse a word would illustrate this usually, but some glue code has been added to do it)

sdenton4 3y ago

Arithmetic is a known-bad example because of how the tokenization works. Similar to calling a human unintelligent because we can't recognize bat calls. Additionally 'can't do x, and is therefore a lookup table' is a very poor logical leap. Are humans who can't add just lookup tables as well?

rcme 3y ago
I would say that if you can't teach a human to add, then they might be a lookup table. Also, the tokenization is fine, as shown in other examples. Imagine we didn't call it arithmetic and instead used symbols a and b, s.t. a + a = a, a + b = b, and b + b = a w/ carry b. Binary addition really is just very basic logical rules on symbols.

mschuster91 3y ago

> Why is that a requirement? Some sort of self-awareness is required for an AI system interacting with the real world physically - i.e. via controlling a robot or a vehicle - safely and to react appropriately for situations it has not been taught for. > But the model fails to apply even basic logical rules as entropy increases, indicating that GPT models have a fundamental limit of applying logic to high entropy problems. IIRC, the problem with current GPT AI models is tokenization. Basically, words are long and somewhat visually distinct with clear separation, whereas numbers are far harder to "convert" into tokens the model can use. In the end I believe that this problem will eventually be solved by applying enough compute power, as will the problem with "self awareness" - run a GPT model in a continuous loop that allows self training. The real and actually very hard problem IMO is adversarial training and inputs. In humans, the limits of a human brain in the speed of interaction and the number of other actors one brain can interact with act as a natural hard performance ceiling as well as a quantity barrier for too much adversarial input (you can only get fed bullshit 24 person-hours a day), and yet we're seeing adversarial inputs (aka propaganda) be very effective to the tune that 40% of Americans believe that the 2020 election was stolen or manipulated [1]. An AI system that can interact with hundreds of thousands of people simultaneously? Microsoft proved how fast that can go sideways with Tay 2016, when 4chan and other trolls managed to turn the bot into a raging Nazi in not even a day worth of effort [2]. Now imagine you have an AI running airspace travel control and someone convinces it that a plane got hijacked by terrorists or to lead planes onto a collision course... [1] https://www.newsweek.com/40-americans-think-2020-election-stolen-days-before-midterms-1756218 https://www.newsweek.com/40-americans-think-2020-election-st... [2] https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot)