7 ms·

I disagree that tokenization is important to consider. But just to test it out, I tried adding spaces between each bit and verified that each bit is tokenized s

by rcme 3y ago

I disagree that tokenization is important to consider. But just to test it out, I tried adding spaces between each bit and verified that each bit is tokenized separately: Let's add the two binary numbers provided, limiting the work on each bit to a single line: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 -------------------------------------------------------------- 1 0 1 1 0 1 1 0 1 0 0 1 0 0 0 1 0 1 1 1 1 1 0 0 0 1 1 1 1 1 0 1 So, the result of the addition is the binary number 10110110101001010111111001111101. The answer is still wrong. Think about it like this: even when tokenized into 1-5 digit binary chunks, there are still just a few tokens to consider. If ChatGPT could actually apply the rules of addition to a general, high-entropy example, it would be trivial to use these tokens to derive the correct answer. A human could easily do it.

sdenton4 3y ago
Perhaps also consider that you (the human) are using 2-dimensional visual processing to help you solve the problem by collating your work. You also almost certainly work /backwards/ from right to left. The problem is simplified by scratch space and the human sensorium... (It would be interesting to see how multimodal models handle the problem, actually, given a picture of the addition problem.) The problem given to the LLM is more like: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 = ? Try writing the answer entirely from left-to-right without losing your place... It'll be error prone for humans, to say the least. It's why we invented our particular by-hand addition algorithms in the first place: these algorithms are co-adapted to the human sensorium. Luckily, we can ask any LLM worth its salt to write the single line of python code needed to solve the problem and it'll do just fine... edit: Thinking about this more - and trying to explicitly check the intermediate steps - I'm guessing that the LLM is getting lost in the inputs. The attention mechanism is going to need to find the correct pair of digits at each step, and combine them with some info about the carry variable. I'm guessing this is hard to do - there's a positional embedding used to attach some location info to each token, but in a long run of homogenous text, I would guess that it's hard to find a particular digit and end up with some off-by-one errors in the lookup. (and this is very similar to how we would expect humans to fail at the single-line version of the problem.)
- rcme 3y ago
  I wouldn't expect a human to fail at the single line version. It's pretty easy to do. Write a list, at each position n in the list, compute the sum of the nth least-significant bit and the carry from the line above. You can easily get the nth least significant bit by counting from the right side of each number. The end result is the list printed backwards.
  - shagie 3y ago
    Consider the amount of work that you need to do to reverse a word in GPT. https://twitter.com/npew/status/1525900868305833984?lang=en# https://twitter.com/npew/status/1525900868305833984?lang=en# Note that part of the trick that they did to get it to work better is putting spaces around each letter to enforce a particular tokenization. Trying to do the same with mathematical objects rather than language objects would be even harder because GPT doesn't "know" math.
  - sdenton4 3y ago
    You lost the thread at 'write.' Try to do it in your head; we pretty much have to resort to additional tools to succeed at this task. Here's a LLM-generated python solution, which works just fine: def add_binary(a, b): """Adds two binary numbers given as strings. Args: a: The first binary number. b: The second binary number. Returns: The sum of the two binary numbers. """ # Convert the binary strings to numbers. a = int(a, 2) b = int(b, 2) # Add the numbers. sum = a + b # Convert the sum back to a binary string. return bin(sum)[2:] add_binary('10011111110100011100011011110010', '00111010100101000100000101111011')
    - rcme 3y ago
      I feel like we're missing the forest for the trees here. I'm not really asking it to add two numbers. I'm asking it to apply a simple set of logical rules on two symbols with an operation '+'. Of course it can generate code to add two numbers, because the code to add two numbers exists in its training data. What transformers seem to be good at is generalizing over the training data. Where they seem to perform poorly is data outside of the training set. Binary numbers are interesting because you can add a bit of entropy just by increasing the size of your binary number, so it's very easy to generate examples outside of the training data. You also see this, albeit less directly, when trying to ask it to generate code that wasn't part of its training data. E.g. https://twitter.com/cHHillee/status/1635790330854526981 https://twitter.com/cHHillee/status/1635790330854526981

by rcme 3y ago

sdenton4 3y ago

Perhaps also consider that you (the human) are using 2-dimensional visual processing to help you solve the problem by collating your work. You also almost certainly work /backwards/ from right to left. The problem is simplified by scratch space and the human sensorium... (It would be interesting to see how multimodal models handle the problem, actually, given a picture of the addition problem.) The problem given to the LLM is more like: 1 0 0 1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 0 0 1 1 0 1 1 1 1 0 0 1 0 + 0 0 1 1 1 0 1 0 1 0 0 1 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 1 1 0 1 1 = ? Try writing the answer entirely from left-to-right without losing your place... It'll be error prone for humans, to say the least. It's why we invented our particular by-hand addition algorithms in the first place: these algorithms are co-adapted to the human sensorium. Luckily, we can ask any LLM worth its salt to write the single line of python code needed to solve the problem and it'll do just fine... edit: Thinking about this more - and trying to explicitly check the intermediate steps - I'm guessing that the LLM is getting lost in the inputs. The attention mechanism is going to need to find the correct pair of digits at each step, and combine them with some info about the carry variable. I'm guessing this is hard to do - there's a positional embedding used to attach some location info to each token, but in a long run of homogenous text, I would guess that it's hard to find a particular digit and end up with some off-by-one errors in the lookup. (and this is very similar to how we would expect humans to fail at the single-line version of the problem.)

rcme 3y ago
I wouldn't expect a human to fail at the single line version. It's pretty easy to do. Write a list, at each position n in the list, compute the sum of the nth least-significant bit and the carry from the line above. You can easily get the nth least significant bit by counting from the right side of each number. The end result is the list printed backwards.
- shagie 3y ago
  Consider the amount of work that you need to do to reverse a word in GPT. https://twitter.com/npew/status/1525900868305833984?lang=en# https://twitter.com/npew/status/1525900868305833984?lang=en# Note that part of the trick that they did to get it to work better is putting spaces around each letter to enforce a particular tokenization. Trying to do the same with mathematical objects rather than language objects would be even harder because GPT doesn't "know" math.
- sdenton4 3y ago
  You lost the thread at 'write.' Try to do it in your head; we pretty much have to resort to additional tools to succeed at this task. Here's a LLM-generated python solution, which works just fine: def add_binary(a, b): """Adds two binary numbers given as strings. Args: a: The first binary number. b: The second binary number. Returns: The sum of the two binary numbers. """ # Convert the binary strings to numbers. a = int(a, 2) b = int(b, 2) # Add the numbers. sum = a + b # Convert the sum back to a binary string. return bin(sum)[2:] add_binary('10011111110100011100011011110010', '00111010100101000100000101111011')
  - rcme 3y ago
    I feel like we're missing the forest for the trees here. I'm not really asking it to add two numbers. I'm asking it to apply a simple set of logical rules on two symbols with an operation '+'. Of course it can generate code to add two numbers, because the code to add two numbers exists in its training data. What transformers seem to be good at is generalizing over the training data. Where they seem to perform poorly is data outside of the training set. Binary numbers are interesting because you can add a bit of entropy just by increasing the size of your binary number, so it's very easy to generate examples outside of the training data. You also see this, albeit less directly, when trying to ask it to generate code that wasn't part of its training data. E.g. https://twitter.com/cHHillee/status/1635790330854526981 https://twitter.com/cHHillee/status/1635790330854526981