3 ms·

> “LLMs can’t even do addition” > We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no roo

> “LLMs can’t even do addition” > We show that they CAN add! To teach algos to LLMs, the trick is to describe the algo in enough detail so that there is no room for misinterpretation Algorithmic prompting paper https://arxiv.org/pdf/2211.09066.pdf https://arxiv.org/pdf/2211.09066.pdf

rpearl 4y ago
> the trick is to describe the algo in enough detail so that there is no room for misinterpretation We could create a syntactically unambiguous form of textual representation of instructions. We could call such a piece of text something specific, like a "program".
- visarga 4y ago
  Yes, but this paper showed there is no intrinsic problem with the LM, it can execute an algorithm like addition correctly. It's billions of times more inefficient - with 175B parameters it has to do at least 175B multiplications for each output token. Incredibly inefficient, but still of interest because it could be applied to "soft concepts", unlike computer code.
rav 4y ago
Without reading the linked paper, I thought to myself "How would I ask ChatGPT to run an addition program?" So I tried the following prompt: > Imagine a finite state machine with two states A and B. The machine reads a long list of digit-pairs, one pair at a time, and depending on its current state and the digit pair, it outputs some digit and it transitions to a new state. > When the machine is in state A and it reads a pair of digits, it adds together the two digits. If the sum is less than 10, it outputs the single-digit sum and it stays in state A. Otherwise, it outputs the least significant digit of the sum and it goes to state B. > When the machine is in state B and it reads a pair of digits, it adds together the two digits and adds 1 to the sum. If the sum is less than 10, it outputs the single-digit sum and it goes to state A. Otherwise, it outputs the least significant digit of the sum and it stays in state B. > With the machine in state A, and the two input digits being 4 and 4, what is the output digit, and what is the next state? Unfortunately, ChatGPT failed to apply the transition rule correctly - its output: > When the machine is in state A and it reads the digit pair (4, 4), it adds the two digits together to get 8. Since 8 is greater than 10, the machine outputs the least significant digit of the sum, which is 8. The machine then transitions to state B. > So the output digit is 8 and the next state is B. It seems like my approach is "instruction-only" if I try to follow the paper's nomenclature, but the paper doesn't seem to give examples of what their "instruction-only" prompts look like - those would be interesting to try to feed into ChatGPT.

rpearl 4y ago

> the trick is to describe the algo in enough detail so that there is no room for misinterpretation We could create a syntactically unambiguous form of textual representation of instructions. We could call such a piece of text something specific, like a "program".

visarga 4y ago
Yes, but this paper showed there is no intrinsic problem with the LM, it can execute an algorithm like addition correctly. It's billions of times more inefficient - with 175B parameters it has to do at least 175B multiplications for each output token. Incredibly inefficient, but still of interest because it could be applied to "soft concepts", unlike computer code.

rav 4y ago

Without reading the linked paper, I thought to myself "How would I ask ChatGPT to run an addition program?" So I tried the following prompt: > Imagine a finite state machine with two states A and B. The machine reads a long list of digit-pairs, one pair at a time, and depending on its current state and the digit pair, it outputs some digit and it transitions to a new state. > When the machine is in state A and it reads a pair of digits, it adds together the two digits. If the sum is less than 10, it outputs the single-digit sum and it stays in state A. Otherwise, it outputs the least significant digit of the sum and it goes to state B. > When the machine is in state B and it reads a pair of digits, it adds together the two digits and adds 1 to the sum. If the sum is less than 10, it outputs the single-digit sum and it goes to state A. Otherwise, it outputs the least significant digit of the sum and it stays in state B. > With the machine in state A, and the two input digits being 4 and 4, what is the output digit, and what is the next state? Unfortunately, ChatGPT failed to apply the transition rule correctly - its output: > When the machine is in state A and it reads the digit pair (4, 4), it adds the two digits together to get 8. Since 8 is greater than 10, the machine outputs the least significant digit of the sum, which is 8. The machine then transitions to state B. > So the output digit is 8 and the next state is B. It seems like my approach is "instruction-only" if I try to follow the paper's nomenclature, but the paper doesn't seem to give examples of what their "instruction-only" prompts look like - those would be interesting to try to feed into ChatGPT.