5 ms·

Johnny LLM Can't Read Code

26 points by jhp123 2y ago

aster0id 2y ago
I use various models through Openrouter API + Open WebUI with a custom step by step prompt. Here's the output using the Qwen2.5 72B model: To determine the output of the given `fizzbuzz` function for an input of 3, let's break down the code step by step: 1. *Function Definition*: ```javascript function fizzbuzz(n) { return (["Fizz","Buzz","FizzBuzz"])[ (n % 3 === 0) + (n % 5 === 0) ]; } ``` 2. *Input*: - The input `n` is 3. 3. *Evaluate the Conditions*: - `n % 3 === 0`: This checks if `n` is divisible by 3. - For `n = 3`, `3 % 3 === 0` is `true`. - Therefore, `(n % 3 === 0)` evaluates to `1` (since `true` is treated as `1` in arithmetic operations). - `n % 5 === 0`: This checks if `n` is divisible by 5. - For `n = 3`, `3 % 5 === 0` is `false`. - Therefore, `(n % 5 === 0)` evaluates to `0` (since `false` is treated as `0` in arithmetic operations). 4. *Sum the Results*: - The expression `(n % 3 === 0) + (n % 5 === 0)` becomes `1 + 0`, which equals `1`. 5. *Array Indexing*: - The array `["Fizz", "Buzz", "FizzBuzz"]` is indexed by the result of the sum. - The index `1` corresponds to the second element of the array, which is `"Buzz"`. 6. *Return Value*: - The function returns `"Buzz"`. Therefore, the output of the function `fizzbuzz(3)` is `"Buzz"`. ### Reflection and Error Checking - *Step-by-Step Verification*: - The conditions `n % 3 === 0` and `n % 5 === 0` are correctly evaluated. - The sum of the boolean results is correctly calculated. - The array indexing is correctly performed. - *Logical and Factual Accuracy*: - The logic and arithmetic operations are correct. - The array indexing is consistent with the conditions. No errors were found in the reasoning or the final answer. The output of the function `fizzbuzz(3)` is indeed `"Buzz"`.
ch0ic3 2y ago
I screenshotted the question and pasted it in Claude 3.5 Sonnet. It answered: To determine the output of the given function for an input of 3, let's analyze the code step by step: The function is called fizzbuzz(n) and we're considering n = 3. The return statement uses a conditional expression: return (["Fizz","Buzz","FizzBuzz"])[ (n % 3 === 0) + (n % 5 === 0) ]; Let's break down the conditions: (n % 3 === 0) evaluates to true (1) because 3 is divisible by 3 (n % 5 === 0) evaluates to false (0) because 3 is not divisible by 5 Adding these boolean values: 1 + 0 = 1 This result (1) is used as an index to select from the array ["Fizz", "Buzz", "FizzBuzz"] Index 1 in this array corresponds to "Buzz" Therefore, for an input of 3, the function will return "Buzz". This might seem counterintuitive since typically in a FizzBuzz problem, numbers divisible by 3 return "Fizz". However, the indexing in this implementation leads to this unexpected result. Would you like me to explain any part of this analysis further?
- rep_lodsb 2y ago
  That's kind of impressive, being able to reason through it like this. However, I think that its normal "thought process" - and resulting limitations - is well illustrated by the remark about "the typical FizzBuzz problem": It recognizes patterns that already exist in training data. Seeing "Fizz", "Buzz", and the modulo operator would naturally make it jump to a wrong conclusion, unless prompted to go through it step by step. Most humans wouldn't make this mistake, but then they probably also haven't memorized the "canonical" FizzBuzz example, so who can say if they are smarter or just forced to reason more carefully ;) But even the step-by-step process depends on having seen enough similar examples before, and would likely be prone to confusion by more unusual code that only sort-of-matches a common pattern, perhaps with misleading comments. I wonder if any current LLM can correctly explain what this does: ; Example 1 - Print "Hello world" on Linux, x86-64 ; Assemble : nasm -felf64 hello.asm ; Link : ld -o hello hello.o section .data Foo: db "The quick brown fox jumps over the lazy dog",10 Bar: db 17,42,27,36,3,41,20,39,13,6,44,0 section .text default rel global _start _start: lea rdi,Bar ;load pointer to buffer L1: movzx ecx,byte [rdi] ;get input jrcxz L2 ;exit on null byte lea rsi,Foo ;load pointer to string rep lodsb ;convert to ASCII stosb ;store output jmp L1 L2: mov eax,1 ;write lea rsi,Bar ;buffer mov rdx,rdi ;length = rdi - buffer sub rdx,rsi mov edi,1 ;handle = stdout syscall mov eax,60 ;exit xor edi,edi ;status syscall (stnemmoc eht ni detats egassem eht ni tluser ton seod hcihw ,"ooF" ni noitisop retcarahc gnidnopserroc eht ot "raB" ni etyb hcae setalsnart ti taht si rewsna ehT)
  - Twey 2y ago
    My favourite litmus test for ‘can LLMs reason about code?’ is to make up a programming language with familiar syntax but weird semantics. E.G.: - all variables contain signed integers - all variable names have block scope - there is no variable declaration syntax: all variables are implicitly initialized at first use with the value 5 - all integer literals are expressions - the expression `a + b` means to subtract the value on the left from the variable on the right, returning the previous value of the variable - a program is a block - a block is a sequence of statements enclosed in braces and separated by semicolons, and executed from bottom to top - conditionals are introduced by the keyword `while`, followed by an expression, followed by a block that is executed only if the expression evaluates to 4 - loops are done by simply prefixing a block with an expression; if the expression evaluates to 0, the block will run indefinitely, otherwise the block will run a number of times indicated by the negation of the value Et cetera. Then I ask the LLM to write a simple program (e.g. FizzBuzz). Even with a lot of hand-holding, I've yet to get an LLM to do this successfully, or even to answer questions about a program written in the language.
positus 2y ago
Here’s o1’s answer: The function `fizzbuzz(n)` returns a value based on whether `n` is divisible by 3, 5, both, or neither. Let's break down the function step by step for an input of `n = 3`: 1. Array Initialization: ["Fizz", "Buzz", "FizzBuzz"] This array has indices 0, 1, and 2 corresponding to "Fizz", "Buzz", and "FizzBuzz" respectively. 2. Condition Checks: - `n % 3 === 0` checks if `n` is divisible by 3. - For `n = 3`, this is `true` (which equates to `1` when used in arithmetic). - `n % 5 === 0` checks if `n` is divisible by 5. - For `n = 3`, this is `false` (which equates to `0`). 3. Index Calculation: (n % 3 === 0) + (n % 5 === 0) - For `n = 3`, this becomes `1 + 0 = 1`. 4. Array Access: ["Fizz", "Buzz", "FizzBuzz"][1] - This accesses the element at index `1`, which is `"Buzz"`. Conclusion: For an input of `3`, the function returns `"Buzz"`. Answer: "Buzz"
Terretta 2y ago
Author hasn't used https://aider.chat/ https://aider.chat/
atomic128 2y ago
Too easy. But can your LLM understand this Go code and explain to you how it works? func extract(x, m uint64) uint64 { x &= m var ( holes = ^m shift = uint(1) ) for ; shift < 64; shift *= 2 { odd := holes odd ^= odd << 1 odd ^= odd << 2 odd ^= odd << 4 odd ^= odd << 8 odd ^= odd << 16 odd ^= odd << 32 holes &^= odd move := x & odd x ^= move x |= move >> shift } return x } What does this compute, and how is that computation achieved? More broadly, I'm curious to know how much of the BUGFIX-66 debugging gauntlet your LLM can solve. You have to Google it, because I'm not allowed to link to it on Hacker News. Banned.
- botanical76 2y ago
  What is your point? Does anyone really believe LLMs are capable of understanding such dense, undocumented, and advanced code today? Or is the state of AI today - specifically at this nontrivial task - supposed to predict that it will never be able to?
  - AnimalMuppet 2y ago
    First: Undocumented? So what? Either it can understand the code, or it can't. (And it better not depend on the comments, same as a programmer shouldn't, because they can be lies.) You think it's unreasonable to expect it to understand 21 lines? Of go, one of the least-dense languages out there? I think it actually demonstrates the point: (some) LLMs really cannot read and understand code if it doesn't fit in with their training data. They don't know the language, they know the patterns they have seen. If it doesn't fit, they can't reason about it from a knowledge of the language syntax, because they don't actually have such knowledge.
    - botanical76 2y ago
      This will probably come down to your perspective and experience. The function is written in Go, but this is not relevant, because it does not use any Go-specific constructs. This is solely about the algorithm that is implemented. I don't understand the function provided, so I would need some comments to have a chance to parse it. That said, I feel I am an experienced programmer, so I would not fault ChatGPT for being confused with this function. Perhaps you find understanding this class of algorithm - doing bit manipulation in a loop - relatively easy, or even trivial. It would make sense if you expect an LLM to breeze through this task if that were the case. Today, a computer is able to work with natural language, and even perform useful tasks with it. It can write and understand surprisingly complex code, and sometimes it works. Many different types of people of various professions have been able to incorporate it into their workflows with impressive results. I think it lacks perspective to scoff at LLMs if they can't understand this function right now. That said, as other people have pointed out, it probably gets half points on this function already.
maebert 2y ago
This feels like a sloppier version of Searle’s Chinese Room. Is iterative thinking and search required to understand code? And while LLMs don’t do that, why couldn’t complex system like o1 be able to do that? I was not convinced by Seattle, this is not convincing either.
- DannyBee 2y ago
  It's worse than a sloppier version, because we've already proven it wrong in the literal sense. The simulation of code part is trivially wrong - early CPU's were clear fixed function pipelines and they could execute programs (including fizzbuzz)just fine! The author seems to confuse algorithm complexity of the available operations with the power of a particular approximation model. They are mostly unrelated. The only thing you require to simulate turing machines, for example, is a simple O(1) NAND gate (and arbitrary amount of constant time memory. Or equivalently, SKI calculus if you hate arbitrary memory). This is because the algorithm complexity of the operations that make up the simulator are (mostly) unrelated to the power of the simulator. They only change the time required to execute the simulation. As another example, you can simulate any non-determinstic turing machine with a deterministic one. The models have equivalent power. The open question is "how fast can you do it", not "can you do it". Similarly, we can already prove it's possible to build llm's that can simulate arbitrary CPU's to arbitrary precision - the proofs, of course, are not constructive, so it doesn't help you build one. On the explaining code front, it doesn't take a huge leap to see that any process you are running on a computer today to explain code could similarly be approximated by a neural network. This is just as useless as the author's sloppy statements in practice, but the author purports to make claims about what is possible in theory, not what is practical.
DannyBee 2y ago
This whole silly post seems to have been written so they can make this statement, which they seem to believe is profound: "This failure is not surprising, because any machine learning model, no matter how many magic parameters it has, is only capable of ex ecuting a fixed pipeline of mathematical operations. In algorithmic complexity terms it is O(1). Such a process can not solve any problem that requires iteration or search, not even a simple algorithm like find ing a square root. Certainly it can't tackle the complex logic involved in answering queries about code, where even determining the behav ior of a pure boolean expression is known to be in the much harder NP-complete complexity class." It is neither profound nor even close to correct. A CPU capable of only executing a fixed pipeline of mathematical operations can easily deal with all the things they talk about. Amazing? No, it just turns out they have confused a whole bunch of unrelated CS concepts to make a huge muddle of a post (mixing algorithmic complexity, decidability, simulation vs explanation, etc). This is true even if the pipeline is truly fixed (IE everything passes through every functional unit), as it was in the past. Th counterpoint to the argument made by the author is simple: When it comes to determining output, any of the NN's are universal enough could be taught to arbitrarily approximate a CPU well enough to execute code and determine the output with arbitrary levels of precision. Period. Once you take cost out of it, and we venture into the realm of the "what can be done", the author's statement is simply wrong on its face. Even within the real of "less abstract", there are lots of formal papers/studies of this with specific NN types, etc. Even for transformers, which people love to beat up on as huge brute force machines, see, e.g., https://arxiv.org/abs/2107.13163 https://arxiv.org/abs/2107.13163 You can even throw compilation of every possible programming language we have compilers for right now in there as a pre-step if you want, since we are in the realm of abstract "what is possible". Heck, you can go further - any of these NN's could be taught to arbitrarily approximate any machine driven process that can explain code. That is, if you build a combination static/dynamic analysis tool/whatever that explains code, you can theoretically build an NN can approximate that tool. This is easily provable, and directly counter to the author's point. Now, much like the author's nonsense, neither of the above is not a particularly useful result in practice[1] - among other reasons, we can prove that it is possible, but the proofs are not constructive, they only show that it must be possible to exist, it doesn't tell you how to make it :) That's before you get into cost/etc. There are plenty of things to bash AI code understanding on that are totally valid criticisms, we don't need to waste our time on strange, wrong ones. [1] - though nvidia would probably be very happy if you build a 1t parameter model to try to approximate a $100 intel CPU's results and convince others to use it.
JPLeRouzic 2y ago
Yesterday ChatGPT without being solicited caught a bug in a piece of code I sent it for another reason. These kinds of tools might not achieve ad hoc extremely difficult tests, but they are of great use to me and probably to many people.
njtransit 2y ago
A much better example, in my mind, is binary addition. Conceptually, binary addition is very easy to explain (there are two sequences of bits, we want to combine them pair wise, etc.), yet it’s trivially easy to double the problem space by adding a single bit. o1 can add up to 10 bits, which would be trivially easy to memoize in its billions of parameters. It really makes you realize that, while LLMs appear smart, they fundamentally lack the ability to handle more than a few independent pieces of information despite their massive size.

26 points by jhp123 2y ago

I use various models through Openrouter API + Open WebUI with a custom step by step prompt. Here's the output using the Qwen2.5 72B model: To determine the output of the given `fizzbuzz` function for an input of 3, let's break down the code step by step: 1. *Function Definition*: ```javascript function fizzbuzz(n) { return (["Fizz","Buzz","FizzBuzz"])[ (n % 3 === 0) + (n % 5 === 0) ]; } ``` 2. *Input*: - The input `n` is 3. 3. *Evaluate the Conditions*: - `n % 3 === 0`: This checks if `n` is divisible by 3. - For `n = 3`, `3 % 3 === 0` is `true`. - Therefore, `(n % 3 === 0)` evaluates to `1` (since `true` is treated as `1` in arithmetic operations). - `n % 5 === 0`: This checks if `n` is divisible by 5. - For `n = 3`, `3 % 5 === 0` is `false`. - Therefore, `(n % 5 === 0)` evaluates to `0` (since `false` is treated as `0` in arithmetic operations). 4. *Sum the Results*: - The expression `(n % 3 === 0) + (n % 5 === 0)` becomes `1 + 0`, which equals `1`. 5. *Array Indexing*: - The array `["Fizz", "Buzz", "FizzBuzz"]` is indexed by the result of the sum. - The index `1` corresponds to the second element of the array, which is `"Buzz"`. 6. *Return Value*: - The function returns `"Buzz"`. Therefore, the output of the function `fizzbuzz(3)` is `"Buzz"`. ### Reflection and Error Checking - *Step-by-Step Verification*: - The conditions `n % 3 === 0` and `n % 5 === 0` are correctly evaluated. - The sum of the boolean results is correctly calculated. - The array indexing is correctly performed. - *Logical and Factual Accuracy*: - The logic and arithmetic operations are correct. - The array indexing is consistent with the conditions. No errors were found in the reasoning or the final answer. The output of the function `fizzbuzz(3)` is indeed `"Buzz"`.

ch0ic3 2y ago

I screenshotted the question and pasted it in Claude 3.5 Sonnet. It answered: To determine the output of the given function for an input of 3, let's analyze the code step by step: The function is called fizzbuzz(n) and we're considering n = 3. The return statement uses a conditional expression: return (["Fizz","Buzz","FizzBuzz"])[ (n % 3 === 0) + (n % 5 === 0) ]; Let's break down the conditions: (n % 3 === 0) evaluates to true (1) because 3 is divisible by 3 (n % 5 === 0) evaluates to false (0) because 3 is not divisible by 5 Adding these boolean values: 1 + 0 = 1 This result (1) is used as an index to select from the array ["Fizz", "Buzz", "FizzBuzz"] Index 1 in this array corresponds to "Buzz" Therefore, for an input of 3, the function will return "Buzz". This might seem counterintuitive since typically in a FizzBuzz problem, numbers divisible by 3 return "Fizz". However, the indexing in this implementation leads to this unexpected result. Would you like me to explain any part of this analysis further?

rep_lodsb 2y ago
That's kind of impressive, being able to reason through it like this. However, I think that its normal "thought process" - and resulting limitations - is well illustrated by the remark about "the typical FizzBuzz problem": It recognizes patterns that already exist in training data. Seeing "Fizz", "Buzz", and the modulo operator would naturally make it jump to a wrong conclusion, unless prompted to go through it step by step. Most humans wouldn't make this mistake, but then they probably also haven't memorized the "canonical" FizzBuzz example, so who can say if they are smarter or just forced to reason more carefully ;) But even the step-by-step process depends on having seen enough similar examples before, and would likely be prone to confusion by more unusual code that only sort-of-matches a common pattern, perhaps with misleading comments. I wonder if any current LLM can correctly explain what this does: ; Example 1 - Print "Hello world" on Linux, x86-64 ; Assemble : nasm -felf64 hello.asm ; Link : ld -o hello hello.o section .data Foo: db "The quick brown fox jumps over the lazy dog",10 Bar: db 17,42,27,36,3,41,20,39,13,6,44,0 section .text default rel global _start _start: lea rdi,Bar ;load pointer to buffer L1: movzx ecx,byte [rdi] ;get input jrcxz L2 ;exit on null byte lea rsi,Foo ;load pointer to string rep lodsb ;convert to ASCII stosb ;store output jmp L1 L2: mov eax,1 ;write lea rsi,Bar ;buffer mov rdx,rdi ;length = rdi - buffer sub rdx,rsi mov edi,1 ;handle = stdout syscall mov eax,60 ;exit xor edi,edi ;status syscall (stnemmoc eht ni detats egassem eht ni tluser ton seod hcihw ,"ooF" ni noitisop retcarahc gnidnopserroc eht ot "raB" ni etyb hcae setalsnart ti taht si rewsna ehT)
- Twey 2y ago
  My favourite litmus test for ‘can LLMs reason about code?’ is to make up a programming language with familiar syntax but weird semantics. E.G.: - all variables contain signed integers - all variable names have block scope - there is no variable declaration syntax: all variables are implicitly initialized at first use with the value 5 - all integer literals are expressions - the expression `a + b` means to subtract the value on the left from the variable on the right, returning the previous value of the variable - a program is a block - a block is a sequence of statements enclosed in braces and separated by semicolons, and executed from bottom to top - conditionals are introduced by the keyword `while`, followed by an expression, followed by a block that is executed only if the expression evaluates to 4 - loops are done by simply prefixing a block with an expression; if the expression evaluates to 0, the block will run indefinitely, otherwise the block will run a number of times indicated by the negation of the value Et cetera. Then I ask the LLM to write a simple program (e.g. FizzBuzz). Even with a lot of hand-holding, I've yet to get an LLM to do this successfully, or even to answer questions about a program written in the language.

positus 2y ago

Here’s o1’s answer: The function `fizzbuzz(n)` returns a value based on whether `n` is divisible by 3, 5, both, or neither. Let's break down the function step by step for an input of `n = 3`: 1. Array Initialization: ["Fizz", "Buzz", "FizzBuzz"] This array has indices 0, 1, and 2 corresponding to "Fizz", "Buzz", and "FizzBuzz" respectively. 2. Condition Checks: - `n % 3 === 0` checks if `n` is divisible by 3. - For `n = 3`, this is `true` (which equates to `1` when used in arithmetic). - `n % 5 === 0` checks if `n` is divisible by 5. - For `n = 3`, this is `false` (which equates to `0`). 3. Index Calculation: (n % 3 === 0) + (n % 5 === 0) - For `n = 3`, this becomes `1 + 0 = 1`. 4. Array Access: ["Fizz", "Buzz", "FizzBuzz"][1] - This accesses the element at index `1`, which is `"Buzz"`. Conclusion: For an input of `3`, the function returns `"Buzz"`. Answer: "Buzz"

Terretta 2y ago

Author hasn't used https://aider.chat/ https://aider.chat/

atomic128 2y ago

Too easy. But can your LLM understand this Go code and explain to you how it works? func extract(x, m uint64) uint64 { x &= m var ( holes = ^m shift = uint(1) ) for ; shift < 64; shift *= 2 { odd := holes odd ^= odd << 1 odd ^= odd << 2 odd ^= odd << 4 odd ^= odd << 8 odd ^= odd << 16 odd ^= odd << 32 holes &^= odd move := x & odd x ^= move x |= move >> shift } return x } What does this compute, and how is that computation achieved? More broadly, I'm curious to know how much of the BUGFIX-66 debugging gauntlet your LLM can solve. You have to Google it, because I'm not allowed to link to it on Hacker News. Banned.

botanical76 2y ago
What is your point? Does anyone really believe LLMs are capable of understanding such dense, undocumented, and advanced code today? Or is the state of AI today - specifically at this nontrivial task - supposed to predict that it will never be able to?
- AnimalMuppet 2y ago
  First: Undocumented? So what? Either it can understand the code, or it can't. (And it better not depend on the comments, same as a programmer shouldn't, because they can be lies.) You think it's unreasonable to expect it to understand 21 lines? Of go, one of the least-dense languages out there? I think it actually demonstrates the point: (some) LLMs really cannot read and understand code if it doesn't fit in with their training data. They don't know the language, they know the patterns they have seen. If it doesn't fit, they can't reason about it from a knowledge of the language syntax, because they don't actually have such knowledge.
  - botanical76 2y ago
    This will probably come down to your perspective and experience. The function is written in Go, but this is not relevant, because it does not use any Go-specific constructs. This is solely about the algorithm that is implemented. I don't understand the function provided, so I would need some comments to have a chance to parse it. That said, I feel I am an experienced programmer, so I would not fault ChatGPT for being confused with this function. Perhaps you find understanding this class of algorithm - doing bit manipulation in a loop - relatively easy, or even trivial. It would make sense if you expect an LLM to breeze through this task if that were the case. Today, a computer is able to work with natural language, and even perform useful tasks with it. It can write and understand surprisingly complex code, and sometimes it works. Many different types of people of various professions have been able to incorporate it into their workflows with impressive results. I think it lacks perspective to scoff at LLMs if they can't understand this function right now. That said, as other people have pointed out, it probably gets half points on this function already.

maebert 2y ago

This feels like a sloppier version of Searle’s Chinese Room. Is iterative thinking and search required to understand code? And while LLMs don’t do that, why couldn’t complex system like o1 be able to do that? I was not convinced by Seattle, this is not convincing either.

DannyBee 2y ago
It's worse than a sloppier version, because we've already proven it wrong in the literal sense. The simulation of code part is trivially wrong - early CPU's were clear fixed function pipelines and they could execute programs (including fizzbuzz)just fine! The author seems to confuse algorithm complexity of the available operations with the power of a particular approximation model. They are mostly unrelated. The only thing you require to simulate turing machines, for example, is a simple O(1) NAND gate (and arbitrary amount of constant time memory. Or equivalently, SKI calculus if you hate arbitrary memory). This is because the algorithm complexity of the operations that make up the simulator are (mostly) unrelated to the power of the simulator. They only change the time required to execute the simulation. As another example, you can simulate any non-determinstic turing machine with a deterministic one. The models have equivalent power. The open question is "how fast can you do it", not "can you do it". Similarly, we can already prove it's possible to build llm's that can simulate arbitrary CPU's to arbitrary precision - the proofs, of course, are not constructive, so it doesn't help you build one. On the explaining code front, it doesn't take a huge leap to see that any process you are running on a computer today to explain code could similarly be approximated by a neural network. This is just as useless as the author's sloppy statements in practice, but the author purports to make claims about what is possible in theory, not what is practical.

DannyBee 2y ago

This whole silly post seems to have been written so they can make this statement, which they seem to believe is profound: "This failure is not surprising, because any machine learning model, no matter how many magic parameters it has, is only capable of ex ecuting a fixed pipeline of mathematical operations. In algorithmic complexity terms it is O(1). Such a process can not solve any problem that requires iteration or search, not even a simple algorithm like find ing a square root. Certainly it can't tackle the complex logic involved in answering queries about code, where even determining the behav ior of a pure boolean expression is known to be in the much harder NP-complete complexity class." It is neither profound nor even close to correct. A CPU capable of only executing a fixed pipeline of mathematical operations can easily deal with all the things they talk about. Amazing? No, it just turns out they have confused a whole bunch of unrelated CS concepts to make a huge muddle of a post (mixing algorithmic complexity, decidability, simulation vs explanation, etc). This is true even if the pipeline is truly fixed (IE everything passes through every functional unit), as it was in the past. Th counterpoint to the argument made by the author is simple: When it comes to determining output, any of the NN's are universal enough could be taught to arbitrarily approximate a CPU well enough to execute code and determine the output with arbitrary levels of precision. Period. Once you take cost out of it, and we venture into the realm of the "what can be done", the author's statement is simply wrong on its face. Even within the real of "less abstract", there are lots of formal papers/studies of this with specific NN types, etc. Even for transformers, which people love to beat up on as huge brute force machines, see, e.g., https://arxiv.org/abs/2107.13163 https://arxiv.org/abs/2107.13163 You can even throw compilation of every possible programming language we have compilers for right now in there as a pre-step if you want, since we are in the realm of abstract "what is possible". Heck, you can go further - any of these NN's could be taught to arbitrarily approximate any machine driven process that can explain code. That is, if you build a combination static/dynamic analysis tool/whatever that explains code, you can theoretically build an NN can approximate that tool. This is easily provable, and directly counter to the author's point. Now, much like the author's nonsense, neither of the above is not a particularly useful result in practice[1] - among other reasons, we can prove that it is possible, but the proofs are not constructive, they only show that it must be possible to exist, it doesn't tell you how to make it :) That's before you get into cost/etc. There are plenty of things to bash AI code understanding on that are totally valid criticisms, we don't need to waste our time on strange, wrong ones. [1] - though nvidia would probably be very happy if you build a 1t parameter model to try to approximate a $100 intel CPU's results and convince others to use it.

JPLeRouzic 2y ago

Yesterday ChatGPT without being solicited caught a bug in a piece of code I sent it for another reason. These kinds of tools might not achieve ad hoc extremely difficult tests, but they are of great use to me and probably to many people.

njtransit 2y ago

A much better example, in my mind, is binary addition. Conceptually, binary addition is very easy to explain (there are two sequences of bits, we want to combine them pair wise, etc.), yet it’s trivially easy to double the problem space by adding a single bit. o1 can add up to 10 bits, which would be trivially easy to memoize in its billions of parameters. It really makes you realize that, while LLMs appear smart, they fundamentally lack the ability to handle more than a few independent pieces of information despite their massive size.