5 ms·

It will, however from time to time insert lines and variables that do nothing, but could result in bugs or confusion if not removed. I’ve encountered these hall

It will, however from time to time insert lines and variables that do nothing, but could result in bugs or confusion if not removed. I’ve encountered these hallucinations a few times. Overall, I agree with your sentiment, but I think it’s important to note that running isn’t always the indicator or correctness we think it to be.

BbzzbB 3y ago
You're still supposed to read it, like you hopefully wouldn't blindly paste a big code block from SO. A useless/unused line or variable doesn't seem that hard to spot?
- bugglebeetle 3y ago
  Of course, but ease of detection can vary relative to the complexity of the code being returned. GPT-4, correctly prompted, can produce some pretty complicated stuff. But it also hallucinates in ways that are more subtle than one might think. The example I’m thinking of, it created an unused variable in a set of fairly complex ML training set up scripts that I mostly caught because I was familiar with all the proper inputs. But the unused variable was quite plausible if you were not familiar, new to the domain etc.
  - gowld 3y ago
    Compilers automatically detect unused variables. Unused variables are the last of your problems. You should be far, far more worried about all the misused variables.
JCharante 3y ago
Don't IDEs highlight variables that aren't read/written to?
- lobsterthief 3y ago
  In this case “unused” might mean declaring or initializing a variable and then assigning or reassigning a value to it later. In this case, it is technically used, so most linters won’t pick that up. But it actually does nothing so it’s wasted cycles.
  - gowld 3y ago
    No decent compiler would waste a cycle on that.
ChatGTP 3y ago
I've noticed some other funny things, maybe harmless but undesirable. One example was really really hard to spot. Once I queried GPT 3.5 for a function to do X, and it did pretty well, when I looked closer though, it had wrapped 90% of the code in an unnecessary if statement. I looked at the code an thought something was off until I realized. My point here is, if that was easy to spot, who knows what else people are missing because even in a simple case, unless you're actually trying to spot issues, you likely won't see them.

BbzzbB 3y ago

You're still supposed to read it, like you hopefully wouldn't blindly paste a big code block from SO. A useless/unused line or variable doesn't seem that hard to spot?

bugglebeetle 3y ago
Of course, but ease of detection can vary relative to the complexity of the code being returned. GPT-4, correctly prompted, can produce some pretty complicated stuff. But it also hallucinates in ways that are more subtle than one might think. The example I’m thinking of, it created an unused variable in a set of fairly complex ML training set up scripts that I mostly caught because I was familiar with all the proper inputs. But the unused variable was quite plausible if you were not familiar, new to the domain etc.
- gowld 3y ago
  Compilers automatically detect unused variables. Unused variables are the last of your problems. You should be far, far more worried about all the misused variables.

JCharante 3y ago

Don't IDEs highlight variables that aren't read/written to?

lobsterthief 3y ago
In this case “unused” might mean declaring or initializing a variable and then assigning or reassigning a value to it later. In this case, it is technically used, so most linters won’t pick that up. But it actually does nothing so it’s wasted cycles.
- gowld 3y ago
  No decent compiler would waste a cycle on that.

ChatGTP 3y ago

I've noticed some other funny things, maybe harmless but undesirable. One example was really really hard to spot. Once I queried GPT 3.5 for a function to do X, and it did pretty well, when I looked closer though, it had wrapped 90% of the code in an unnecessary if statement. I looked at the code an thought something was off until I realized. My point here is, if that was easy to spot, who knows what else people are missing because even in a simple case, unless you're actually trying to spot issues, you likely won't see them.