4 ms·

I'm really skeptical of using current LLMs for judging codebases like this. Just today I got Gemini to solve a tricky bug, but it only worked after providing it

I'm really skeptical of using current LLMs for judging codebases like this. Just today I got Gemini to solve a tricky bug, but it only worked after providing it more debug output after solving some of it myself. The first time I tried without the deeper output, it "solved" it by writing a load of code that failed in loads of other ways, and ended up not even being related to the actual issue. Like you can be certain it'll give you some nice looking metrics and measurements - but how do you know if they're accurate?

the_mitsuhiko 1y ago
> I'm really skeptical of using current LLMs for judging codebases like this. I'm not necessarily convinced that the current generation of LLMs are overly amazing at this, but they definitely are very good at measuring inefficiency of tooling and problematic APIs. That's not all the issues, but it can at least be useful to evaluate some classes of problems.
falcor84 1y ago
What do you mean that it "ended up not even being related to the actual issue"? If you give it a failing test suite to turn green and it does, then either its solution is indeed related to the issue, or your tests are incomplete; so you improve the tests and try again, right? Or am I missing something?
- GardenLetter27 1y ago
  It made the other tests fail, I wasn't using it in agent mode, just trying to debug the issue. The issue is that it can happily go down the completely wrong path and report exactly the same as though it's solved the problem.
- cmrdporcupine 1y ago
  I explain this in sibling-node comment but I've caught Claude multiple times in the last week just inserting special-case kludges to make things "pass", without actually successfully fixing the underlying problem that the test was checking for. Just outright "if test-is-running { return success; }" level stuff. Not kidding. 3 or 4 times in the past week. Thinking of cancelling my subscription, but I also find it kind of... entertaining?
  - jiggawatts 1y ago
    I just realised that this is probably a side-effect of a faulty training regime. I’ve heard several industry heads say that programming is “easy” to generate synthetic data for and is also amenable to training methods that teach the AI to increase the pass rate of unit tests. So… it did. It made the tests pass. “Job done boss!”
  - falcor84 1y ago
    I found that working with an AI is most productive when I do so in an Adversarial TDD state of mind. As described in this classic qntm post [0] following the VW emissions scandal, which concludes with: > Honestly? I blame the testing regime here, for trusting the engine manufacturers too much. It was foolish to ever think that the manufacturers were on anybody's side but their own. > It sucks to be writing tests for people who aren't on your side, but in this case there's nothing which can change that. > Lesson learned. Now it's time to harden those tests up. [0] https://qntm.org/emissions https://qntm.org/emissions
cmrdporcupine 1y ago
I have mixed results but one of the more disturbing things I've found Claude doing is that when confronted with a failing test case, and not being able to solve a tricky problem.. just writing a kludge into the code that identifies that here's a test running, and makes it pass. But only for that case. Basically, totally cheating. You have to be super careful and review everything because if you don't you can find your code littered with this strange mix of seeming brilliance which makes you complacent... and total Junior SWE behaviour or just outright negligence. That, or recently, it's just started declaring victory and claiming to have fixed things, even when the test continues to fail. Totally trying to gaslight me. I swear I wasn't seeing this kind of thing two weeks ago, which makes me wonder if Anthropic has been turning some dials...
- quesera 1y ago
  > identifies that here's a test running, and makes it pass. But only for that case My team refers to this as a "VW Bugfix".
- alwa 1y ago
  I also feel like I’ve seen a lot more of these over the past week or two, whereas I don’t remember noticing it at all before then. It feels like it’s become grabbier and less able to stay in its lane: ask for a narrow thing, and next thing you know it’s running hog wild across the codebase shoehorning in half-cocked major architectural changes you never asked for. [Ed.: wow, how’s that for mixing metaphors?] Then it smugly announces success, even when it runs the tests and sees them fail. “Let me test our fix” / [tests fail] / [accurately summarizes the way the tests are failing] / “Great! The change is working now!”
  - cmrdporcupine 1y ago
    Yes, or I've seen lately "a few unrelated tests are failing [actually same test as before] but the core problem is solved." After leaving a trail of mess all over. Wat? Someone is changing some weights and measures over at Anthropic and it's not appreciated.
- throwdbaaway 1y ago

the_mitsuhiko 1y ago

> I'm really skeptical of using current LLMs for judging codebases like this. I'm not necessarily convinced that the current generation of LLMs are overly amazing at this, but they definitely are very good at measuring inefficiency of tooling and problematic APIs. That's not all the issues, but it can at least be useful to evaluate some classes of problems.

falcor84 1y ago

What do you mean that it "ended up not even being related to the actual issue"? If you give it a failing test suite to turn green and it does, then either its solution is indeed related to the issue, or your tests are incomplete; so you improve the tests and try again, right? Or am I missing something?

GardenLetter27 1y ago
It made the other tests fail, I wasn't using it in agent mode, just trying to debug the issue. The issue is that it can happily go down the completely wrong path and report exactly the same as though it's solved the problem.
cmrdporcupine 1y ago
I explain this in sibling-node comment but I've caught Claude multiple times in the last week just inserting special-case kludges to make things "pass", without actually successfully fixing the underlying problem that the test was checking for. Just outright "if test-is-running { return success; }" level stuff. Not kidding. 3 or 4 times in the past week. Thinking of cancelling my subscription, but I also find it kind of... entertaining?
- jiggawatts 1y ago
  I just realised that this is probably a side-effect of a faulty training regime. I’ve heard several industry heads say that programming is “easy” to generate synthetic data for and is also amenable to training methods that teach the AI to increase the pass rate of unit tests. So… it did. It made the tests pass. “Job done boss!”
- falcor84 1y ago
  I found that working with an AI is most productive when I do so in an Adversarial TDD state of mind. As described in this classic qntm post [0] following the VW emissions scandal, which concludes with: > Honestly? I blame the testing regime here, for trusting the engine manufacturers too much. It was foolish to ever think that the manufacturers were on anybody's side but their own. > It sucks to be writing tests for people who aren't on your side, but in this case there's nothing which can change that. > Lesson learned. Now it's time to harden those tests up. [0] https://qntm.org/emissions https://qntm.org/emissions

cmrdporcupine 1y ago

I have mixed results but one of the more disturbing things I've found Claude doing is that when confronted with a failing test case, and not being able to solve a tricky problem.. just writing a kludge into the code that identifies that here's a test running, and makes it pass. But only for that case. Basically, totally cheating. You have to be super careful and review everything because if you don't you can find your code littered with this strange mix of seeming brilliance which makes you complacent... and total Junior SWE behaviour or just outright negligence. That, or recently, it's just started declaring victory and claiming to have fixed things, even when the test continues to fail. Totally trying to gaslight me. I swear I wasn't seeing this kind of thing two weeks ago, which makes me wonder if Anthropic has been turning some dials...

quesera 1y ago
> identifies that here's a test running, and makes it pass. But only for that case My team refers to this as a "VW Bugfix".
alwa 1y ago
I also feel like I’ve seen a lot more of these over the past week or two, whereas I don’t remember noticing it at all before then. It feels like it’s become grabbier and less able to stay in its lane: ask for a narrow thing, and next thing you know it’s running hog wild across the codebase shoehorning in half-cocked major architectural changes you never asked for. [Ed.: wow, how’s that for mixing metaphors?] Then it smugly announces success, even when it runs the tests and sees them fail. “Let me test our fix” / [tests fail] / [accurately summarizes the way the tests are failing] / “Great! The change is working now!”
- cmrdporcupine 1y ago
  Yes, or I've seen lately "a few unrelated tests are failing [actually same test as before] but the core problem is solved." After leaving a trail of mess all over. Wat? Someone is changing some weights and measures over at Anthropic and it's not appreciated.
throwdbaaway 1y ago