4 ms·
Aren’t LLMs notorious for just making tests pass and not actually testing functionality?
by vips7L 4mo ago
Aren’t LLMs notorious for just making tests pass and not actually testing functionality?
- cinntaile 4mo agoYou have to keep an eye on them, but they don't just make tests pass.
- kdjkskdndn 4mo agoClaude sonnet 4 (this time last year) did do this. It once made simulation if a test script passing. Literally a script that just echoed test names and then said pass.
- cinntaile 4mo agoChange happens fast, a year old model is pretty outdated. I'm sure it can happen, hence why I said to keep an eye out. Its main mode of operation is not to cook the tests however.
- yw3410 4mo agoHappened to me, 3 days ago - deleted some tests and flipped assertions after outlining that it wasn't to change any assertions. Our team was doing a similar task to move between test frameworks, and I had to do a git diff of hundreds of thousands of lines to try and work out where a test had disappeared to.
- LtWorf 4mo ago> 3 days ago Your fault. You should have used a model from 0.000005 seconds ago!
- cinntaile 4mo agoReading is difficult.
- LtWorf 4mo ago[flagged]
- cinntaile 4mo agoTry to put in a little more effort next time. It makes for better discussions instead of "jokes" that aren't even relevant.
- LtWorf 4mo agoI discuss with civilised people who are up for civilised discussions.
- cinntaile 4mo agoYou are the one ruining this discussion, it's worrying that you don't even realize it. I pointed out that models change quite a bit over time (I said more than that) and you ridicule my reply. "Your fault. You should have used a model from 0.000005 seconds ago!" Civilised. Please.
- customguy 4mo ago> Change happens fast, a year old model is pretty outdated. What change? That you should not fake the results of a test because that defeats the whole purpose of a test has been known before there were computers.
- cinntaile 4mo agoI don't know, the weather?
- mcintyre1994 4mo agoI’ve never seen Claude do that. It makes the new tests pass by fixing previously unknown bugs in my experience.
- weird-eye-issue 4mo agoI love Claude but on several occasions I've had it do some really funky stuff just to get tests passing
- bcrosby95 4mo agoI had it do it about a month ago. It changed test data which caused another test to fail and instead of isolating things it decided to flip an assert.
- cyclopeanutopia 4mo agoThat's because Opus needed vacation and they routed your requests to its less sophisticated cousin, Claude Dynamite. ;)
- senordevnyc 4mo agoYeah, in 2024.