4 ms·

When I look at the commits themselves, most of the ones generated by Claude are testsuite changes, or at least labelled as such. https://github.com/RsyncProjec

When I look at the commits themselves, most of the ones generated by Claude are testsuite changes, or at least labelled as such. https://github.com/RsyncProject/rsync/commits/master/ https://github.com/RsyncProject/rsync/commits/master/

shimman 4mo ago
Is that suppose to make this better? IME the most valuable tests are those that test specific regressions. It's the scaffolding we build for ourselves to enable feature development. Remove that scaffolding and you get accidents. Pray to your god of choice these accidents don't cause harm or loss of life. It should really be considered negligence at this point. Some of this software is extremely valuable, it's how we flourish as humans. Purposely fucking with that should bear some real world consequence. We do the same in every other industry, software is just as important too.
- abuob 4mo ago
  In my perspective, "Analyze code, come up with edge cases and gaps and create unit tests for them" is one of the use-cases where AI was starting to get really good at, so I can see why someone would want to extend their test-suite dramatically using it. But yes, using AI to then generate code that still causes regressions doesn't quite square with that. Given the huge amount of test-changes I'd still assume good faith by the maintainer; possibly just a bit of overexcitement paired with a dash of too much confidence into the new tools that is now hitting reality.
- ornornor 4mo ago
  I hear you, OTOH if this software was so valuable how come we aren’t funding it? A lot of the world runs on OSS with a coupe overwhelmed maintainers who get treated as if they owed everybody working software yet can’t make a living off it.
  - shimman 4mo ago
    We should fund it. Go read the types of comments I make in my profile. I always advocate for explicitly taxing big tech to publicly fund open source development. Also it's why we need to pass things like medicare for all and universal childcare to give workers some breathing room if they want to change jobs/industries without condemning them to death or poverty.
- ncruces 4mo ago
  Taken at face value, most commit descriptions mention adding - not skipping - tests and assertions. So basically, we're all in our high horses, not reviewing code, scalding the unpaid maintainer for … not reviewing code. Time for - whoever actually cares - to do better.
- scared_together 4mo ago
  > Is that suppose to make this better? When I first saw the 26k changes statistic I was shocked. It made me think a large chunk of code running on people’s machines was AI-generated. But the knowledge that a lot of the changes might be testsuite changes made me change my perspective. If for instance 25k of the changes were test changes and only 1k of the changes actually affected the .so and other artifacts used downstream, that would be a lot less dramatic. I haven’t reviewed the code, only the messages, so I don’t know if these changes were removing or adding test cases. And there are a minority of Claude-assisted changes which are not listed as tests.
vips7L 4mo ago
Aren’t LLMs notorious for just making tests pass and not actually testing functionality?
- cinntaile 4mo ago
  You have to keep an eye on them, but they don't just make tests pass.
  - kdjkskdndn 4mo ago
    Claude sonnet 4 (this time last year) did do this. It once made simulation if a test script passing. Literally a script that just echoed test names and then said pass.
    - cinntaile 4mo ago
      Change happens fast, a year old model is pretty outdated. I'm sure it can happen, hence why I said to keep an eye out. Its main mode of operation is not to cook the tests however.
      - yw3410 4mo ago
        Happened to me, 3 days ago - deleted some tests and flipped assertions after outlining that it wasn't to change any assertions. Our team was doing a similar task to move between test frameworks, and I had to do a git diff of hundreds of thousands of lines to try and work out where a test had disappeared to.
        LtWorf 4mo ago
        > 3 days ago Your fault. You should have used a model from 0.000005 seconds ago!
        cinntaile 4mo ago
        Reading is difficult.
layer8 4mo ago
The correctness of tests is as important as the correctness of the main code. Changing test code isn’t somehow less critical than changing the main code.

shimman 4mo ago

Is that suppose to make this better? IME the most valuable tests are those that test specific regressions. It's the scaffolding we build for ourselves to enable feature development. Remove that scaffolding and you get accidents. Pray to your god of choice these accidents don't cause harm or loss of life. It should really be considered negligence at this point. Some of this software is extremely valuable, it's how we flourish as humans. Purposely fucking with that should bear some real world consequence. We do the same in every other industry, software is just as important too.

abuob 4mo ago
In my perspective, "Analyze code, come up with edge cases and gaps and create unit tests for them" is one of the use-cases where AI was starting to get really good at, so I can see why someone would want to extend their test-suite dramatically using it. But yes, using AI to then generate code that still causes regressions doesn't quite square with that. Given the huge amount of test-changes I'd still assume good faith by the maintainer; possibly just a bit of overexcitement paired with a dash of too much confidence into the new tools that is now hitting reality.
ornornor 4mo ago
I hear you, OTOH if this software was so valuable how come we aren’t funding it? A lot of the world runs on OSS with a coupe overwhelmed maintainers who get treated as if they owed everybody working software yet can’t make a living off it.
- shimman 4mo ago
  We should fund it. Go read the types of comments I make in my profile. I always advocate for explicitly taxing big tech to publicly fund open source development. Also it's why we need to pass things like medicare for all and universal childcare to give workers some breathing room if they want to change jobs/industries without condemning them to death or poverty.
ncruces 4mo ago
Taken at face value, most commit descriptions mention adding - not skipping - tests and assertions. So basically, we're all in our high horses, not reviewing code, scalding the unpaid maintainer for … not reviewing code. Time for - whoever actually cares - to do better.
scared_together 4mo ago
> Is that suppose to make this better? When I first saw the 26k changes statistic I was shocked. It made me think a large chunk of code running on people’s machines was AI-generated. But the knowledge that a lot of the changes might be testsuite changes made me change my perspective. If for instance 25k of the changes were test changes and only 1k of the changes actually affected the .so and other artifacts used downstream, that would be a lot less dramatic. I haven’t reviewed the code, only the messages, so I don’t know if these changes were removing or adding test cases. And there are a minority of Claude-assisted changes which are not listed as tests.

vips7L 4mo ago

Aren’t LLMs notorious for just making tests pass and not actually testing functionality?

cinntaile 4mo ago
You have to keep an eye on them, but they don't just make tests pass.
- kdjkskdndn 4mo ago
  Claude sonnet 4 (this time last year) did do this. It once made simulation if a test script passing. Literally a script that just echoed test names and then said pass.
  - cinntaile 4mo ago
    Change happens fast, a year old model is pretty outdated. I'm sure it can happen, hence why I said to keep an eye out. Its main mode of operation is not to cook the tests however.
    - yw3410 4mo ago
      Happened to me, 3 days ago - deleted some tests and flipped assertions after outlining that it wasn't to change any assertions. Our team was doing a similar task to move between test frameworks, and I had to do a git diff of hundreds of thousands of lines to try and work out where a test had disappeared to.
      - LtWorf 4mo ago
        > 3 days ago Your fault. You should have used a model from 0.000005 seconds ago!
        cinntaile 4mo ago
        Reading is difficult.

layer8 4mo ago

The correctness of tests is as important as the correctness of the main code. Changing test code isn’t somehow less critical than changing the main code.