3 ms·
There's a meta-level of irony here that's important to note. TFA is defending the use of AI, and it very clearly (to me) used AI to analyze the data and presen
by xmddmx 4mo ago
There's a meta-level of irony here that's important to note.
TFA is defending the use of AI, and it very clearly (to me) used AI to analyze the data and present the results.
In doing so, the author used statistics in a way they do not appear to understand, and ended up making numerous false claims (you can see the thread discussing these here https://news.ycombinator.com/item?id=48417626 https://news.ycombinator.com/item?id=48417626 )
In short, the study doesn't have sufficient statistical power, and is making "no difference" claims that aren't justified.
The meta-irony is this: the author used an LLM to interpret data in this study, and seems to have made the same category of mistake (confidently asserting falsehoods) that the study was supposed to be investigating (confidently submitting bad commits to the rsync project).
- classified 4mo agoAI is so much like a religion. There is nothing you can say to a believer that will make them question their believes. Or more generally, you cannot reason anyone out of something that they want to believe.
- Joel_Mckay 4mo agoIt gets pretty dark if you pull that thread of reason. =3 https://en.wikipedia.org/wiki/The_True_Believer https://en.wikipedia.org/wiki/The_True_Believer
- newsoftheday 4mo agoAI is nothing like religion. People behave similarly to AI when debating their favorite sports team, or for Java coders, Checked vs Runtime exceptions. Religion is about faith and what people feel and sense as much as believe.
- logicprog 4mo agoThe statistical methodology I used is mine. As is the interpretation. Completely. To the degree that I misunderstood statistics (and it is under debate even in the thread you link, and the people accusing me of misunderstanding statistics there are universally misrepresenting my point, which is to point out a total absence of evidence for any difference, not to prove the null hypothesis) that's on me
- MichaelDickens 4mo agoFWIW I understood your point just fine. It seemed to me that you made a clear enough distinction between "evidence that Claude didn't increase bugs" and "no good evidence either way".
- runarberg 4mo agoI don’t think that point is clear at all, at least in this comment sections you can find a few posts like these: > Did Claude increase bugs in rsync? > > TFA answered this, the answer is “no”. https://news.ycombinator.com/item?id=48423811 https://news.ycombinator.com/item?id=48423811
- karagenit 4mo agoIt seems like based on e.g. [1] the article originally made some stronger claims about “no difference in bugs” that have been corrected. I agree that now it seems fine, but those edits might be why it feels like some commenters read a different article than you. [1]: https://news.ycombinator.com/item?id=48418186 https://news.ycombinator.com/item?id=48418186
- simianwords 4mo agoThe meta-meta level irony is that the reaction to this post is based on vibes and misunderstands the point of the article to wage ideological warfare -- much the same way the original github issue was written.