6 ms·
Chezmoi introduces ban on LLM-generated contributions
- singiamtel 1y agoDoes this mean Copilot tab complete is banned too? What about asking an LLM for advice and then writing all the code yourself?
- Lalabadie 1y agoI'm pretty sure the point is that anything clearly generated will result in an instant ban. That seems rather fair, you want contributors who only submit code they can fully understand and reason about.
- odie5533 1y agoThe language clearly says "If you use an LLM [...] to make any kind of contribution".
- pkilgore 1y ago[I was wrong and posted a link to an earlier policy/discussion overridden by the OP]
- Groxx 1y agoyou seem to be reading that backwards, that's the content that was removed. it now just says "if LLM, banned": https://github.com/twpayne/chezmoi/blob/master/.github/CODE_OF_CONDUCT.md https://github.com/twpayne/chezmoi/blob/master/.github/CODE_...
- marcandre 1y agoThe part you are quoting is being removed. The policy used to state "If you contribute un-reviewed LLM generated...", now simply states "If you use an LLM to make any kind of contribution then you will immediately be banned without recourse."
- baby_souffle 1y ago> Does this mean Copilot tab complete is banned too? What about asking an LLM for advice and then writing all the code yourself? You're brushing up against some of the reasons why I am pretty sure policies like this will be futile. They may not diminish in popularity but they will be largely unenforceable. They may serve as an excuse for rejecting poor quality code or code that doesn't fit the existing conventions/patterns but did maintainers need a new reason to reject those PRs? How does one show that no assistive technologies below some threshold were used?
- jbstack 1y ago> How does one show that no assistive technologies below some threshold were used? In this case, you don't: > immediately be banned without recourse In other words, if the maintainer(s) think it's LLM-generated, right or wrong, you're banned.
- koakuma-chan 1y agoIdk why anyone would contribute to a project with an attitude like this
- hitarpetar 1y agothat's fine, they probably don't want you then
- koakuma-chan 1y agoI don't want them either. I'll find someone else who likes me the way I am. Plenty of fish in the pond.
- SoftTalker 1y agoMake useful, carefully reviewed contributions and you'll be fine.
- IncreasePosts 1y ago
- odie5533 1y agoTab completions by LLM are code generated by an LLM.
- polonbike 1y agoI am wondering why you are posting this link, then asking this question to the HN community, instead of asking the project directly for more details. I does look like your intent is to stir some turmoil over the project position, and not to contribute constructively to the project.
- tverbeure 1y agoThat kind of point could be made for a large fraction of HN comments, but that aside: if a project’s policy is to ban for any LLM usage, without recourse, just asking a question about it could put you on a list of future suspects…
- singiamtel 1y agoGood point, I'll include my question on the original discussion
- qsort 1y agoNot sure about this project in particular, but many more popular projects (curl comes to mind) have adopted similar policies not out of spite but because they'd get submerged by slop. Sure, a smart guy with a tool can do so much more, but an idiot with a tool can ruin it for everyone.
- jbstack 1y agoIsn't it then more reasonable to have a policy that "people who submit low quality PRs will be banned"? Target the actual problem rather than an unreliable proxy of the problem. LLM-generated code can be high quality just as human-generated code can be low quality. Also, having a "no recourse" policy is a bit hostile to your community. There will no doubt be people who get flagged as using LLMs when they didn't and denying them even a chance to defend themselves is harsh.
- Ekaros 1y agoBanning LLMs can result in shorter arguments. "Low quality" is overly subjective and will probably take a lot of time to argue about. And then the possible outrage if it is taken to social media.
- jbstack 1y ago> Banning LLMs can result in shorter argument Can it really? "You submitted LLM-generated contributions" is also highly subjective. Arguably more so since you can't ever really be sure if somethingi s AI generated while with quality issues there are concrete things you can point to (e.g. it the code simply doesn't work, doesn't meet the contributor guidelines, uses obvious anti-patterns etc.).
- 63stack 1y agoIf you rtfa, you will find it's actually the other way around. The linked PR from the AI has "concrete things you can point to" like "the code simply doesn't work".
- 1y ago
- pkilgore 1y ago[I was wrong and posted a link to an earlier policy/discussion overridden by the OP]
- marcandre 1y agoDid you read the policy? "If you use an LLM to make any kind of contribution..."
- jbstack 1y agoIt's you who isn't reading the policy. What you are reading instead is deleted sections in a Git commit which are not part of the policy.
- rane 1y agoSaid policy: * Any contribution of any LLM-generated content will be rejected and result in an immediate ban for the contributor, without recourse.
- mbreese 1y agoThat’s not what it says. It’s pretty clear… > Any contribution of any LLM-generated content will be rejected and result in an immediate ban for the contributor, without recourse. You can argue it’s unenforceable, unproductive, or a bad idea. But it says nothing about unreviewed code. Any LLM generated code. I’m not sure how great of an idea it is, but then again, it’s not my project. Personally, I’d rather read a story about how this came to be. Either the owner of the project really hates LLMs or someone submitted something stupid. Either would be a good read.
- 63stack 1y agoThe linked thread includes the story
- Valodim 1y agoThe language is actually: > Any contribution of any LLM-generated content I read this as "LLM-generated contributions" are not welcome, not "any contribution that used LLMs in any way". More generally, this is clearly a rule to point to in order to end discussions with low effort net-negative contributors. I doubt it's going to be a problem for actually valuable contributions.
- mambo_giro 1y agoSpecific change: https://github.com/twpayne/chezmoi/commit/7938c65ca55aaeaf6fc56d37b0b32073d2e5c461 https://github.com/twpayne/chezmoi/commit/7938c65ca55aaeaf6f... And corresponding discussion: https://github.com/twpayne/chezmoi/discussions/4010 https://github.com/twpayne/chezmoi/discussions/4010
- koakuma-chan 1y ago> an immediate ban for the contributor, without recourse. Maintainer sounds angry
- threatofrain 1y agoOr maintainer needs money to pay for more help.
- daveguy 1y agoOr just sick of the slop.
- rsynnott 1y agoMaintainer probably has to deal with a lot of spam nonsense LLM pull requests. It would be astonishing if they were _not_.
- bryanlarsen 1y agoIt's interesting that the final policy is significantly harsher than the initial more reasonable sounding proposal.
- squigz 1y ago> Users posting unreviewed LLM-generated content without any admission will be immediately be banned without recourse. Yikes. If maintainers want to ban people for wasting their time, that's great, but considering how paranoid people have gotten about whether something is from an LLM or not, this seems heavy-handed. There needs to be some kind of recourse. How many legitimate-but-simply-wrong contributors will be banned due to policies like this?
- 1y ago
- jolux 1y agochezmoi is a great tool, and I admire this project taking a strong stand. However I can’t help but feel that policies like this are essentially unenforceable as stated: there’s no way to prove an LLM wasn’t used to generate code. In many cases it may be obvious, but not all.
- delusional 1y agoI don't think rules like that are meant to be 100% perfectly enforced. It's essentially a policy you can point to when banning somebody, and the a locus of disagreement. If you get banned for alleged AI use, you have to argue that you didn't use AI. It doesn't matter to the project if you were helpful and kind, the policy is no AI.
- pkilgore 1y ago[I was wrong and posted a link to an earlier policy/discussion overridden by the OP]
- deleted 1y ago[deleted]
- stavros 1y agoHere it is: > Any contribution of any LLM-generated content will be rejected and result in an immediate ban for the contributor, without recourse. What about it changes the parent comment?
- CGamesPlay 1y agoWhat are you talking about? The OP says "If you use ... banned without recourse" and the "more information" link manages to have even less information.
- colonwqbang 1y agoSome people post vulnerability disclosures or pull requests which are obviously fake and generated by LLM. One example: https://hackerone.com/reports/2298307 https://hackerone.com/reports/2298307 These people are collaborating in bad faith and basically just wasting project time and resources. I think banning them is very legitimate and useful. It does not matter if you manage to "catch" exactly 100% of all such cases or not.
- pkilgore 1y ago[I was wrong and wrote a defense of an earlier policy/discussion overridden by the OP]
- johnisgood 1y agoYou first have to determine that code in the PR was generated by LLM(s). How do you do that? What about false positives?
- pkilgore 1y agoI don't think you realize how painfully obvious it is when people submit LLM generated crap. Most of the time, they literally admit it as soon as you ask, or, concurrently with the submission. It's a one time tax you pay sure, but after the ban at least you know you'll never deal with that use again. And a lot of these contributions come from the same people.
- hitarpetar 1y agoone of many questions that would have been good to ask before this technology was widely available
- senordevnyc 1y agoWhy? If we can't tell the difference...isn't that a good thing? https://xkcd.com/810/ https://xkcd.com/810/
- luckydata 1y agoThis is dumb. Llms are a tool, a very useful one. Bad PRs should be rejected always no matter the source, but banning a tool because some people can't use it is not what engineering is about.
- pkilgore 1y ago[I was wrong and posted a link to an earlier policy/discussion overridden by the OP]
- jbstack 1y agoYou've made this comment more than once in this thread. Have you correctly understood that the policy is only the green parts in that link and not the red parts?
- pkilgore 1y agoAh fuck, not careful enough with red green colorblindness and misled by my existing knowledge of the old policy. Fixed those replies, thanks for flagging.
- Groxx 1y agomentally adding this to my "obviously a problem in retrospect" list. hadn't thought of it before though. OTOH, do you think a diagonal-lined-background on the removed side would work better? like striped light and dark red, vs solid green. I don't think I've seen anything do different than "red/green background of similar lightness", which is sorta surprising...
- willahmad 1y agoThis sounds limiting. I compare LLM generated content to autocomplete. When autocomplete shows you options, you can choose any of the options blindly and obviously things will fail, but you can also pick right method to call and continue your contribution. When it comes to LLM generated content, its better if you provide guidelines for contribution rather than banning it. For example: * if you want to generate any doc use our llms_doc_writing.txt * for coding use our llms_coding.txt
- JoshTriplett 1y ago> This sounds limiting. Coding guidelines generally are, by design. > * if you want to generate any doc use our llms_doc_writing.txt That's exactly what the project is providing here. The guidelines for how to use LLMs for this project are "don't". You say "generally better to", but that depends on what you're trying to achieve. Your suggestion is better if you want to change how people use LLMs, the project's is better if the project is trying to change whether people use LLMs.
- willahmad 1y agoThis is not a guideline on the code itself, its about tools you use to produce that code. You can similarly ban code written using IntelliJ IDEA and accept only code written using vim or VS Code, but you wouldn't even know if it was written in IDEA or VSCode. Saner guideline would be: * before submitting your LLM generated code, review your code * respect yours and our time * if LLM spit out 1k line of code, its on you to split it and make it manageable for us to review, because humans review this code * if we find that you used LLM but wasn't respectful to our community by not following above, please f.... off from our community, and we will ban you * submitting PR using solely automated PR slop generators will be banned forever
- JoshTriplett 1y agoThat sure is a different thing that some people might prefer. It it not what this project prefers, and that's fine. To give another example of what is impossible to perfectly detect but still reasonable to prohibit: most projects have a (written or unwritten) requirement of "don't copy code from other projects without respecting their license". There's no reliable way to perfectly detect that someone copied a pile of code from one of their company's proprietary projects, or from some Open Source project under a different license or without attribution. But it's still reasonable for projects to prohibit such "contributions".
- Luker88 1y agoHas the situation changed on AI code legally speaking? Am I now assured that the copyright is mine if the code is generated by AI? Worldwide? (or at least North America-EU wide)? Do projects still risk becoming public domain if they are all AI generated? Does anyone know of companies that have received *direct lawyer* clearance on this, or are we still at the stage "run and break, we'll fix later"? Maybe having a clear policy like this might be a defense in case this actually becomes a problem in court.
- JustFinishedBSG 1y ago> Has the situation changed on AI code legally speaking? I think the position has shifted to "let's pretend this problem doesn't exist because the AI market is too big to fail"
- roguecoder 1y agoIt is going to be so interesting now that most software is going to be public domain. It's going to be us and the fashion world working just fine without intellectual property rights.
- fao_ 1y ago> Has the situation changed on AI code legally speaking? lol, l m a o, essentially people who use LLMs have been betting that courts will rule on their favour, because shit would hit the fan if it didn't. The courts however, have consistently ruled against AI-generated content. It's really only a matter of time until either the bubble bursts, or legislation happens that pops the bubble. Some people here might hope otherwise, of course, depending on reliant they are on the hallucinating LSD-ridden mechanical turks.
- koakuma-chan 1y ago> The courts however, have consistently ruled against AI-generated content. Have they? I only heard of courts ruling it is fair use.
- pkilgore 1y ago[I was wrong and posted a link to an earlier policy/discussion overridden by the OP]
- bryanlarsen 1y ago> Note that I don't care if people use an LLM to help them generate content, but I do expect them to review it for correctness before posting it here. The final policy posted contradicts this statement.
- nightpool 1y agoMaybe because the lack of nuance in "no LLM content at all" is easier for LLMs to understand :P EDIT: Looks like the quote you had was for an earlier version of the policy, which was changed because people did not/could not abide by it: https://news.ycombinator.com/item?id=45669846 https://news.ycombinator.com/item?id=45669846
- numpad0 1y agoI'm minimally exposed to vibecoding, but already finding it immensely useful. That said, one thing I don't want to do, is to touch that autogenerated code, hardly opening in an editor. Anyone feeling the same? That they're not for humans to see?
- roguecoder 1y agoYes, and that means that it should never be used for anything connected to the internet or where there is a human cost if it is wrong. Vibe coding is great for local tools where security isn't a concern and where it is easy for the user to verify correctness. It is when people want to do that professionally, on software that actually needs to work, that it becomes a massive ethical problem.
- rufo 1y agoWhat's interesting is the change in the policy. Old policy: > If you use an LLM (Large Language Model, like ChatGPT, Claude, Gemini, GitHub Copilot, or Llama) to make a contribution then you must say so in your contribution and you must carefully review your contribution for correctness before sharing it. If you share un-reviewed LLM-generated content then you will be immediately banned. ...and the new one: > If you use an LLM (Large Language Model, like ChatGPT, Claude, Gemini, GitHub Copilot, or Llama) to make any kind of contribution then you will immediately be banned without recourse. Looking at twpayne's discussion about the LLM policy[1], it seems like he got fed up with people not following those instructions: > I stumbled across an LLM-generated podcast about chezmoi today. It was bland, impersonal, dull, and un-insightful, just like every LLM-generated contribution so far. > I will update chezmoi's contribution guide for LLM-generated content to say simply "no LLM-generated content is allowed and if you submit anything that looks even slightly LLM-generated then you will be immediately be banned." [1]: https://github.com/twpayne/chezmoi/discussions/4010#discussioncomment-14723899 https://github.com/twpayne/chezmoi/discussions/4010#discussi...
- squigz 1y agoEven more yikes. They found a third-party LLM-generated podcast and made the policy even harsher because of it? What happens when they continue to run into more LLM-generated content out in the wild? Interestingly, this is exactly the sort of behavior people have been losing their minds about lately with regards to Codes of Conduct.
- rufo 1y agoI think it's that the low quality of the LLM-generated podcast caused him to reflect on the last year's worth of (apparently, largely low-quality) LLM-generated pull requests opened on the project; not that the podcast itself was the direct cause of the change in policy.
- WhitneyLand 1y agoSay I prepare a contribution on my own that meets all guidelines and quality standards. Then before submitting if I ask an LLM to review my code and it proposes a few changed lines that are more efficient. Should I then - Leave my less efficient code unchanged? - Try to rewrite what was suggested in a way that’s not too similar to what the LLM suggested?
- muli_d 1y ago"Users posting unreviewed LLM-generated content with the admission that they do not understand the code" Unreviewed is a key word here.
- senordevnyc 1y agoThat's not the policy: Any contribution of any LLM-generated content will be rejected and result in an immediate ban for the contributor, without recourse.
- WhitneyLand 1y agoNo, unreviewed was the previous policy. Now it’s none at all.
- deepanwadhwa 1y agoWait, can anyone help me understand how would they enforce this? All the AI detection tools I have reviewed failed miserably at detecting AI in text.
- senordevnyc 1y agoIt seems clear to me that this isn't a well thought out policy, but more of a tantrum by yet another developer angry about the industry changing out from under them. Sadly, it won't help, it'll just hasten this project's death.
- roguecoder 1y agoI'm going to spend the rest of my career charging twice what I used to charge cleaning up the unmaintainable, non-functional-but-provably-valuable messes these tools are producing, but that doesn't mean I want to have to do the same in the community work when there is absolutely no reason for it.
- roguecoder 1y agoMany humans, on the other hand, are extremely good at telling AI-generated text from non-AI-generated text. Personally it's like looking at a ransom note made up of letters cut out of magazines & having people tell me how beautiful the handwriting is.
- deepanwadhwa 1y agoI agree with you but is it scalable?
- alt187 1y ago> Isn't it more reasonable to explain in excruciating detail what kind of contributions you will allow? No, it's not. You can read the rule as "If it's obvious enough your code has been LLM-generated, you will get banned" if you feel like the conciseness of the current rule makes you uneasy about using Copilot. Besides, I suspect in the maintainer's case, banning unreviewed LLM contributions is effectively congruent to banning all LLM contributions. If you think the rule is unfair towards LLMs because they can do such good, feel free to open a good, clean, useful PR clearly stating how you used the LLM to generate code.