4 ms·
I think we will be getting into an interesting situation soon, where project maintainers use LLMs because they truly are useful in many cases, but will ban cont
by lukaslalinsky 7mo ago
I think we will be getting into an interesting situation soon, where project maintainers use LLMs because they truly are useful in many cases, but will ban contributors for doing so, because they can't review how well did the user guide the LLM.
- dlillard0 7mo agoI think guiding the LLM to write code is easy for them to write code by themselves.
- mfld 7mo agoMaybe a future direction will be the submission of detailed research, specifications and change plans for feature requests. Something that can be assessed by a human and turned into working code by both slides.
- konschubert 7mo agoI wonder if that is an opportunity to build an Open-Source platform focused on this, replacing GitHub as the collaboration platform of a time where code was valuable.
- konschubert 7mo agoThe bottlenecks today are: * understanding the problem * modelling a solution that is consistent with the existing modelling/architecture of the software and moves modelling and architecture in the right direction * verifying that the the implementation of the solution is not introducing accidental complexity These are the things LLMs can't do well yet. That's where contributions will be most appreciated. Producing code won't be it, maintainers have their own LLM subscriptions.
- advancespace 7mo ago[flagged]
- naasking 7mo agoI don't know, I've had good experiences getting LLMs to understand and follow architecture and style guidelines. It may depend on how modular your codebase already is, because that by itself would focus/minimize any changes.
- ZaoLahma 7mo agoSome months back I would have agreed with you without any "but", but it really does help even if it only takes over "typing code". Once you do understand the problem deep enough to know exactly what to ask for without ambiguity, the AI will produce the code that exactly solves your problem a heck of a lot quicker than you. And the time you don't spend on figuring out language syntax, you can instead spend on tweaking the code on a higher architecture level. Spend time where you, as a human, are better than the AI.
- lukaslalinsky 7mo agoI still think there is value in external contributors solving problems using LLMs, assuming they do the research and know what they are doing. Getting a well written and tested solution from LLM is not as easy as writing a good prompt, it's a much longer/iterative process.
- lkjdsklf 7mo ago> assuming they do the research and know what they are doing. This is the assumption that has almost always failed and thus has lead to the banning of AI code altogether in a lot of projects.
- pydry 7mo agoThe "interesting situation" is that maintainers are unable to cheaply distinguish slop from good contributions so they will simply stop accepting outside contributions. This will cut off one of the genuine entry points to the industry where all you really needed was raw talent.
- mixedbit 7mo agoIf an author of a PR just generated code with an LLM, the GitHub PR becomes an incredibly inefficient interface between a repository owner and the LLM. A much better use of the owner time would be to interact with LLM directly instead of responding to LLM generated PR, waiting for updates, responding again, etc.
- silverwind 7mo agoYep, the indirection through the PR author is almost always inefficient and error-prone unless the author is really knowledgable about the code (many aren't).
- lukaslalinsky 7mo agoAs a project maintainer, I don't want to interact with someone's LLM. If a person submits a PR, using LLM or not, the person is responsible for any problems with it. How they respond to review is a good indicator if they actually understand the code. And if they used a bot to submit the PR, I'd simply consider it a spam.
- pjc50 7mo agoThe GPL talks about "the preferred form for modification of the software", and I'm starting to think that anything which involves any kind of LLM agent should be including all the text that the user gave to it as well. Prompts, etc. Of course, even then it's not reproducible and requires proprietary software!
- rswail 7mo agoExcept the GPL is dependent on the author having copyright over the original software but the output of an LLM may not be covered by copyright as a derivative work. That breaks "copyleft" entirely.
- est31 7mo agoI still think the source code is the preferred form for modification because it is what you point the AI at when you want it to make a change. Sure there might be md documents that you created that the AI used to implement the software, but maybe those documents themselves have been AI written from prompts (due to how context works in LLMs, it's better for larger projects to first make an md document about them, even if an LLM is used for it in the first place). As for proprietary software, the chinese models are not far behind the cutting edge of the US models.
- zigzag312 7mo agoSome sort of LLM audit trail is needed (containing prompts used, model identifier and marking all code written by LLM). It could be even signed by LLM providers (but that wouldn't work with local models). Append only standard format that is required to be included in PR. It wouldn't be perfect (e.g. deleting the log completely), but it might help with code reviews. This would probably be more useful to help you see what (and how) was written by LLMs. Not really to catch bad actors trying to hide LLM use.
- chatmasta 7mo agoThis would be a useful feature to bake into the commits generated by agents. Heck you don’t even need to wait — just change your prompt to tell it to include more context in its commit messages and to sign them as Claude rather than yourself…
- zhangchen 7mo ago[flagged]
- hijnksforall956 7mo agoHow is that different than before LLMs? You have no idea how the person came up with it, or whether they really understood. We are inventing problems here. Fact is, an LLM writes better code than 95% of developers out there today. Yes, yes this is Lake Wobegone, everyone here is in the 1%. But for the world at large, I bet code quality goes up.
- duskdozer 7mo agoIt's a lot harder for someone who has no clue what they're doing to write a lot of plausible-but-wrong code.
- bandrami 7mo agoAnd in general a lot more people want to use LLMs to generate things than want to consume the things LLMs generate. Some of the more bullish people should think harder about this pretty clear trend.
- riffraff 7mo agosome projects (I came across Mastodon's policy[0] which they say was inspired by ghostty and others) which have a more relaxed policy of basically "LLMs are ok so long as you understand what they did, and you own it". But I think different projects have different needs. [0] https://github.com/mastodon/.github/blob/main/AI_POLICY.md https://github.com/mastodon/.github/blob/main/AI_POLICY.md