3 ms·
Another, related benefit of LLMs in this situation is that we can observe their hallucinations and use them for design. I've come up with a couple situations wh
by yujzgzc 1y ago
Another, related benefit of LLMs in this situation is that we can observe their hallucinations and use them for design. I've come up with a couple situations where I saw Copilot hallucinate a method, and I agreed that that method should've been there. It helps confirm whether the naming of things makes sense too.
What's ironic about this is that the very things that TFA points out are needed for success (test coverage, debuggability, a way to run locally etc) are exactly the things that typical LLMs themselves lack.
- crazygringo 1y agoI've found LLM's to be extremely helpful in naming and general function/API design, where there a lot of different ways to express combinations of parameters. I know what seems natural to me but that's because I'm extremely familiar with the internal workings of the project. LLM's seem to be very good at coming with names that are just descriptive enough but not too long, and most importantly follow "general conventions" from similar projects that I may not be aware of. I can't count the number of times an LLM has given me a name for a function that I've thought, oh of course, that's a much clearer name that what I was using. And I thought I was already pretty good at naming things...
- autobodie 1y agoMore often, LLMs give me wildly complex and over-engineered solutions.
- steveklabnik 1y agoI have had this happen too, but a "this seems complex, do we need that complexity? can you justify it? or can we make this simpler?" or similar has them come back with something much better.
- istjohn 1y agoI get good results with: "Can this be improved while maintaining simplicity and concision?"
- autobodie 1y agoWhen I do that, they come back with something more simple than before but either wrong or still unnecessarily complex. Over the past week, I have been writing a small library for midi controller I/O and simple/elegant is the priority. I am not really that opinionated. I just want it to not be overengineered. AI has been able to make some suggestions when I give it a specific goal for refactoring a specific class, but it cannot solve a problem on its own without overengineered solutions.
- sfn42 1y agoYou just have to be more specific. Don't just tell it to refactor, tell it how to refactor. I usually start out a bit vague then I add more specific instructions when I want specific changes. I often just make the changes myself because it's faster than describing them. You do the thinking, LLM does the writing. The LLM doesn't solve problems, that's your job. The LLMs job is to help you do the job more efficiently. Not just do it for you.
- creatonez 1y agoTrue, but... a good chunk of the time, when an API doesn't exist, it really is because it shouldn't exist. I've seen many examples where an LLM hallucinates an API it wishes existed, but in reality implementing it upstream would turn it into an insane jumble or violate the internal logic in a very bad way. For example: inserting to the middle of a concurrent queue data structure.