4 ms·

I'd ask things like "which LLM are you using", and "what language or APIs are you asking it to write for". For the standard answers of "GPT-4 or above", "claud

I'd ask things like "which LLM are you using", and "what language or APIs are you asking it to write for". For the standard answers of "GPT-4 or above", "claude sonnet or haiku", or models of similar power and well known languages like Python, Javascript, Java, or C and assuming no particularly niche or unheard of APIs or project contexts the failure rate of 4-5 line of code scripts in my experience is less than 1%.

rudiksz 2y ago
It's mostly Go, some Python, and I'm not asking anything niche. I'm asking for basic utility functions that I could implement in 10-20 lines of code. There's something broken every single time and I spend more time debugging the generated code than actually writing it out. I'm pretty sure everybody measures "failure rate" differently and grossly exaggerate the success rate. There's a lot of suggestions below about "tweaking", but if I have to "tweak" generated code in any way then that is a failure for me. So the failure rate of generated code is about 99%.