3 ms·
FWIW, I'd expect mismatched parentheses to be particularly likely to occur in Lisp. There are very few forms of writing where two tokens () account for 30-50% o
by lolinder 3y ago
FWIW, I'd expect mismatched parentheses to be particularly likely to occur in Lisp. There are very few forms of writing where two tokens () account for 30-50% of the total token size of the text. It's unsurprising that with even a tiny bit of temperature the model gets overzealous and places one too early or too late.
Add to that the fact that matching parentheses is hard for humans to reliably do correctly—especially when typing code into a textbox on Stack Overflow—and now you have a model whose architecture struggles with knowing when to stop in highly repetitive sequences and that was trained on data that is disproportionately likely to have incorrect syntax for the language you're using.
Other languages don't have those features, and I've never found Copilot to give me a syntax error in, say, JavaScript.
That said, I also don't know any professional developer who trusts LLMs to code. Everyone I know who uses them uses them as a tool and takes responsibility for cleaning up their output.
- Leherenn 3y agoCopilot likes to swallow my } in C++ for some reasons. E.g. if you generate a new function at the end of a namespace block, it will usually use the bracket of the namespace block at the terminator of the function, leaving you one short.
- chrisjj 3y ago> FWIW, I'd expect mismatched parentheses to be particularly likely to occur in Lisp. FWIW, this example had one more ) than ( ! I'll post it as a comment to the root.
- lolinder 3y agoYep, that's entirely unsurprising. Generative AI in general has a hard time knowing when to stop highly repetitive patterns—image generators struggle with getting the right number of fingers, and transformers will often get stuck in loops if things get too repetitive. Lisp is just a uniquely hard language for a transformer to cope with—not that it doesn't make mistakes in other languages, but I've almost never seen a syntax mistake.