4 ms·
TFA actually has four key findings: (1) ChatGPT is better at generating functionally correct code for problems before 2021 in different languages than problems
by KineticLensman 2y ago
TFA actually has four key findings:
(1) ChatGPT is better at generating functionally correct code for problems before 2021 in different languages than problems after 2021 with 48% advantage in Accepted rate on judgment platform, but ChatGPT 's ability to directly fix erroneous code with multi-round fixing process to achieve correct functionality is relatively weak;
(2) the distribution of cyclomatic and cognitive complexity levels for code snippets in different languages varies. Furthermore, the multi-round fixing process with ChatGPT generally preserves or increases the complexity levels of code snippets;
(3) in algorithm scenarios with languages of C, C++, and Java, and CWE scenarios with languages of C and Python3, the code generated by ChatGPT has relevant vulnerabilities. However, the multi-round fixing process for vulnerable code snippets demonstrates promising results, with more than 89% of vulnerabilities successfully addressed; and
(4) code generation may be affected by ChatGPT 's non-determinism factor, resulting in variations of code snippets in functional correctness, complexity, and security. Overall, our findings uncover potential issues and limitations that arise in the ChatGPT -based code generation and lay the groundwork for improving AI and LLM-based code generation techniques.
- p1esk 2y agoThey used GPT 3.5 so these findings are irrelevant.
- lolinder 2y agoThe AI community is worse than JavaScript for acting like everything needs to be thrown out every six months. Aside from the very important fact that GPT-3.5 is still far and away the most frequently used LLM model, it's not like GPT-4 has a completely different architecture with completely different characteristics. It's clearly better, but much of what they describe should generalize to LLMs as a whole (for example, knowledge cutoff dates matter a lot and these things likely have memorized a lot more than we thought they did).
- p1esk 2y agoIt doesn’t matter what is the most commonly used model. If you want to make claims about LLM capabilities you better be using the best model available.
- deleted 2y ago[deleted]