4 ms·
Ok but what are we calling API misuses? We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse p
by keyle 3y ago
Ok but what are we calling API misuses?
We collect 1208 coding questions from StackOverflow on 24 representative Java APIs. We summarize thecommon misuse patterns of these APIs and evaluate them oncurrent popular LLMs. The evaluation results show that evenfor GPT-4, 62% of the generated code contains API misuses,which would cause unexpected consequences if the code isintroduced into real-world software.
Should have used ChatGPT to correct those sentences. /s
What do we consider an API misuse at the end of the day? And also against the Java APIs... Which are some of the oldest.
I'm not saying ChatGPT answers work, 50% of the time they don't work out of the box. But overal, the time savings are incredible, compared to the fossil digging way of googling. And I value the time savings towards the equally correct way of googling for answers. You reap what you sow. The knowledge the LLM is based upon is the same stuff you used to distill manually. It shouldn't be more or less correct, if you think of it that way.
Surprise, ChatGPT will not replace devs. But it is like having the electrical drill vs. screwdriver type of situation.
- makestuff 3y agoI have been trying to use ChatGPT to make a VueJS website (I’m a backend dev). What I have noticed with VueJS specifically is that it has little to no knowledge of the composition API, and when I ask it to help build a component it is pretty good at the scaffolding. The component will generally work; however, it is pretty useless for CSS styling. I probably should try Copilot as I bet that would work better. Also the VS Code integration would be really nice.
- keyle 3y agoCopilot probably wins because it holds more of _your_ code against the desired output. I find ChatGPT is really great at getting something going, think scaffolding wise. But if you enter the realm of gnarly stuff, it can get completely delirious with answers full of red herrings and sends you running in circles very quickly.
- coffeebeqn 3y agoThe only truly helpful use-case I’ve found is in generating some well defined text in a different form. Say - here’s a JSON object , please write a MD file describing it or turn it to X language struct/object.
- earthboundkid 3y agoChatGPT is very bad at CSS. The only good part is you can test it and see that it’s not working very quickly. Unfortunately if you go back to ChatGPT it will waste your time with endless apologies and more broken CSS.
- ramraj07 3y agoCan you confirm if you used gpt-4? This was not my experience
- simonw 3y agoHow stable was the Vue composition API prior to the September 2021 training cut-off date for the OpenAI models? Looks to me like https://blog.vuejs.org/posts/vue-2-7-naruto https://blog.vuejs.org/posts/vue-2-7-naruto back-ported it in July of 2022, and the earliest snapshot of the documentation for it in Vue 3 was https://web.archive.org/web/20220210121247/https://vuejs.org/guide/extras/composition-api-faq.html https://web.archive.org/web/20220210121247/https://vuejs.org... in February 2022 - so my guess is it mostly happened after that magic date.
- deleted 3y ago[deleted]
- wredue 3y agoI just don’t see how it could save time. Programmers nearly universally agree that reading code is harder than writing it. So when you have chat GPT writing your code, you have to read and understand it to ensure it’s actually doing what you need it to without awkward bugs or problems. Given that it’s harder to read than write code, it seems to stand to reason that the “time savings” must be nil.
- Yiin 3y agoTo write the code you need to know what to write. In many cases I find it easier to read proposed solution to my problem and then figure out from there how I actually want to implement a solution. It's great starting point.
- sharemywin 3y agoChatGPT is pretty good about telling you what a line of code does. So I just copy it in and ask it.
- simonw 3y ago"I just don’t see how it could save time." Have you tried it much? It's saved me a ton of time over the past 8 months. I don't know what else to say - I'm not the kind of person to deceive myself into thinking something is saving me time when it isn't. The time saved is mainly in the micro-research you no longer have to do - the bits where you have to go and look up how to write a for loop in Bash, or how to call super() in a Python class, or which exceptions you need to catch for a call to httpx.get(). GPT-4 writes the code right 90% of the time. The 10% of the time it doesn't you catch in the same testing you would have done against code you had written yourself the long way.
- maxbond 3y agoI have found it to save time when I'm learning a new, widely used framework. (Presumably this would hold for a language as well.) I used it to learn Keras and React + Redux, and it was helpful there. For instance I was able to ask it about what sort of messages would be emitted by Redux in a given circumstance, and the precise values were wrong but it didn't really matter. That was very cool. Once I have a good sense for how things work, I'd much rather go to the documentation. Most of the time I can do this straight from my editor by typing the method I want and querying the LSP for documentation. In those cases it's more or less instant, there's no improvement to be made there. Maybe in some of the other cases it would be faster via an LLM integrated with my editor, but I don't think it would justify the cognitive load of considering whether or not it's correct. I'd like to worry about whether the application is correct. And I don't really think I'm wasting time waiting for the Python documentation to load, I'm continuing to chew on the problem. Additionally, when I go to the documentation, I'm looking for callouts about the safety of an API, and I take the lack of such callouts as authoritative. Eg, recently I was reading that in node-postgres you have to return connections to the pool, which was a good refresher for me, because previously I'd been using Rust where resources are generally cleaned up automatically. I just don't trust a lack of a callout from GPT-4 as being a confirmation of a lack of a safety issue.
- dventimi 3y agoEvidently, THEY are calling the following API misuses: "To evaluate the API usage correctness in code, RO- BUSTAPI detects the API misuses against the API usage rules by extracting call consequences and control structures from the code snippet, as shown in Figure 2. The code checker firstly check the code snippets to see whether it is a snippet of a method or a method of a class, so that it can enclose this code snippet and constructs abstract syntax tree (AST) from the code snippet. Then the checker traverses the AST to record all the method calls and control structures in order, which generate a call sequence. Next, the checker compare the call sequence against the API usage rules. It infers the instance type of each method calls and use the type and method as keys to retrieve corresponding API us- age rules. Finally, the checker computes the longest common sequence between the call sequence and the API usage rules. If the call sequence does not match the expected API usage rules, the checker will report API misuses."