4 ms·

> I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. So the

by alpos 4y ago

> I've already run into scenarios where ChatGPT generated code that looked perfectly plausible, except for that the actual API used didn't really exist. So the next question has to be: Was this still the right answer? I've personally had plenty of instances in my programming career where the code I was working on really needed functions which were best shopped out to a common API. To avoid interrupting my flow and to better inform the API I'd be designing for this, I just continued to write as if the API did exist. Then I went on to implement the functions that were needed. Perhaps the bot was right to presume that there should be an API for this. You might even be able to then prompt ChatGPT to create each of the functions in that API.

lolinder 4y ago
Well, this is ChatGPT, not Copilot, so I'd assume that OP was looking for a snippet using a public library rather than an internal API. In that context, suggesting you use an API that doesn't exist is just wrong. I've definitely done this with Copilot, though—it will suggest an API that doesn't actually exist but logically should in order to be consistent, and I'll go create it.
- alpos 4y ago
  That seems more like misplaced expectations. Someone may have given you the impression that copilot was supposed to do things like that where that expectation seems to not be present for you in relation to ChatGPT. However, as far as I know, the OpenAI team has not made it a goal to have ChatGPT only produce functional code using existing APIs. So I'm not sure we can call that an incorrect answer based on context. If the API it demonstrated using logically should exist, it seems like the right answer is still to just go create it.
  - lolinder 4y ago
    If I asked a coworker how to do X in framework Y, giving me the name of a function that should exist but doesn't is not a correct answer. If they told me "well, that's the function that should exist, you should just go submit a PR to framework Y", I would stop asking that coworker for help. The difference I was drawing between ChatGPT and Copilot wasn't that Copilot has functionality ChatGPT doesn't, it's that it has context ChatGPT doesn't, so it suggests things related to internal APIs. In a conversation with ChatGPT it would be very difficult to get help with internal APIs, hence my assumption that OP wasn't referring to APIs they have any control over.
    - alpos 4y ago
      I was not suggesting the bot was telling the user to go make a PR to an open source framework but rather that they could create a library that contains those functions if that was the logical thing to do. Which is why I asked if that actually seemed the longer term right thing to do. While I can easily agree that Copilot is probably the better tool for such questions, it is not clear from the parent comment whether their prompt to ChatGPT was asking to create code to do X or create code to do X using only existing publicly available libraries. It's not immediately obvious that the bot failed to understand the question or that the answer was an example of the bot failing to understand the programming language. It could easily be that the user had an implied expectation of usability in a context they did not give to the bot. That scenario is more like you asking a random person on the street, who happens to know Y framework, how to do X in that framework. Your coworker can be expected to get that you are looking for an answer that gets your current task done faster than you would be able to do without their assistance. The person on the street could not reasonably be expected to get that unless you give them that context.
dangom 4y ago
Exactly, that there is an end to the rabbit hole is a limitation of today's models. If something does not exist, it should be generated on the spot. GPT5 should check for the existence of an API and if it exists, test and validate it. If it fails tests or doesn't exist, create it.

by alpos 4y ago

lolinder 4y ago

Well, this is ChatGPT, not Copilot, so I'd assume that OP was looking for a snippet using a public library rather than an internal API. In that context, suggesting you use an API that doesn't exist is just wrong. I've definitely done this with Copilot, though—it will suggest an API that doesn't actually exist but logically should in order to be consistent, and I'll go create it.

alpos 4y ago
That seems more like misplaced expectations. Someone may have given you the impression that copilot was supposed to do things like that where that expectation seems to not be present for you in relation to ChatGPT. However, as far as I know, the OpenAI team has not made it a goal to have ChatGPT only produce functional code using existing APIs. So I'm not sure we can call that an incorrect answer based on context. If the API it demonstrated using logically should exist, it seems like the right answer is still to just go create it.
- lolinder 4y ago
  If I asked a coworker how to do X in framework Y, giving me the name of a function that should exist but doesn't is not a correct answer. If they told me "well, that's the function that should exist, you should just go submit a PR to framework Y", I would stop asking that coworker for help. The difference I was drawing between ChatGPT and Copilot wasn't that Copilot has functionality ChatGPT doesn't, it's that it has context ChatGPT doesn't, so it suggests things related to internal APIs. In a conversation with ChatGPT it would be very difficult to get help with internal APIs, hence my assumption that OP wasn't referring to APIs they have any control over.
  - alpos 4y ago
    I was not suggesting the bot was telling the user to go make a PR to an open source framework but rather that they could create a library that contains those functions if that was the logical thing to do. Which is why I asked if that actually seemed the longer term right thing to do. While I can easily agree that Copilot is probably the better tool for such questions, it is not clear from the parent comment whether their prompt to ChatGPT was asking to create code to do X or create code to do X using only existing publicly available libraries. It's not immediately obvious that the bot failed to understand the question or that the answer was an example of the bot failing to understand the programming language. It could easily be that the user had an implied expectation of usability in a context they did not give to the bot. That scenario is more like you asking a random person on the street, who happens to know Y framework, how to do X in that framework. Your coworker can be expected to get that you are looking for an answer that gets your current task done faster than you would be able to do without their assistance. The person on the street could not reasonably be expected to get that unless you give them that context.

dangom 4y ago

Exactly, that there is an end to the rabbit hole is a limitation of today's models. If something does not exist, it should be generated on the spot. GPT5 should check for the existence of an API and if it exists, test and validate it. If it fails tests or doesn't exist, create it.