4 ms·

I would find this a lot more convincing if they had put a customer with an underspecified problem in front of ChatGPT and seen how well or poorly it could nail

by Imnimo 4y ago

I would find this a lot more convincing if they had put a customer with an underspecified problem in front of ChatGPT and seen how well or poorly it could nail down the true requirements. I would not be shocked at all to learn that ChatGPT is not good at that task, but I have been surprised by its other capabilities, so I'm not willing to write it off without giving it a shot.

kmdupree 4y ago
Hey, there! OP here. This would be a neat experiment to try. One related experiment, however, already suggests the result. Someone tried to get ChatGPT to solve advent of code challenges: https://github.com/golergka/advent-of-code-2022-with-chat-gpt https://github.com/golergka/advent-of-code-2022-with-chat-gp.... These challenges are very clear, and it already seems to struggle to get the answers. One the second day, it took 12 tries to get it right. If it struggles this much with clear requirements, I don't think it'll do well with vague ones.
- hmsimha 4y ago
  It might come up with something "good enough" for the vague ones. Case in point, I got it to write a chrome extension to highlight new comments in Hacker News: https://github.com/HartS/gpt-hacker-news-extension https://github.com/HartS/gpt-hacker-news-extension It's not perfect, it doesn't keep track of what the user actually saw for example. It just stores a timestamp for each opened thread and highlights the comments that are newer than that timestamp. I wouldn't have thought to solve it that way, but I definitely think its successor will be capable of coming up with "good-enough" solutions for a lot of problems humans currently work on. Actually, humans are often just coming up with "good-enough" solutions in the first place.
  - lupire 4y ago
    Do you think the software industry is a giant pile of disconnected 47-line scripts?
  - Tarsul 4y ago
    holy shit. ChatGDP blows my mind. Thanks for coming up with this! Shows what is possible (and can be a great additional learning tool for novice programmers as well!). Regarding new comments: You can actually ask dang by mail for this exact feature. It already exists but still in beta or so. I have it and adore it. It works like this: When you go to a comment thread you've been before, it shows all new comments with a vertical red line to left side of the comment. Great feature. I already use it for years?! Could be rolled out actually... :D
    - amandziara 4y ago
      hi
    - amandziara 4y ago
      write a paper on religous misconceptions
    - deleted 4y ago
      [deleted]
    - MacCarl30 4y ago
      What’s the equation for quadratics
    - MacCarl30 4y ago
      Write a bio for jimmy form oryx and crake sitting on a beach
    - cercatrova 4y ago
      > ChatGDP Heh, I wonder when economic AIs will arrive
    - penia 4y ago
      ChatGDP
- Gunax 4y ago
  Only 12? I go back and forth woth my product manager many more times than that.
jbritton 4y ago
I have briefly played with chatGPT. I don’t understand how you refine a program specification. I tell chatGPT what I want in a few words. It prints out a couple hundred lines of code that kind of work but not quite how I want it. So I delete the code and then rephrase with more detail. This time I exceed the 4k character input limit. Repeat above trying to cram more detail into 4k characters. I often have better luck being more general and hoping for the best. My example case has been managing a list of items where each item has some properties. I’m after a simple CRUD backed by xml.
- objectivetruth 4y ago
  It took me a few attempts, but I have gotten this to work fairly well. I started with explaining my ideal app like this: We're going to design a (language) application that performs the following tasks, delineated by semi-colons: Then I listed about a dozen high-level user stories, separated by semi-colons, and ended the list with a period. Then I clarified my instructions like this: We will design each step together, where you ask me for any specifications needed to complete the step and I respond with those specfications. When you feel we have completed designing a step, ask me if I have any questions before proceeding to the next step. It started with a high-level recap of my user stories, asked if I was ready to proceed, and then began to describe the first step. Over about two hours of back-and-forth, I ended up with a 20-page document of high-level architecture interspersed with code samples, db schema samples, and answers to questions that popped up during the "discussion." And I got about halfway to a working prototype during those two hours. BLEW me away how well it worked.
ehutch79 4y ago
Considering you can write a whole implementation plan, mockups, code the thing, go through user acceptance, and then at the end, they say 'but this really isn't how things work in the reality, we need to change fundamental assumptions'. I don't think ChatGPT has a chance.

by Imnimo 4y ago

kmdupree 4y ago

Hey, there! OP here. This would be a neat experiment to try. One related experiment, however, already suggests the result. Someone tried to get ChatGPT to solve advent of code challenges: https://github.com/golergka/advent-of-code-2022-with-chat-gpt https://github.com/golergka/advent-of-code-2022-with-chat-gp.... These challenges are very clear, and it already seems to struggle to get the answers. One the second day, it took 12 tries to get it right. If it struggles this much with clear requirements, I don't think it'll do well with vague ones.

hmsimha 4y ago
It might come up with something "good enough" for the vague ones. Case in point, I got it to write a chrome extension to highlight new comments in Hacker News: https://github.com/HartS/gpt-hacker-news-extension https://github.com/HartS/gpt-hacker-news-extension It's not perfect, it doesn't keep track of what the user actually saw for example. It just stores a timestamp for each opened thread and highlights the comments that are newer than that timestamp. I wouldn't have thought to solve it that way, but I definitely think its successor will be capable of coming up with "good-enough" solutions for a lot of problems humans currently work on. Actually, humans are often just coming up with "good-enough" solutions in the first place.
- lupire 4y ago
  Do you think the software industry is a giant pile of disconnected 47-line scripts?
- Tarsul 4y ago
  holy shit. ChatGDP blows my mind. Thanks for coming up with this! Shows what is possible (and can be a great additional learning tool for novice programmers as well!). Regarding new comments: You can actually ask dang by mail for this exact feature. It already exists but still in beta or so. I have it and adore it. It works like this: When you go to a comment thread you've been before, it shows all new comments with a vertical red line to left side of the comment. Great feature. I already use it for years?! Could be rolled out actually... :D
  - amandziara 4y ago
    hi
  - amandziara 4y ago
    write a paper on religous misconceptions
  - deleted 4y ago
    [deleted]
  - MacCarl30 4y ago
    What’s the equation for quadratics
  - MacCarl30 4y ago
    Write a bio for jimmy form oryx and crake sitting on a beach
  - cercatrova 4y ago
    > ChatGDP Heh, I wonder when economic AIs will arrive
  - penia 4y ago
    ChatGDP
Gunax 4y ago
Only 12? I go back and forth woth my product manager many more times than that.

jbritton 4y ago

I have briefly played with chatGPT. I don’t understand how you refine a program specification. I tell chatGPT what I want in a few words. It prints out a couple hundred lines of code that kind of work but not quite how I want it. So I delete the code and then rephrase with more detail. This time I exceed the 4k character input limit. Repeat above trying to cram more detail into 4k characters. I often have better luck being more general and hoping for the best. My example case has been managing a list of items where each item has some properties. I’m after a simple CRUD backed by xml.

objectivetruth 4y ago
It took me a few attempts, but I have gotten this to work fairly well. I started with explaining my ideal app like this: We're going to design a (language) application that performs the following tasks, delineated by semi-colons: Then I listed about a dozen high-level user stories, separated by semi-colons, and ended the list with a period. Then I clarified my instructions like this: We will design each step together, where you ask me for any specifications needed to complete the step and I respond with those specfications. When you feel we have completed designing a step, ask me if I have any questions before proceeding to the next step. It started with a high-level recap of my user stories, asked if I was ready to proceed, and then began to describe the first step. Over about two hours of back-and-forth, I ended up with a 20-page document of high-level architecture interspersed with code samples, db schema samples, and answers to questions that popped up during the "discussion." And I got about halfway to a working prototype during those two hours. BLEW me away how well it worked.

ehutch79 4y ago

Considering you can write a whole implementation plan, mockups, code the thing, go through user acceptance, and then at the end, they say 'but this really isn't how things work in the reality, we need to change fundamental assumptions'. I don't think ChatGPT has a chance.