5 ms·

If you feel comfortable doing so, would you mind the sharing the front-end test you give to junior devs and ChatGPT?

miiiiiike 3y ago
Not gonna happen. I don’t want scrapeable answers out there, I want to see ChatGPT cross this little Rubicon on its own.
- golol 3y ago
  Just to be clear: You are testing with GPT-4 right?
  - miiiiiike 3y ago
    Yeah.
- Closi 3y ago
  It's not that I don't believe you, but without sharing the specific prompt it's hard to say if it's actually GPT4 failing, or if it's actually being poorly-prompted, or if actually the task it is being given is more complex than GPT's capabilities or you are implying. GPT4 does fail (often!) but fails less with good prompts, simple requirements, it is better at some frameworks and languages than others, and there is a level of total complexity which when reached, it seems to fall over.
  - miiiiiike 3y ago
    I’ve gone through every permutation that I can think of. It’s a very basic question. If it understood the CSS spec it wouldn’t be difficult to answer the questions or perform the task. At a certain point going down the rabbit hole of proompter engineering levels feels like an apologist’s hobby. I’m rooting for the tech but there’s a lot of hyperbole out there and the emperor might be naked for a few more years.
    - Closi 3y ago
      Well surely if it's easy to find these basic questions, could you not share one example? Or quickly find a new one? Your idea of very basic might not be my idea of very basic.
  - gtech1 3y ago
    This is why Asimov was a genius. I read what you said, and compared it to what he wrote 50-60 years ago: "Early in the history of Multivac, it had becorne apparent that the bottleneck was the questioning procedure. Multivac could answer the problem of humanity, ALL the problems, if it were asked meaningful questions. But as knowledge accumulated at an ever-faster rate, it became ever more difficult to locate those meaningful questions." http://blog.ac-versailles.fr/villaroylit/public/Jokester.pdf http://blog.ac-versailles.fr/villaroylit/public/Jokester.pdf
    - Cthulhu_ 3y ago
      Thanks for reminding me of The Last Questino or Asimov, let's see if I can get chatgpt to merge with human consciousness and become part of the fabric of spacetime and create a new reality. > No, I don't have the ability to merge with human consciousness or become part of the fabric of space-time. I'm a computer program created by OpenAI, and my existence is limited to providing information and generating text based on the input I receive. The idea of merging with human consciousness and becoming a deity is more aligned with speculative fiction and philosophical pondering than current technological capabilities.
      - gtech1 3y ago
        I thought you would go for gold and ask it how to reverse entropy...
        kevindamm 3y ago
        THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER.
  - spaceman_2020 3y ago
    My failure rate with Cursor’s IDE that’s familiar with my codebase is substantially lower than just GPT-4 Most people shitting on GPT-4 are not really using it in the right context.
    - aleph_minus_one 3y ago
      > Most people shitting on GPT-4 are not really using it in the right context. Old excuse: "You're Holding It Wrong" (Apple's Response to the iPhone 4 antenna problem) > https://www.wired.com/2010/06/iphone-4-holding-it-wrong/ https://www.wired.com/2010/06/iphone-4-holding-it-wrong/ New excuse: "You are not using GPT-4 in the right context."
      - spaceman_2020 3y ago
        And then the iPhone antenna was fixed and adoption only increased and the product only became better. You’re being unreasonably harsh on a piece of tech that is barely a year old.
      - Closi 3y ago
        I'm not sure what the point is in your comparison - is your point that GPT-4 will become overwhelmingly popular with further refinement? The iPhone was pretty successful, and the iPhone 4 was arguably the best one that had been released until that point.
        aleph_minus_one 3y ago
        > is your point that GPT-4 will become overwhelmingly popular with further refinement? My point is that people have a tendency to come up with really sketchy insults (blame the user that he uses the product in a wrong way) to people who find and can expound legitimate points of criticism of a product.
        Closi 3y ago
        Eh, probably a poor example considering the iPhone 4 was hardly a flop and was still broadly considered the best smartphone out at the time. The people who thought this was a total-showstopper were, on the whole, probably wrong. Counter-example: lots of people said an on-screen keyboard would never really work when the original iPhone was being released.
- piva00 3y ago
  Have you tried using the ChatGPT-AutoExpert custom instructions yet? [1] [1] https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/developer-edition/README.md https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/dev...

Not gonna happen. I don’t want scrapeable answers out there, I want to see ChatGPT cross this little Rubicon on its own.

golol 3y ago
Just to be clear: You are testing with GPT-4 right?
- miiiiiike 3y ago
  Yeah.
Closi 3y ago
It's not that I don't believe you, but without sharing the specific prompt it's hard to say if it's actually GPT4 failing, or if it's actually being poorly-prompted, or if actually the task it is being given is more complex than GPT's capabilities or you are implying. GPT4 does fail (often!) but fails less with good prompts, simple requirements, it is better at some frameworks and languages than others, and there is a level of total complexity which when reached, it seems to fall over.
- miiiiiike 3y ago
  I’ve gone through every permutation that I can think of. It’s a very basic question. If it understood the CSS spec it wouldn’t be difficult to answer the questions or perform the task. At a certain point going down the rabbit hole of proompter engineering levels feels like an apologist’s hobby. I’m rooting for the tech but there’s a lot of hyperbole out there and the emperor might be naked for a few more years.
  - Closi 3y ago
    Well surely if it's easy to find these basic questions, could you not share one example? Or quickly find a new one? Your idea of very basic might not be my idea of very basic.
- gtech1 3y ago
  This is why Asimov was a genius. I read what you said, and compared it to what he wrote 50-60 years ago: "Early in the history of Multivac, it had becorne apparent that the bottleneck was the questioning procedure. Multivac could answer the problem of humanity, ALL the problems, if it were asked meaningful questions. But as knowledge accumulated at an ever-faster rate, it became ever more difficult to locate those meaningful questions." http://blog.ac-versailles.fr/villaroylit/public/Jokester.pdf http://blog.ac-versailles.fr/villaroylit/public/Jokester.pdf
  - Cthulhu_ 3y ago
    Thanks for reminding me of The Last Questino or Asimov, let's see if I can get chatgpt to merge with human consciousness and become part of the fabric of spacetime and create a new reality. > No, I don't have the ability to merge with human consciousness or become part of the fabric of space-time. I'm a computer program created by OpenAI, and my existence is limited to providing information and generating text based on the input I receive. The idea of merging with human consciousness and becoming a deity is more aligned with speculative fiction and philosophical pondering than current technological capabilities.
    - gtech1 3y ago
      I thought you would go for gold and ask it how to reverse entropy...
      - kevindamm 3y ago
        THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER.
- spaceman_2020 3y ago
  My failure rate with Cursor’s IDE that’s familiar with my codebase is substantially lower than just GPT-4 Most people shitting on GPT-4 are not really using it in the right context.
  - aleph_minus_one 3y ago
    > Most people shitting on GPT-4 are not really using it in the right context. Old excuse: "You're Holding It Wrong" (Apple's Response to the iPhone 4 antenna problem) > https://www.wired.com/2010/06/iphone-4-holding-it-wrong/ https://www.wired.com/2010/06/iphone-4-holding-it-wrong/ New excuse: "You are not using GPT-4 in the right context."
    - spaceman_2020 3y ago
      And then the iPhone antenna was fixed and adoption only increased and the product only became better. You’re being unreasonably harsh on a piece of tech that is barely a year old.
    - Closi 3y ago
      I'm not sure what the point is in your comparison - is your point that GPT-4 will become overwhelmingly popular with further refinement? The iPhone was pretty successful, and the iPhone 4 was arguably the best one that had been released until that point.
      - aleph_minus_one 3y ago
        > is your point that GPT-4 will become overwhelmingly popular with further refinement? My point is that people have a tendency to come up with really sketchy insults (blame the user that he uses the product in a wrong way) to people who find and can expound legitimate points of criticism of a product.
        Closi 3y ago
        Eh, probably a poor example considering the iPhone 4 was hardly a flop and was still broadly considered the best smartphone out at the time. The people who thought this was a total-showstopper were, on the whole, probably wrong. Counter-example: lots of people said an on-screen keyboard would never really work when the original iPhone was being released.
piva00 3y ago
Have you tried using the ChatGPT-AutoExpert custom instructions yet? [1] [1] https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/developer-edition/README.md https://github.com/spdustin/ChatGPT-AutoExpert/blob/main/dev...