4 ms·

You sound like you’ve had success using this tech for work. Can you tell more about your personal experience, please? I’ve tried ChatGPT a few times a year ago

You sound like you’ve had success using this tech for work. Can you tell more about your personal experience, please? I’ve tried ChatGPT a few times a year ago or so, but it was extremely frustrating, and I gave up.

simonw 2y ago
Here's my "How I use LLMs and ChatGPT" series: https://simonwillison.net/series/using-llms/ https://simonwillison.net/series/using-llms/ Also relevant is my ai-assisted-programming tag: https://simonwillison.net/tags/ai-assisted-programming/ https://simonwillison.net/tags/ai-assisted-programming/
- th0ma5 2y ago
  These examples are not much different than tutorial regurgitation imo. I would not pass this off as solving the problems that programmers have in practice, imo.
  - Wheatman 2y ago
    It is great for cheating at school, at least. And maybe boilerplate.
    - anticensor 2y ago
      That's one of business goals of OpenAI, which is to disrupt work and school.
  - simonw 2y ago
    Did you dismiss two years of my accumulated writing about LLMs as "tutorial regurgitation"? Maybe you clicked the link to my series of posts that are in chronological order - https://simonwillison.net/series/using-llms/ https://simonwillison.net/series/using-llms/ - looked at the very first one about trying out GPT-3 from June 2022 and stopped reading. I probably shouldn't share that link any more!
    - elliotto 2y ago
      I'm a big fan of your blog simonw, and it's interesting to see how a few of the people you've responded to here aren't receptive at all to your responses. I have become somewhat convinced that the 'ai doesn't help with programming crowd' is a little bit obtuse / entirely unwilling to experiment with new tools. It seems too much of a coincidence to see the same crowd that you've responded to struggle to perform basic website navigation and take anything away from your blog posts.
      - deleted 2y ago
        [deleted]
      - th0ma5 2y ago
        I keep experimenting because of comments like this and they still always have the same problems: - Inability to stay self-consistent - Random extra unused and unreachable code - Constant wack a mole with issues they have - Random opposite or adjacent thinking - Assuming that how things work in one library or language must work the same in another I then had someone say well... Just break up the work, but, that's also not what people say works for them, and also, you still have to side-eye the smaller stuff and it just gets exhausting. A ton of studies show that if you know less than 80 about a field it'll get you to that 80 percent. And if you know more than that it'll bring you back down to 80% due to randomness, lack of data in the training, incompleteness in the model that your incomplete prompt amplifies, and just automation bias errors where you trusted something it did that you shouldn't have. I would say that anyone claiming these things do everything are just simply new to systems.
    - th0ma5 2y ago
      [flagged]
- scott01 2y ago
  Sorry, it’s a bit hard to understand from your blog. I’m a bit dyslexic, so there’s a chance I’ve mussed something obvious, but I couldn’t find any examples beyond ‘explain this short snippet’. Also, are your posts AI assisted? I find it hard to read. I’ve tried asking ChatGPT questions about how a large codebase works, what entities should I use to implement some feature, etc. After a couple hours I realised the chat was hallucinating like crazy and turned into a yes-man, confirming all questions that assumed a yes-no answer.
  - rkozik1989 2y ago
    The problem with your example of applying analyzing something as complex and esoteric as a codebase is that LLMs cannot reason they simply return a response based on statistical inference, so unless you followed a standard like PSR for PHP and implemented it to a 't' it simply doesn't have to context to do what you're asking it to do. If you want an LLM to be an effective programmer for a specific application you'd probably need to fine tune and provide it instructions on your coding standards. Basically, how I've become successful using LLMs is that I solve the problem at a 9,000ft view, instruct the LLM to play different personas, have the personas validate my solution, and then instruct the LLM step-by-step to do all of the monkey work. Which doesn't necessarily always save me time upfront but in the long run it does because it makes fewer mistakes implementing my thought experiment.
    - scott01 2y ago
      Fair enough, I might be asking too much indeed, and may not be able to come up with an idea how LLMs can help me. For me, writing code is easy as soon as I understand the problem, and I sometimes spend a lot of time trying to figure out a solution that fits well within the context, so I thought I could ask an LLM what different things do and mean to help me understanding the problem surface better. Again, I may not understand something, but, at this point, I don’t understand what’s the value of code generation after I know how to solve a problem. Do you happen to have a blog post or something showing a concrete problem that LLM helped you to solve?
  - simonw 2y ago
rkozik1989 2y ago
Solve the problem at a high-level in your head, ask the LLM if your concept is correct, and then instruct the LLM to build the solution step-by-step.

simonw 2y ago

Here's my "How I use LLMs and ChatGPT" series: https://simonwillison.net/series/using-llms/ https://simonwillison.net/series/using-llms/ Also relevant is my ai-assisted-programming tag: https://simonwillison.net/tags/ai-assisted-programming/ https://simonwillison.net/tags/ai-assisted-programming/

th0ma5 2y ago
These examples are not much different than tutorial regurgitation imo. I would not pass this off as solving the problems that programmers have in practice, imo.
- Wheatman 2y ago
  It is great for cheating at school, at least. And maybe boilerplate.
  - anticensor 2y ago
    That's one of business goals of OpenAI, which is to disrupt work and school.
- simonw 2y ago
  Did you dismiss two years of my accumulated writing about LLMs as "tutorial regurgitation"? Maybe you clicked the link to my series of posts that are in chronological order - https://simonwillison.net/series/using-llms/ https://simonwillison.net/series/using-llms/ - looked at the very first one about trying out GPT-3 from June 2022 and stopped reading. I probably shouldn't share that link any more!
  - elliotto 2y ago
    I'm a big fan of your blog simonw, and it's interesting to see how a few of the people you've responded to here aren't receptive at all to your responses. I have become somewhat convinced that the 'ai doesn't help with programming crowd' is a little bit obtuse / entirely unwilling to experiment with new tools. It seems too much of a coincidence to see the same crowd that you've responded to struggle to perform basic website navigation and take anything away from your blog posts.
    - deleted 2y ago
      [deleted]
    - th0ma5 2y ago
      I keep experimenting because of comments like this and they still always have the same problems: - Inability to stay self-consistent - Random extra unused and unreachable code - Constant wack a mole with issues they have - Random opposite or adjacent thinking - Assuming that how things work in one library or language must work the same in another I then had someone say well... Just break up the work, but, that's also not what people say works for them, and also, you still have to side-eye the smaller stuff and it just gets exhausting. A ton of studies show that if you know less than 80 about a field it'll get you to that 80 percent. And if you know more than that it'll bring you back down to 80% due to randomness, lack of data in the training, incompleteness in the model that your incomplete prompt amplifies, and just automation bias errors where you trusted something it did that you shouldn't have. I would say that anyone claiming these things do everything are just simply new to systems.
  - th0ma5 2y ago
    [flagged]
scott01 2y ago
Sorry, it’s a bit hard to understand from your blog. I’m a bit dyslexic, so there’s a chance I’ve mussed something obvious, but I couldn’t find any examples beyond ‘explain this short snippet’. Also, are your posts AI assisted? I find it hard to read. I’ve tried asking ChatGPT questions about how a large codebase works, what entities should I use to implement some feature, etc. After a couple hours I realised the chat was hallucinating like crazy and turned into a yes-man, confirming all questions that assumed a yes-no answer.
- rkozik1989 2y ago
  The problem with your example of applying analyzing something as complex and esoteric as a codebase is that LLMs cannot reason they simply return a response based on statistical inference, so unless you followed a standard like PSR for PHP and implemented it to a 't' it simply doesn't have to context to do what you're asking it to do. If you want an LLM to be an effective programmer for a specific application you'd probably need to fine tune and provide it instructions on your coding standards. Basically, how I've become successful using LLMs is that I solve the problem at a 9,000ft view, instruct the LLM to play different personas, have the personas validate my solution, and then instruct the LLM step-by-step to do all of the monkey work. Which doesn't necessarily always save me time upfront but in the long run it does because it makes fewer mistakes implementing my thought experiment.
  - scott01 2y ago
    Fair enough, I might be asking too much indeed, and may not be able to come up with an idea how LLMs can help me. For me, writing code is easy as soon as I understand the problem, and I sometimes spend a lot of time trying to figure out a solution that fits well within the context, so I thought I could ask an LLM what different things do and mean to help me understanding the problem surface better. Again, I may not understand something, but, at this point, I don’t understand what’s the value of code generation after I know how to solve a problem. Do you happen to have a blog post or something showing a concrete problem that LLM helped you to solve?
- simonw 2y ago

rkozik1989 2y ago

Solve the problem at a high-level in your head, ask the LLM if your concept is correct, and then instruct the LLM to build the solution step-by-step.