6 ms·

I have also many years of programming experience and find myself strongly "accelerated" by LLMs when writing code. But, if you think at it, it makes sense that

I have also many years of programming experience and find myself strongly "accelerated" by LLMs when writing code. But, if you think at it, it makes sense that many seasoned programmers are using LLMs better. LLMs are a helpful tool, but also a hard-to-use tool, and in general it's fair to think that better programmers can do a better use of some assistant (human or otherwise): better understanding its strengths, identifying faster the good and bad output, providing better guidance to correct the approach... Other than that, what correlates more strongly with the ability to use LLMs effectively is, I believe, language skills: the ability to describe problems very clearly. LLMs reply quality changes very significantly with the quality of the prompt. Experienced programmers that can also communicate effectively provide the model with many design hints, details where to focus, ..., basically escaping many local minima immediately.

bsenftner 2y ago
Communication skills are the keys to using LLMs. Think about it: every type of information you want is in them, in fact it is there multiple times, with multiple levels of seriousness in the treatment of the idea. If one is casual in their request, using casual language, then the LLM will reply with a casual reply because that matched your request best. To get a hard, factual answer from those that are experts in a subject, use the formal term, use the expert's language and you'll get back a rely more likely to be correct because it's in the same level of formal treatment as correct answers.
- psychoslave 2y ago
  >every type of information you want is in them Actually, I'm afraid that no. It won't give us the step by step scalable processes to make humanity as a whole enter in a loop of indefinitely long period of world peace, with each of us enjoying life in its own thriving manner. That would be great information to broadcast, though. Also it equally has ability to produce large pile of completely delusional answers, that mimics just as well genuinely sincere statements. Of course, we can also receive that kind of misguiding answers from humans. But the amount of output that mere humans can throw out in such a form is far more limited. All that said, it's great to be able to experiment with it, and there are a lot of nice and fun things to do with it. It can be a great additional tool, but it won't be a self-sufficient panacea of information source.
  - bsenftner 2y ago
    > It won't give us the step by step scalable processes to make humanity as a whole enter in a loop of indefinitely long period of world peace That's not anywhere, that's a totally unsolved and open ended problem, why would you think an LLM would have that?
    - fmbb 2y ago
      If what you meant was > Think about it: every type of already solved problem you want information about is in them, in fact it is there multiple times, with multiple levels of seriousness in the treatment of the idea. then that was not clear from your comment saying LLMs contain any information you want. One has to be careful communicating about LLms because the world is full of people that actually believe LLMs are generally intelligent super beings.
      - numpad0 2y ago
        I think GP's saying that it must be in your prompt, not in the weights. If you want LLM make sandwich, you have to tell them you `want triangular sandwiches of standard serving size made with white bread and egg based filling`, not `it's almost noon and I'm wondering if sandwich for lunch is a good idea`. Fine-tuning partially solves that problem but they still like the former.
  - arminiusreturns 2y ago
    After a small prompt engineering: https://0bin.net/paste/zolMrjVz#dgZrZzKU-PlxdkJTdG0pZU9bsCM302sJHIHtpVMkEa6 https://0bin.net/paste/zolMrjVz#dgZrZzKU-PlxdkJTdG0pZU9bsCM3...
    - psychoslave 2y ago
      Interesting, thanks for sharing. Could you also give some insights on the process you followed?
      - 2y ago
mhalle 2y ago
I completely agree that communication skills are critical in extracting useful work or insight from LLMs. The analogy for communicating with people is not far-fetched. Communicating successfully with a specific person requires an understanding of their strengths and weaknesses, their tendencies and blind spots. The same is true for communicating with LLMs. I have actually found that from a documentation point of view, querying LLMs has made me better and explaining things to people. If, given the documentation for a system or API, a modern LLM can't answer specific questions about how to perform a task, a person using the same documentation will also likely struggle. It's proving to be a good way to test the effectiveness of documentation, for humans and for LLMs.
LouisSayers 2y ago
> the ability to describe problems very clearly Yes, and to provide enough context. There's probably a lot that experience is contributing to the interaction as well, for example - knowing when the LLM has gone too far, focusing on what's important vs irrelevant to the task, modularising and refactoring code, testing etc
gen220 2y ago
Hey! Asking because I know you're a fellow vimmer [0]. Have you integrated LLMs into your editor/shell? Or are you largely copy-pasting context between a browser and vim? This context-switching of it all has been a slight hang-up for me in adopting LLMs. Or are you asking more strategic questions where copy-paste is less relevant? [0] your videos on writing systems software were part of what inspired me to make a committed switch into vim. thank you for those!
- qup 2y ago
  You want aider.
rudiksz 2y ago
> "seasoned programmers are using LLMs better". I do not remember a single instance when code provided to me by an LLM worked at all. Even if I ask something small that cand be done in 4-5 lines of code is always broken. From a fellow "seasoned" programmer to another: how the hell do you write the prompts to get back correct working code?
- throwaway4aday 2y ago
  Step 1: https://claude.ai https://claude.ai Step 2: Write out your description of the thing you want to the best of your ability but phrase it as "I would like X, could you please help me better define X by asking me a series of clarifying questions and probing areas of uncertainty." Step 3: Once both Claude and you are satisfied that X is defined, say "Please go ahead and implement X." Step 4a: If feature Y is incorrect, go to Step 2 and repeat the process for Y Step 4b: If there is a bug, describe what happened and ask Claude to fix it. That's the basics of it, should work most of the time.
- jkaptur 2y ago
  The story from the article matches my experience. The LLM's first answer is often a little broken, so I tweak it until it's actually correct.
- numpad0 2y ago
  dc: not a seasoned dev, with <b> and <h1> tags on "not". They can't think for you. All intelligent thinking you have to do. First, give them high level requirement that can be clarified into indented bullet points that looks like code. Or give them such list directly. Don't give them half-open questions usually favored by talented and autonomous individuals. Then let them further decompress that pseudocode bullet points into code. They'll give you back code that resemble a digitized paper test answer. Fix obvious errors and you get a B grade compiling code. They can't do non-conventional structures, Quake style performance optimized codes, realtime robotics, cooperative multithreading, etc., just good old it takes what it takes GUI app API and data manipulation codes. For those use cases with these points in mind, it's a lot faster to let LLM generate tokens than typing `int this_mandatory_function_does_obvious (obvious *obvious){ ...` manually on a keyboard. That should arguably be a productivity boost in the sense that the user of LLM is effectively typing faster.
kragen 2y ago
That's really interesting. What are the most important things you've learned to do with the LLMs to get better results? What do your problem descriptions look like? Are you going back and forth many times, or crafting an especially-high-quality initial prompt?
- antirez 2y ago
  I'm posting a set of videos on my YT channel where I'll show the process I follow. Thanks!
  - kragen 2y ago
    That's fantastic! I thought about asking if you had streamed any of it, but I didn't want to sound demanding and entitled :)

bsenftner 2y ago

Communication skills are the keys to using LLMs. Think about it: every type of information you want is in them, in fact it is there multiple times, with multiple levels of seriousness in the treatment of the idea. If one is casual in their request, using casual language, then the LLM will reply with a casual reply because that matched your request best. To get a hard, factual answer from those that are experts in a subject, use the formal term, use the expert's language and you'll get back a rely more likely to be correct because it's in the same level of formal treatment as correct answers.

psychoslave 2y ago
>every type of information you want is in them Actually, I'm afraid that no. It won't give us the step by step scalable processes to make humanity as a whole enter in a loop of indefinitely long period of world peace, with each of us enjoying life in its own thriving manner. That would be great information to broadcast, though. Also it equally has ability to produce large pile of completely delusional answers, that mimics just as well genuinely sincere statements. Of course, we can also receive that kind of misguiding answers from humans. But the amount of output that mere humans can throw out in such a form is far more limited. All that said, it's great to be able to experiment with it, and there are a lot of nice and fun things to do with it. It can be a great additional tool, but it won't be a self-sufficient panacea of information source.
- bsenftner 2y ago
  > It won't give us the step by step scalable processes to make humanity as a whole enter in a loop of indefinitely long period of world peace That's not anywhere, that's a totally unsolved and open ended problem, why would you think an LLM would have that?
  - fmbb 2y ago
    If what you meant was > Think about it: every type of already solved problem you want information about is in them, in fact it is there multiple times, with multiple levels of seriousness in the treatment of the idea. then that was not clear from your comment saying LLMs contain any information you want. One has to be careful communicating about LLms because the world is full of people that actually believe LLMs are generally intelligent super beings.
    - numpad0 2y ago
      I think GP's saying that it must be in your prompt, not in the weights. If you want LLM make sandwich, you have to tell them you `want triangular sandwiches of standard serving size made with white bread and egg based filling`, not `it's almost noon and I'm wondering if sandwich for lunch is a good idea`. Fine-tuning partially solves that problem but they still like the former.
- arminiusreturns 2y ago
  After a small prompt engineering: https://0bin.net/paste/zolMrjVz#dgZrZzKU-PlxdkJTdG0pZU9bsCM302sJHIHtpVMkEa6 https://0bin.net/paste/zolMrjVz#dgZrZzKU-PlxdkJTdG0pZU9bsCM3...
  - psychoslave 2y ago
    Interesting, thanks for sharing. Could you also give some insights on the process you followed?
    - 2y ago

mhalle 2y ago

I completely agree that communication skills are critical in extracting useful work or insight from LLMs. The analogy for communicating with people is not far-fetched. Communicating successfully with a specific person requires an understanding of their strengths and weaknesses, their tendencies and blind spots. The same is true for communicating with LLMs. I have actually found that from a documentation point of view, querying LLMs has made me better and explaining things to people. If, given the documentation for a system or API, a modern LLM can't answer specific questions about how to perform a task, a person using the same documentation will also likely struggle. It's proving to be a good way to test the effectiveness of documentation, for humans and for LLMs.

LouisSayers 2y ago

> the ability to describe problems very clearly Yes, and to provide enough context. There's probably a lot that experience is contributing to the interaction as well, for example - knowing when the LLM has gone too far, focusing on what's important vs irrelevant to the task, modularising and refactoring code, testing etc

gen220 2y ago

Hey! Asking because I know you're a fellow vimmer [0]. Have you integrated LLMs into your editor/shell? Or are you largely copy-pasting context between a browser and vim? This context-switching of it all has been a slight hang-up for me in adopting LLMs. Or are you asking more strategic questions where copy-paste is less relevant? [0] your videos on writing systems software were part of what inspired me to make a committed switch into vim. thank you for those!

qup 2y ago
You want aider.

rudiksz 2y ago

> "seasoned programmers are using LLMs better". I do not remember a single instance when code provided to me by an LLM worked at all. Even if I ask something small that cand be done in 4-5 lines of code is always broken. From a fellow "seasoned" programmer to another: how the hell do you write the prompts to get back correct working code?

throwaway4aday 2y ago
Step 1: https://claude.ai https://claude.ai Step 2: Write out your description of the thing you want to the best of your ability but phrase it as "I would like X, could you please help me better define X by asking me a series of clarifying questions and probing areas of uncertainty." Step 3: Once both Claude and you are satisfied that X is defined, say "Please go ahead and implement X." Step 4a: If feature Y is incorrect, go to Step 2 and repeat the process for Y Step 4b: If there is a bug, describe what happened and ask Claude to fix it. That's the basics of it, should work most of the time.
jkaptur 2y ago
The story from the article matches my experience. The LLM's first answer is often a little broken, so I tweak it until it's actually correct.
numpad0 2y ago
dc: not a seasoned dev, with <b> and <h1> tags on "not". They can't think for you. All intelligent thinking you have to do. First, give them high level requirement that can be clarified into indented bullet points that looks like code. Or give them such list directly. Don't give them half-open questions usually favored by talented and autonomous individuals. Then let them further decompress that pseudocode bullet points into code. They'll give you back code that resemble a digitized paper test answer. Fix obvious errors and you get a B grade compiling code. They can't do non-conventional structures, Quake style performance optimized codes, realtime robotics, cooperative multithreading, etc., just good old it takes what it takes GUI app API and data manipulation codes. For those use cases with these points in mind, it's a lot faster to let LLM generate tokens than typing `int this_mandatory_function_does_obvious (obvious *obvious){ ...` manually on a keyboard. That should arguably be a productivity boost in the sense that the user of LLM is effectively typing faster.

kragen 2y ago

That's really interesting. What are the most important things you've learned to do with the LLMs to get better results? What do your problem descriptions look like? Are you going back and forth many times, or crafting an especially-high-quality initial prompt?

antirez 2y ago
I'm posting a set of videos on my YT channel where I'll show the process I follow. Thanks!
- kragen 2y ago
  That's fantastic! I thought about asking if you had streamed any of it, but I didn't want to sound demanding and entitled :)