3 ms·

You are just bad with prompting or working with very obscure language/framework or bad coding pattern or all of it. I had a talk with a seasoned engineer who h

You are just bad with prompting or working with very obscure language/framework or bad coding pattern or all of it. I had a talk with a seasoned engineer who has been coding for 50 years and has created many amazing things over lifetime about him having really bad results with AI tools I suggested for him. When I use AI for the same purposes in the same repo he's working on, it works nicely. When he does it, results are always not what he wants. It comes down to a combination of him not understanding how to guide the LLMs to correct direction and using a language/framework (he's not familiar with) he can't judge the LLMs output. It is really important to know what you want, be able to describe it in short points (but important points). Points that you know ai will mess up if you don't specify. And also be able to figure out which direction the ai is heading with the solution and correct it EARLY rather than later. Not overloading context/memory with unnecessary things. Focusing on key areas to improve and much more. I'm using AI to get solutions done that I can definitely do myself but it'll take a certain amount of time to hunt down all documentation, API/lib calls etc. With AI, 1/10th time is enough. I've had massive success with java, js/TS, html css, go, rust, python, bitbucket pipelines/GitHub actions, cdk, docker compose, SQL, flutter/dart, swift etc.

CopyOnWrite 1y ago
I do not rule out, that I am just very bad with prompting. It just surprises me, that you write you had massive successes with "java, js/TS, html css, go, rust, python, bitbucket pipelines/GitHub actions, cdk, docker compose, SQL, flutter/dart, swift etc.", if you include the usual libraries/frameworks and the diverse application areas for these technologies, even with LLMs support it seems to me crazy to be able to make meaningful contributions in non trivial code bases. Concerning SQL I can report another fail with LLMs, in a trivial code base with a handful of entities the LLM cannot come up with basic window functions. I would be very interested if you could write up a blog post or could make a youtube video demonstrating your prompting skills... Perhaps demonstrating with a bigger open source project in any of the mentioned languages how to add a non trivial feature with your prompting skills?
- CrimsonRain 1y ago
  Unfortunately, I'm at a stage of personal life where I do not have time to blog. I'd love to but :( The stuff I work on for company is confidential and even getting authorization to use AI was such a hassle. Based on some of your replies, I think you have an impression of current generation AIs that is 100% wrong. I can not blame you as the impression you have, is what the AI companies want you to have, that's what they are hyping. In another comment, you mentioned someone should demo how AI can add a non-leaf feature to a non-trivial LOC codebase. This is what AI companies say AI can do. But the truth is, (current gen) AIs can not do this except a few rare cases. I can not demo this to you as I can't do this and do not attempt to do it either on day to day tasks. The issue is context. What you are asking requires AI to have a huge amount of context that it simply is not equipped to handle (at least not right now). What AIs are really good at is to do small fragment of a task given enough clear requirements. When I want AI to write a Handler in my Controller, I don't just ask it to "write a function to handle POST call for entity E." I write the javadoc /* */ comment that defines the signature and explains a little about how the core process of this handling works. I can even copy/paste similar handler from another controller if I think that will help. Ultimately, my brain already knows the input, output and key transformations that needs to happen in this function. I just write minimal amount (esp comments) and get AI to complete the rest. So if I need to write a non-leaf feature, I will break it down to several leaf features and then pass it on to AI and if needed, manually assemble them. I had to write 500LOC bash script to handle a software deployment. This is not the way to do it but I was forced by circumstances created by someone else. Anyways, if I had to write the whole thing by hand, it'd take multiple days as bash syntax is not forgiving and the stuff I needed to do in the script were quite complex/crazy (and stupid). I think I wrote about 50+ lines of text describing the whole process, which you can think of as a requirement document. With a few tries, I was able to get the whole script with near accuracy. My reading revealed some issues. Pointed them to AI. It fixed them. Tests revealed some other issues. Again, AI fixed them after pointing out. I was able to get the whole thing done in just an hour or so.
douglasisshiny 1y ago
I've had the same experience as the person to whom you're responding. After reading your post, I have to ask: if you're putting so much effort into prompting it with specific points, correcting it often, etc., why not just write the code yourself? It sounds like you're putting a good deal of effort into prompting it. Aren't you worried that overtime you'll rely on it too much and your offhand knowledge will get worse?
- CopyOnWrite 1y ago
  I have read somewhere, that LLMs are mostly helpful to junior developers. Is it possible the person claiming success with all these languages/tools/technologies is just on a junior level and is subjectively correct but has no point of reference how fast coding is for seniors and how quality code looks like?
  - CrimsonRain 1y ago
    1. You have read somewhere...but you don't have any experience. LLMs are really bad for Junior developers. A. They have no skills/exp to judge AI output B. They don't learn from sudden magical wall of code output C. They don't get to explore; thus don't learn. D. Ultimately, LLMs act as a bad drug to them that keeps them dependent and stagnant. E. LLMs are really good for higher end of senior devs. This also means, they don't need as many Juniors anymore and they don't mentor juniors much. This is the biggest loss for Juniors. 2. I think you are absolutely right. I became a Staff Engineer with junior level LLM coding. More power to me I guess :(
- xandrius 1y ago
  Not OP, it be comes natural and doesn't take a lot of time. Anyway, if you want to, LLMs can today help with a ton of programming languages and frameworks. If you use any of the top 5 languages and it still doesn't work for you, either you're doing some esoteric work or you're doing it wrong.
  - CopyOnWrite 1y ago
    Could you point me to a youtube video or a blog post which demonstrates how LLMs help writing code which outperforms a proficient human? My only conditions: - It must be demonstrated by adding a feature on a bigger code base (>= 20 LOC) - The added feature cannot be a leaf feature (means it must integrate with the rest of the system at multiple points) - The prompting has to be less effort/faster than to type the solution in the programming language You can chose any programming language/framework that you want. I don't care if it is Java, JavaScript, Typescript, C, Python, ... hell, I am fine with any language with or w/o a framework.
thi2 1y ago
> You are just bad with prompting or working with very obscure language/framework or bad coding pattern or all of it You just described every existing legacy project^^

CopyOnWrite 1y ago

I do not rule out, that I am just very bad with prompting. It just surprises me, that you write you had massive successes with "java, js/TS, html css, go, rust, python, bitbucket pipelines/GitHub actions, cdk, docker compose, SQL, flutter/dart, swift etc.", if you include the usual libraries/frameworks and the diverse application areas for these technologies, even with LLMs support it seems to me crazy to be able to make meaningful contributions in non trivial code bases. Concerning SQL I can report another fail with LLMs, in a trivial code base with a handful of entities the LLM cannot come up with basic window functions. I would be very interested if you could write up a blog post or could make a youtube video demonstrating your prompting skills... Perhaps demonstrating with a bigger open source project in any of the mentioned languages how to add a non trivial feature with your prompting skills?

CrimsonRain 1y ago
Unfortunately, I'm at a stage of personal life where I do not have time to blog. I'd love to but :( The stuff I work on for company is confidential and even getting authorization to use AI was such a hassle. Based on some of your replies, I think you have an impression of current generation AIs that is 100% wrong. I can not blame you as the impression you have, is what the AI companies want you to have, that's what they are hyping. In another comment, you mentioned someone should demo how AI can add a non-leaf feature to a non-trivial LOC codebase. This is what AI companies say AI can do. But the truth is, (current gen) AIs can not do this except a few rare cases. I can not demo this to you as I can't do this and do not attempt to do it either on day to day tasks. The issue is context. What you are asking requires AI to have a huge amount of context that it simply is not equipped to handle (at least not right now). What AIs are really good at is to do small fragment of a task given enough clear requirements. When I want AI to write a Handler in my Controller, I don't just ask it to "write a function to handle POST call for entity E." I write the javadoc /* */ comment that defines the signature and explains a little about how the core process of this handling works. I can even copy/paste similar handler from another controller if I think that will help. Ultimately, my brain already knows the input, output and key transformations that needs to happen in this function. I just write minimal amount (esp comments) and get AI to complete the rest. So if I need to write a non-leaf feature, I will break it down to several leaf features and then pass it on to AI and if needed, manually assemble them. I had to write 500LOC bash script to handle a software deployment. This is not the way to do it but I was forced by circumstances created by someone else. Anyways, if I had to write the whole thing by hand, it'd take multiple days as bash syntax is not forgiving and the stuff I needed to do in the script were quite complex/crazy (and stupid). I think I wrote about 50+ lines of text describing the whole process, which you can think of as a requirement document. With a few tries, I was able to get the whole script with near accuracy. My reading revealed some issues. Pointed them to AI. It fixed them. Tests revealed some other issues. Again, AI fixed them after pointing out. I was able to get the whole thing done in just an hour or so.

douglasisshiny 1y ago

I've had the same experience as the person to whom you're responding. After reading your post, I have to ask: if you're putting so much effort into prompting it with specific points, correcting it often, etc., why not just write the code yourself? It sounds like you're putting a good deal of effort into prompting it. Aren't you worried that overtime you'll rely on it too much and your offhand knowledge will get worse?

CopyOnWrite 1y ago
I have read somewhere, that LLMs are mostly helpful to junior developers. Is it possible the person claiming success with all these languages/tools/technologies is just on a junior level and is subjectively correct but has no point of reference how fast coding is for seniors and how quality code looks like?
- CrimsonRain 1y ago
  1. You have read somewhere...but you don't have any experience. LLMs are really bad for Junior developers. A. They have no skills/exp to judge AI output B. They don't learn from sudden magical wall of code output C. They don't get to explore; thus don't learn. D. Ultimately, LLMs act as a bad drug to them that keeps them dependent and stagnant. E. LLMs are really good for higher end of senior devs. This also means, they don't need as many Juniors anymore and they don't mentor juniors much. This is the biggest loss for Juniors. 2. I think you are absolutely right. I became a Staff Engineer with junior level LLM coding. More power to me I guess :(
xandrius 1y ago
Not OP, it be comes natural and doesn't take a lot of time. Anyway, if you want to, LLMs can today help with a ton of programming languages and frameworks. If you use any of the top 5 languages and it still doesn't work for you, either you're doing some esoteric work or you're doing it wrong.
- CopyOnWrite 1y ago
  Could you point me to a youtube video or a blog post which demonstrates how LLMs help writing code which outperforms a proficient human? My only conditions: - It must be demonstrated by adding a feature on a bigger code base (>= 20 LOC) - The added feature cannot be a leaf feature (means it must integrate with the rest of the system at multiple points) - The prompting has to be less effort/faster than to type the solution in the programming language You can chose any programming language/framework that you want. I don't care if it is Java, JavaScript, Typescript, C, Python, ... hell, I am fine with any language with or w/o a framework.

thi2 1y ago

> You are just bad with prompting or working with very obscure language/framework or bad coding pattern or all of it You just described every existing legacy project^^