3 ms·

I must be living in a different universe, every single LLM I've tried are shit at the details/nuances of writing code.

swah 2y ago
80% of folks are doing web stuff, as soon as you deviate from that the quality must go from 9/10 to 3/10 or whatever Also have you tried shorter blocks ? "Write a function like this" instead of "write the app"
sarreph 2y ago
Claude has no trouble with SQL queries, writing tests, utility functions, component sketches (React). But as soon as I start giving it reasonably complex stuff like interdependent generics (TypeScript) or intricate custom logic implementation on top of well-known frameworks, it quite often goes around in circles or outputs code with incorrect syntax. You know, the stuff that I would actually feel as though I was receiving intellectual help with. Current LLMs are _great_ at doing chores. They are not so great at producing cohesive, professional-grade code.
steve_adams_86 2y ago
I tried getting Claude to help me model an event-based store with moderate type safety and it was a total disaster. It was randomly trying to convince me to turn it into a state machine using a library in Python (I was working in TypeScript). This is so far from awesome as far as productivity goes. I do find cursor suggestions useful here and there, but it’s typically filling in the simpler details of an implementation I had to design without its help. Ultimately it doesn’t save nearly as much time as I hoped it might a couple years ago. Still, cool development in terms of dev tools and I’ll keep experimenting.
- GiorgioG 2y ago
  LLMs...60 percent of the time it works every time.
  - deleted 2y ago
    [deleted]
- kamaal 2y ago
  >>I tried getting Claude to help me model an event-based store with moderate type safety and it was a total disaster. That's just not how you work with a LLM. You don't give LLMs problems to solve. You do the thinking part and solve problems, and ask the LLM to implement small blocks of code, like the smallest possible, and you incrementally go from there. This really is some what like asking the whole question in the Google search bar, not seeing any result and saying Google search doesn't work. >>it’s typically filling in the simpler details of an implementation I had to design without its help. Most complex solutions are just reusing and connecting simpler solutions.
  - GiorgioG 2y ago
    > ...and ask the LLM to implement small blocks of code, like the smallest possible, and you incrementally go from there. So it's autocomplete++? Talk about an insane level of hype for an incremental improvement to what we already have...if that's actually what we can hope to expect from these LLMs.
    - kamaal 2y ago
      At their heart LLMs are basically a mechanism to guess(predict based on probability) the next word based on what it has already done/seen so far. How it goes about guessing them, or how it gets the context is based on attention and multihead attention functions. You could say that these functions provide which part(word) of a question/sentence must get a higher weight. That basically provides context based on which it predicts what needs to come next. How it predicts is basically your plain old neural network. Just like how in linear regression you know, if your model is predicting a straight line, you know in which area the next points are likely to appear. Similar mechanisms are used here to guess what the next words are likely to be. It is a extreme auto complete feature in the context of code for sure. Note, LLMs are not sentient. That means they can't be held responsible for making decisions. Even more so code decisions. Now you can argue its nothing special. But its some what like arguing eclipse/intellij are not special when compared to vim/emacs. That is just splitting hairs. IDE's definitely do a lot of productive work compared to plain text editors. The initial demo's on LLMs confuse new users a lot. The demos go on the lines of giving a sentence like 'Implement a todo list app' or something like that and LLM writes some code implementing it. That's a wrong way to work with LLMs. Don't outsource your thinking or give it whole blanket problem statement to solve it. Think about LLMs like tools that do a lot of quick text writing for you you given the smallest, non ambiguous, atomically implementable/rollback statement possible. It gets a while to get used to this, but once you are, you are more productive.
joegibbs 2y ago
Which language are you using, and what kind of code are you writing? So far I've had good results for Javascript/Typescript and SQL for web apps and Unreal C++ for game development. I can imagine it would do way worse though if you were writing something with a lot of domain-specific knowledge, really cutting-edge stuff or in a more obscure language like Fortran - but then again, I think that most developers would also struggle more in those conditions. Part of what I find useful about LLMs is that it's almost like rubber duck debugging, except that the duck talks back. You have to explain to the LLM exactly what it is that you want it to do to get the right result, then it gives you a solution. Mostly the solution is right, but when it's wrong you'll better understand the problem through the act of writing it out.
- GiorgioG 2y ago
  > You have to explain to the LLM exactly what it is that you want it to do to get the right result, then it gives you a solution. Mostly the solution is right, but when it's wrong you'll better understand the problem through the act of writing it out. The solution is either correct or incorrect. In general its level of incorrectness is not trivial. My language of choice is C#. Anytime I ask it to generate a series of classes (say a data model of a particular domain, using specific well-known patterns), it gets it wrong, quite wrong and it would take more effort to explain it to unfuck itself, by which time it would have forgotten 3 deliberate instructions from 5 interactions ago. It’s a brilliant moron.
  - kamaal 2y ago
    >>The solution is either correct or incorrect. Not even humans write 100% correct solutions. Everyday programming is not a theorem proving exercise. Programs are correct enough, to cover a agreeable amount of test cases. >>Anytime I ask it to generate a series of classes Generating a class is too big and broad of question to ask an LLM to generate. My chunks of inputs and outputs to/from an LLM are often on the lines of a single for loop, if block or a small chunk of code that can't be written in more than one way. There are several steps from here to a complete class. LLM is your classic socratic thinking tool. You really to have to learn to ask and work in chunks of questions that are smallest, and involve only one change from the previous question, and are atomic enough to do a easy roll back. And you need to build from here. As experience with the series of books The Little Schemer shows, this unfortunately is not everyone's cup of tea. A lot of people struggle to keep a train of thought in their minds and work in that workflow. To that extent I see LLM are not for everyone. Just like how Google search was great and helped a lot of people, but the majority just couldn't bring themselves to sit in front of a screen and work for hours stitching solutions from various places on internet into a working solution.

swah 2y ago

80% of folks are doing web stuff, as soon as you deviate from that the quality must go from 9/10 to 3/10 or whatever Also have you tried shorter blocks ? "Write a function like this" instead of "write the app"

sarreph 2y ago

Claude has no trouble with SQL queries, writing tests, utility functions, component sketches (React). But as soon as I start giving it reasonably complex stuff like interdependent generics (TypeScript) or intricate custom logic implementation on top of well-known frameworks, it quite often goes around in circles or outputs code with incorrect syntax. You know, the stuff that I would actually feel as though I was receiving intellectual help with. Current LLMs are _great_ at doing chores. They are not so great at producing cohesive, professional-grade code.

steve_adams_86 2y ago

I tried getting Claude to help me model an event-based store with moderate type safety and it was a total disaster. It was randomly trying to convince me to turn it into a state machine using a library in Python (I was working in TypeScript). This is so far from awesome as far as productivity goes. I do find cursor suggestions useful here and there, but it’s typically filling in the simpler details of an implementation I had to design without its help. Ultimately it doesn’t save nearly as much time as I hoped it might a couple years ago. Still, cool development in terms of dev tools and I’ll keep experimenting.

GiorgioG 2y ago
LLMs...60 percent of the time it works every time.
- deleted 2y ago
  [deleted]
kamaal 2y ago
>>I tried getting Claude to help me model an event-based store with moderate type safety and it was a total disaster. That's just not how you work with a LLM. You don't give LLMs problems to solve. You do the thinking part and solve problems, and ask the LLM to implement small blocks of code, like the smallest possible, and you incrementally go from there. This really is some what like asking the whole question in the Google search bar, not seeing any result and saying Google search doesn't work. >>it’s typically filling in the simpler details of an implementation I had to design without its help. Most complex solutions are just reusing and connecting simpler solutions.
- GiorgioG 2y ago
  > ...and ask the LLM to implement small blocks of code, like the smallest possible, and you incrementally go from there. So it's autocomplete++? Talk about an insane level of hype for an incremental improvement to what we already have...if that's actually what we can hope to expect from these LLMs.
  - kamaal 2y ago
    At their heart LLMs are basically a mechanism to guess(predict based on probability) the next word based on what it has already done/seen so far. How it goes about guessing them, or how it gets the context is based on attention and multihead attention functions. You could say that these functions provide which part(word) of a question/sentence must get a higher weight. That basically provides context based on which it predicts what needs to come next. How it predicts is basically your plain old neural network. Just like how in linear regression you know, if your model is predicting a straight line, you know in which area the next points are likely to appear. Similar mechanisms are used here to guess what the next words are likely to be. It is a extreme auto complete feature in the context of code for sure. Note, LLMs are not sentient. That means they can't be held responsible for making decisions. Even more so code decisions. Now you can argue its nothing special. But its some what like arguing eclipse/intellij are not special when compared to vim/emacs. That is just splitting hairs. IDE's definitely do a lot of productive work compared to plain text editors. The initial demo's on LLMs confuse new users a lot. The demos go on the lines of giving a sentence like 'Implement a todo list app' or something like that and LLM writes some code implementing it. That's a wrong way to work with LLMs. Don't outsource your thinking or give it whole blanket problem statement to solve it. Think about LLMs like tools that do a lot of quick text writing for you you given the smallest, non ambiguous, atomically implementable/rollback statement possible. It gets a while to get used to this, but once you are, you are more productive.

joegibbs 2y ago

Which language are you using, and what kind of code are you writing? So far I've had good results for Javascript/Typescript and SQL for web apps and Unreal C++ for game development. I can imagine it would do way worse though if you were writing something with a lot of domain-specific knowledge, really cutting-edge stuff or in a more obscure language like Fortran - but then again, I think that most developers would also struggle more in those conditions. Part of what I find useful about LLMs is that it's almost like rubber duck debugging, except that the duck talks back. You have to explain to the LLM exactly what it is that you want it to do to get the right result, then it gives you a solution. Mostly the solution is right, but when it's wrong you'll better understand the problem through the act of writing it out.

GiorgioG 2y ago
> You have to explain to the LLM exactly what it is that you want it to do to get the right result, then it gives you a solution. Mostly the solution is right, but when it's wrong you'll better understand the problem through the act of writing it out. The solution is either correct or incorrect. In general its level of incorrectness is not trivial. My language of choice is C#. Anytime I ask it to generate a series of classes (say a data model of a particular domain, using specific well-known patterns), it gets it wrong, quite wrong and it would take more effort to explain it to unfuck itself, by which time it would have forgotten 3 deliberate instructions from 5 interactions ago. It’s a brilliant moron.
- kamaal 2y ago
  >>The solution is either correct or incorrect. Not even humans write 100% correct solutions. Everyday programming is not a theorem proving exercise. Programs are correct enough, to cover a agreeable amount of test cases. >>Anytime I ask it to generate a series of classes Generating a class is too big and broad of question to ask an LLM to generate. My chunks of inputs and outputs to/from an LLM are often on the lines of a single for loop, if block or a small chunk of code that can't be written in more than one way. There are several steps from here to a complete class. LLM is your classic socratic thinking tool. You really to have to learn to ask and work in chunks of questions that are smallest, and involve only one change from the previous question, and are atomic enough to do a easy roll back. And you need to build from here. As experience with the series of books The Little Schemer shows, this unfortunately is not everyone's cup of tea. A lot of people struggle to keep a train of thought in their minds and work in that workflow. To that extent I see LLM are not for everyone. Just like how Google search was great and helped a lot of people, but the majority just couldn't bring themselves to sit in front of a screen and work for hours stitching solutions from various places on internet into a working solution.