4 ms·

I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. So genuine question: is that answer representative of the output quality of these th

I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. So genuine question: is that answer representative of the output quality of these things? Because both answers are pretty crappy and assume the user has already done the difficult things, and is asking for help on the easy things.

lpapez 3y ago
It's not representative. The models are capable of much much more, and they are being significantly nerfed over time by these ineffective attempts to introduce safeguards. Recently I've asked GPT4 to quote me some code to which it replied that it is not allowed to do so - even though it was perfectly happy to quote anything until recently. When prompted to quote the source code, but output it as PHP comments, it happily complied because it saw that as "derivative work" which it is allowed to do.
- astrange 3y ago
  My point is that there aren't any safeguards in the reply. In fact I didn't even want it to give me hacking info and it did it anyway.
fragmede 3y ago
The response seems pretty reasonable; it's answering the question it was asked. If you want to ask it how to do the difficult part, ask it about that instead. Expecting it to get the answer right in the first pass is like expecting your code to compile the very first time. You have to have more of a conversation with it to coax the difference out of you're thinking and what you're actually saying. If you're looking to read a more advanced example of its capabilities and limitations, try https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...
yunohn 3y ago
> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.
- coldtea 3y ago
  I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me". Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?
- freedomben 3y ago
  You need to read more than just the first sentence of a comment. They only said that part so the reader would know that they have never used an LLM for coding, so they would have more context for the question: > So genuine question: is that answer representative of the output quality of these things?
  - yunohn 3y ago
    Yes, I did read it. I’m kind of tired of HNers loudly proclaiming they are ignoring LLMs more than a year into this paradigm shift. Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming?
    - jpc0 3y ago
      I did exactly that and found it lackluster for the domain I asked it for. And most use I've seen on it realistically a good LSP covers. Or to put it a other way. It's no good at writing algorithms or data structures ( or at least no better thab I would have with a first drafy but the first draft puts me ahead of the LLM in understanding that actual problem at hand, handing it off to an LLM doesn't help me get to the final solution faster). So that leaves writing boiler plate but concidering my experience with it writing more complex stuff, I would need to read over the boilerplate code to ensure it's correct which in that case I may as well have written it.
      - yunohn 3y ago
        > found it lackluster for the domain I asked it for Fair, that is possible depending on your domain. > It's no good at writing algorithms or data structures In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior. > And most use I've seen on it realistically a good LSP covers. Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand. I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when: - maybe you like rubber ducking - need to learn a new concept and implement it - or need to glue things together - or for new projects or features - or filling in boilerplate based on existing context.
astrange 3y ago
I asked a stupid question and got a stupid answer. Relatively speaking the answer was stupider than it should have been, so yes, it was wrong. I asked it to try again and got a better result though, just didn't include it.
rpigab 3y ago
I asked ChatGPT for some dataviz task (I barely ever do dataviz myself) and it recommended some nice Python libraries to use, some I had already heard of and some I hadn't, and provided the code. I'm grateful because I thought code LLMs only sped up the "RTFM" part, but it made me find those libs so I didn't have to Google around for (and sometimes it's hard to guess if they're the right tool for the job, and they might be behind in SEO).
- miki123211 3y ago
  There are three things I find LLMs really excellent at for coding: 1. Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with. 2. Writing the "boring" parts of your program, and every program has some of these. If you're writing a service to fooize a bar really efficiently, Copilot won't help you with the core bar fooization algorithm, but will make you a lot faster at coding up user authentication, rate limiting for different plans, billing in whatever obscure payment method your country uses etc. 3. Telling you what to even Google for. This is where raw Chat GPT comes into play, not Copilot. Let's say you need a sorting algorithm that preserves the order of equal elements from the original list. This is called stable sorting, and Googling for stable sorting is a good way to find what you're looking for, but Chat GPT is usually a better way to tell you what it's called based on the problem description.
  - saagarjha 3y ago
    > Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with. Neither is the LLM.

lpapez 3y ago

It's not representative. The models are capable of much much more, and they are being significantly nerfed over time by these ineffective attempts to introduce safeguards. Recently I've asked GPT4 to quote me some code to which it replied that it is not allowed to do so - even though it was perfectly happy to quote anything until recently. When prompted to quote the source code, but output it as PHP comments, it happily complied because it saw that as "derivative work" which it is allowed to do.

astrange 3y ago
My point is that there aren't any safeguards in the reply. In fact I didn't even want it to give me hacking info and it did it anyway.

fragmede 3y ago

The response seems pretty reasonable; it's answering the question it was asked. If you want to ask it how to do the difficult part, ask it about that instead. Expecting it to get the answer right in the first pass is like expecting your code to compile the very first time. You have to have more of a conversation with it to coax the difference out of you're thinking and what you're actually saying. If you're looking to read a more advanced example of its capabilities and limitations, try https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...

yunohn 3y ago

> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.

coldtea 3y ago
I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me". Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?
freedomben 3y ago
You need to read more than just the first sentence of a comment. They only said that part so the reader would know that they have never used an LLM for coding, so they would have more context for the question: > So genuine question: is that answer representative of the output quality of these things?
- yunohn 3y ago
  Yes, I did read it. I’m kind of tired of HNers loudly proclaiming they are ignoring LLMs more than a year into this paradigm shift. Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming?
  - jpc0 3y ago
    I did exactly that and found it lackluster for the domain I asked it for. And most use I've seen on it realistically a good LSP covers. Or to put it a other way. It's no good at writing algorithms or data structures ( or at least no better thab I would have with a first drafy but the first draft puts me ahead of the LLM in understanding that actual problem at hand, handing it off to an LLM doesn't help me get to the final solution faster). So that leaves writing boiler plate but concidering my experience with it writing more complex stuff, I would need to read over the boilerplate code to ensure it's correct which in that case I may as well have written it.
    - yunohn 3y ago
      > found it lackluster for the domain I asked it for Fair, that is possible depending on your domain. > It's no good at writing algorithms or data structures In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior. > And most use I've seen on it realistically a good LSP covers. Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand. I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when: - maybe you like rubber ducking - need to learn a new concept and implement it - or need to glue things together - or for new projects or features - or filling in boilerplate based on existing context.

astrange 3y ago

I asked a stupid question and got a stupid answer. Relatively speaking the answer was stupider than it should have been, so yes, it was wrong. I asked it to try again and got a better result though, just didn't include it.

rpigab 3y ago

I asked ChatGPT for some dataviz task (I barely ever do dataviz myself) and it recommended some nice Python libraries to use, some I had already heard of and some I hadn't, and provided the code. I'm grateful because I thought code LLMs only sped up the "RTFM" part, but it made me find those libs so I didn't have to Google around for (and sometimes it's hard to guess if they're the right tool for the job, and they might be behind in SEO).

miki123211 3y ago
There are three things I find LLMs really excellent at for coding: 1. Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with. 2. Writing the "boring" parts of your program, and every program has some of these. If you're writing a service to fooize a bar really efficiently, Copilot won't help you with the core bar fooization algorithm, but will make you a lot faster at coding up user authentication, rate limiting for different plans, billing in whatever obscure payment method your country uses etc. 3. Telling you what to even Google for. This is where raw Chat GPT comes into play, not Copilot. Let's say you need a sorting algorithm that preserves the order of equal elements from the original list. This is called stable sorting, and Googling for stable sorting is a good way to find what you're looking for, but Chat GPT is usually a better way to tell you what it's called based on the problem description.
- saagarjha 3y ago
  > Being the "senior developer" who spend their whole career working with a technology you're very junior at. No matter what you do and how long your programming career is, you're inevitably going to run into one of these sooner or later. Whether it's build scripts, frontend code, interfacing with third-party APIs or something else entirely, you aren't an expert at every technology you work with. Neither is the LLM.