3 ms·

> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.

by yunohn 3y ago

coldtea 3y ago
I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me". Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?
freedomben 3y ago
You need to read more than just the first sentence of a comment. They only said that part so the reader would know that they have never used an LLM for coding, so they would have more context for the question: > So genuine question: is that answer representative of the output quality of these things?
- yunohn 3y ago
  Yes, I did read it. I’m kind of tired of HNers loudly proclaiming they are ignoring LLMs more than a year into this paradigm shift. Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming?
  - jpc0 3y ago
    I did exactly that and found it lackluster for the domain I asked it for. And most use I've seen on it realistically a good LSP covers. Or to put it a other way. It's no good at writing algorithms or data structures ( or at least no better thab I would have with a first drafy but the first draft puts me ahead of the LLM in understanding that actual problem at hand, handing it off to an LLM doesn't help me get to the final solution faster). So that leaves writing boiler plate but concidering my experience with it writing more complex stuff, I would need to read over the boilerplate code to ensure it's correct which in that case I may as well have written it.
    - yunohn 3y ago
      > found it lackluster for the domain I asked it for Fair, that is possible depending on your domain. > It's no good at writing algorithms or data structures In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior. > And most use I've seen on it realistically a good LSP covers. Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand. I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when: - maybe you like rubber ducking - need to learn a new concept and implement it - or need to glue things together - or for new projects or features - or filling in boilerplate based on existing context.
      - jpc0 3y ago
        https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e7f5000 https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e... https://godbolt.org/z/s9Yvnjz7K https://godbolt.org/z/s9Yvnjz7K I mean I could write the algorithm by hand pretty quickly in C++ and would follow the exact same thought pattern but also deal with the edge cases. And factoring in the loss of productivity from the context switch that is a net negative. This algorithm is also not generic over enough cases but that is just up to the prompt. If I can't trust it to write `strip_whitespace` correctly which is like 5 lines of code, can I trust it to do more without a thorough review of the code and writing a ton of unit tests... Well I was going to do that anyway. The argument that I just need to learn better prompt engineering to make the LLM do what I want just doesn't sit with me when instead I could just spend the time writing the code. As I said your last point is absolutely the place I can see LLMs being actually useful but then I need to spend a significant amount of time in code review for generated code from an "employee" who is known to make up interfaces or entire libraries that doesn't exist.
        mrtranscendence 3y ago
        I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4: > Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases. It gave me this: https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd93b350 https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd... I'm not sure what other edge cases there might be, however. This only covers one of them. In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.

> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.

by yunohn 3y ago

> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.

coldtea 3y ago

I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me". Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?

freedomben 3y ago

You need to read more than just the first sentence of a comment. They only said that part so the reader would know that they have never used an LLM for coding, so they would have more context for the question: > So genuine question: is that answer representative of the output quality of these things?

yunohn 3y ago
Yes, I did read it. I’m kind of tired of HNers loudly proclaiming they are ignoring LLMs more than a year into this paradigm shift. Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming?
- jpc0 3y ago
  I did exactly that and found it lackluster for the domain I asked it for. And most use I've seen on it realistically a good LSP covers. Or to put it a other way. It's no good at writing algorithms or data structures ( or at least no better thab I would have with a first drafy but the first draft puts me ahead of the LLM in understanding that actual problem at hand, handing it off to an LLM doesn't help me get to the final solution faster). So that leaves writing boiler plate but concidering my experience with it writing more complex stuff, I would need to read over the boilerplate code to ensure it's correct which in that case I may as well have written it.
  - yunohn 3y ago
    > found it lackluster for the domain I asked it for Fair, that is possible depending on your domain. > It's no good at writing algorithms or data structures In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior. > And most use I've seen on it realistically a good LSP covers. Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand. I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when: - maybe you like rubber ducking - need to learn a new concept and implement it - or need to glue things together - or for new projects or features - or filling in boilerplate based on existing context.
    - jpc0 3y ago
      https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e7f5000 https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e... https://godbolt.org/z/s9Yvnjz7K https://godbolt.org/z/s9Yvnjz7K I mean I could write the algorithm by hand pretty quickly in C++ and would follow the exact same thought pattern but also deal with the edge cases. And factoring in the loss of productivity from the context switch that is a net negative. This algorithm is also not generic over enough cases but that is just up to the prompt. If I can't trust it to write `strip_whitespace` correctly which is like 5 lines of code, can I trust it to do more without a thorough review of the code and writing a ton of unit tests... Well I was going to do that anyway. The argument that I just need to learn better prompt engineering to make the LLM do what I want just doesn't sit with me when instead I could just spend the time writing the code. As I said your last point is absolutely the place I can see LLMs being actually useful but then I need to spend a significant amount of time in code review for generated code from an "employee" who is known to make up interfaces or entire libraries that doesn't exist.
      - mrtranscendence 3y ago
        I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4: > Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases. It gave me this: https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd93b350 https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd... I'm not sure what other edge cases there might be, however. This only covers one of them. In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.