4 ms·

> found it lackluster for the domain I asked it for Fair, that is possible depending on your domain. > It's no good at writing algorithms or data structures

by yunohn 3y ago

> found it lackluster for the domain I asked it for Fair, that is possible depending on your domain. > It's no good at writing algorithms or data structures In my experience, this is untrue. I’ve gotten it to write algorithms with various constraints I had. You can even tell it to use specific function signatures instead of any stdlib, and make changes to tweak behavior. > And most use I've seen on it realistically a good LSP covers. Again, I really don’t understand this comparison. LSPs and LLMs go hand in hand. I think it’s more of a workflow clash. One really needs to change how they operate to effectively use LLMs for programming. If you’re just typing nonstop, maybe it would feel like Copilot is just an LSP. But, if you try harder, LLMs are game changers when: - maybe you like rubber ducking - need to learn a new concept and implement it - or need to glue things together - or for new projects or features - or filling in boilerplate based on existing context.

jpc0 3y ago
https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e7f5000 https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e... https://godbolt.org/z/s9Yvnjz7K https://godbolt.org/z/s9Yvnjz7K I mean I could write the algorithm by hand pretty quickly in C++ and would follow the exact same thought pattern but also deal with the edge cases. And factoring in the loss of productivity from the context switch that is a net negative. This algorithm is also not generic over enough cases but that is just up to the prompt. If I can't trust it to write `strip_whitespace` correctly which is like 5 lines of code, can I trust it to do more without a thorough review of the code and writing a ton of unit tests... Well I was going to do that anyway. The argument that I just need to learn better prompt engineering to make the LLM do what I want just doesn't sit with me when instead I could just spend the time writing the code. As I said your last point is absolutely the place I can see LLMs being actually useful but then I need to spend a significant amount of time in code review for generated code from an "employee" who is known to make up interfaces or entire libraries that doesn't exist.
- mrtranscendence 3y ago
  I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4: > Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases. It gave me this: https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd93b350 https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd... I'm not sure what other edge cases there might be, however. This only covers one of them. In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.
  - jpc0 3y ago
    My guess is that this was generated using GPT4? Free GPT I get https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e799ca49 https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e7... which has exactly the same problem as my previous example and doesn't consider the string of all whitespace. Sure GPT4 is better at that, it wasn't the argument made. The example you gave absolutely was the code I would write on a first draft since it does cover the edge cases (assuming we aren't dealing with the full UTF charset and all that could be considered a space there). However this is code that is trivial to write in any language and the "Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming? " argument doesn't hold up. Am I to believe it will implement something more complex correctly. This is also code that would absolutely be in hundreds of codebases so GPT has tons of context for it.
    - yunohn 3y ago
      Yeah honestly, I think you have a completely different expectation and style of usage than what is optimal with LLMs. I don’t have the energy to convince you further, but maybe one day it’ll click for you? No worries either way.
      - jpc0 3y ago

by yunohn 3y ago

jpc0 3y ago

https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e7f5000 https://chat.openai.com/share/c8c19f42-240f-44e7-baf4-50ee5e... https://godbolt.org/z/s9Yvnjz7K https://godbolt.org/z/s9Yvnjz7K I mean I could write the algorithm by hand pretty quickly in C++ and would follow the exact same thought pattern but also deal with the edge cases. And factoring in the loss of productivity from the context switch that is a net negative. This algorithm is also not generic over enough cases but that is just up to the prompt. If I can't trust it to write `strip_whitespace` correctly which is like 5 lines of code, can I trust it to do more without a thorough review of the code and writing a ton of unit tests... Well I was going to do that anyway. The argument that I just need to learn better prompt engineering to make the LLM do what I want just doesn't sit with me when instead I could just spend the time writing the code. As I said your last point is absolutely the place I can see LLMs being actually useful but then I need to spend a significant amount of time in code review for generated code from an "employee" who is known to make up interfaces or entire libraries that doesn't exist.

mrtranscendence 3y ago
I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4: > Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases. It gave me this: https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd93b350 https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd... I'm not sure what other edge cases there might be, however. This only covers one of them. In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.
- jpc0 3y ago
  My guess is that this was generated using GPT4? Free GPT I get https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e799ca49 https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e7... which has exactly the same problem as my previous example and doesn't consider the string of all whitespace. Sure GPT4 is better at that, it wasn't the argument made. The example you gave absolutely was the code I would write on a first draft since it does cover the edge cases (assuming we aren't dealing with the full UTF charset and all that could be considered a space there). However this is code that is trivial to write in any language and the "Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming? " argument doesn't hold up. Am I to believe it will implement something more complex correctly. This is also code that would absolutely be in hundreds of codebases so GPT has tons of context for it.
  - yunohn 3y ago
    Yeah honestly, I think you have a completely different expectation and style of usage than what is optimal with LLMs. I don’t have the energy to convince you further, but maybe one day it’ll click for you? No worries either way.
    - jpc0 3y ago