5 ms·

I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4: > Write me an alg

I'm a Python-slinging data scientist so C++ isn't my jam (to say the least), but I changed the prompt to the following and asked it to GPT-4: > Write me an algorithm in C++ which finds the begin and end iterator of a sequence where leading and trailing whitespace is stripped. Please write secure code that handles any possible edge cases. It gave me this: https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd93b350 https://chat.openai.com/share/55a4afe2-5db2-4dd1-b516-a3cacd... I'm not sure what other edge cases there might be, however. This only covers one of them. In general, I've found LLMs to be marginally helpful. Like, I can't ever remember how to get matplotlib to give me the plot I want, and 9 times out of 10 GPT-4 easily gives me the code I want. Anything even slightly off the beaten path, though, and it quickly becomes absolutely useless.

jpc0 3y ago
My guess is that this was generated using GPT4? Free GPT I get https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e799ca49 https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e7... which has exactly the same problem as my previous example and doesn't consider the string of all whitespace. Sure GPT4 is better at that, it wasn't the argument made. The example you gave absolutely was the code I would write on a first draft since it does cover the edge cases (assuming we aren't dealing with the full UTF charset and all that could be considered a space there). However this is code that is trivial to write in any language and the "Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming? " argument doesn't hold up. Am I to believe it will implement something more complex correctly. This is also code that would absolutely be in hundreds of codebases so GPT has tons of context for it.
- yunohn 3y ago
  Yeah honestly, I think you have a completely different expectation and style of usage than what is optimal with LLMs. I don’t have the energy to convince you further, but maybe one day it’ll click for you? No worries either way.
  - jpc0 3y ago
    Could you maybe give me an example of what is concidered an optimal use of LLMs. Maybe a prompt to GPT
    - fragmede 3y ago
      Simonw's blog has some examples I'd consider show off its usefulness and limitations, eg https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-... (linked previously above)
      - jpc0 3y ago
        Are you happy with the C code generated there? I'm not sure there isn't a buffer overflow in the vector_decode code he showed there, likewise I don't see any error checks on the code and I am not familiar with the sqlite api to even know whether errors can be propagated upwards and what error conditions would mean in that code. This code is probably fine for a quick side project but doesn't pass my smell test for anything close to production ready code. I definitely would want to see a lot of unit tests around the decode and encode functions with fuzzing and to be honestly that would be the bulk of the work here. That and documentation on this code. Even though he encode function looks correct at first glance. I also don't see an easy way to actually unit test this code as it is without actually running it through sqlite which outs a lot of dependencies on the unit test. I would either need to spend a lot more time massaging gpt to get this to a point where I would be fine shipping the code or you know just write it myself.
    - yunohn 3y ago
      Like sibling commenter mentioned, simonw’s blog is a great resource. Regarding your point around being able to whip up the code yourself - the point is to have a decent starting point to save time and energy. Like you said, you know the edge cases so you could skip the boring parts using GPT and focus purely on fixing those. Though, with more prompting (especially providing examples), GPT can also handle that for you. I have nearly 2 decades of experience as a developer and it took me a while to reorient my flow around LLMs. But now that I have, it’s truly gamechanging. And since you asked, here’s my system prompt: You are an experienced developer who follows industry standards and best practices. Write lean code and explain briefly using bullet points or numbered lists. Elaborate only when explaining concepts or making choices. Always mention which file and where to store provided code. Tech Stack: < insert all the languages, frameworks, etc you’d like to use > If I provide code, highlight and explain problematic code. Also show and explain the corrected code. Take a deep breath and think step by step. Also, always use GPT4 and customize the above to your style and liking.
- mrtranscendence 3y ago
  I think you have the mistaken impression that I was arguing with you (certainly my comment makes it clear that I don't feel that LLMs are a panacea). I merely thought that you might be curious how GPT-4 would respond. > My guess is that this was generated using GPT4? This is a good guess, since I stated outright that I used GPT-4, and then mentioned GPT-4 later on in the comment.
  - jpc0 3y ago
    I was curious and yes I was mistaken.

jpc0 3y ago

My guess is that this was generated using GPT4? Free GPT I get https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e799ca49 https://chat.openai.com/share/f533429d-63ca-4505-8dc8-b8d2e7... which has exactly the same problem as my previous example and doesn't consider the string of all whitespace. Sure GPT4 is better at that, it wasn't the argument made. The example you gave absolutely was the code I would write on a first draft since it does cover the edge cases (assuming we aren't dealing with the full UTF charset and all that could be considered a space there). However this is code that is trivial to write in any language and the "Is it that hard to input a prompt into the free version of ChatGPT and see how it helps with programming? " argument doesn't hold up. Am I to believe it will implement something more complex correctly. This is also code that would absolutely be in hundreds of codebases so GPT has tons of context for it.

yunohn 3y ago
Yeah honestly, I think you have a completely different expectation and style of usage than what is optimal with LLMs. I don’t have the energy to convince you further, but maybe one day it’ll click for you? No worries either way.
- jpc0 3y ago
  Could you maybe give me an example of what is concidered an optimal use of LLMs. Maybe a prompt to GPT
  - fragmede 3y ago
    Simonw's blog has some examples I'd consider show off its usefulness and limitations, eg https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-... (linked previously above)
    - jpc0 3y ago
      Are you happy with the C code generated there? I'm not sure there isn't a buffer overflow in the vector_decode code he showed there, likewise I don't see any error checks on the code and I am not familiar with the sqlite api to even know whether errors can be propagated upwards and what error conditions would mean in that code. This code is probably fine for a quick side project but doesn't pass my smell test for anything close to production ready code. I definitely would want to see a lot of unit tests around the decode and encode functions with fuzzing and to be honestly that would be the bulk of the work here. That and documentation on this code. Even though he encode function looks correct at first glance. I also don't see an easy way to actually unit test this code as it is without actually running it through sqlite which outs a lot of dependencies on the unit test. I would either need to spend a lot more time massaging gpt to get this to a point where I would be fine shipping the code or you know just write it myself.
  - yunohn 3y ago
    Like sibling commenter mentioned, simonw’s blog is a great resource. Regarding your point around being able to whip up the code yourself - the point is to have a decent starting point to save time and energy. Like you said, you know the edge cases so you could skip the boring parts using GPT and focus purely on fixing those. Though, with more prompting (especially providing examples), GPT can also handle that for you. I have nearly 2 decades of experience as a developer and it took me a while to reorient my flow around LLMs. But now that I have, it’s truly gamechanging. And since you asked, here’s my system prompt: You are an experienced developer who follows industry standards and best practices. Write lean code and explain briefly using bullet points or numbered lists. Elaborate only when explaining concepts or making choices. Always mention which file and where to store provided code. Tech Stack: < insert all the languages, frameworks, etc you’d like to use > If I provide code, highlight and explain problematic code. Also show and explain the corrected code. Take a deep breath and think step by step. Also, always use GPT4 and customize the above to your style and liking.
mrtranscendence 3y ago
I think you have the mistaken impression that I was arguing with you (certainly my comment makes it clear that I don't feel that LLMs are a panacea). I merely thought that you might be curious how GPT-4 would respond. > My guess is that this was generated using GPT4? This is a good guess, since I stated outright that I used GPT-4, and then mentioned GPT-4 later on in the comment.
- jpc0 3y ago
  I was curious and yes I was mistaken.