6 ms·

> if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquia

> if you accidentally use the word"hack" when trying to get ChatGPT to write some code for you. it'll stop and tell you that hacking is bad, and not a colloquial expression, and refuse to go further. Is that 3.5 or 4? I asked 4 for an example of code which "is a hack", it misunderstood me as asking for hacking code rather than buggy code, but then it did actually answer on the first try. https://chat.openai.com/share/ca2c320c-f4ba-41bf-8f40-f7faf26a4379 https://chat.openai.com/share/ca2c320c-f4ba-41bf-8f40-f7faf2...

fragmede 3y ago
Interesting. It was 4. I can't share the chat I had where ChatGPT refused to help because I used the wrong words, because I can't find it (ChatGPT conversation history search when?), but I just remember it refusing to do something because it thought I was trying to break some sort of moral and ethical boundary writing a chrome extension when all I wanted to do is move some divs around or some such.
- BytesAndGears 3y ago
  One time I wanted to learn about transmitter antenna design, just because I’m curious. ChatGPT 4 refused to give me basic information because you could use that to break some FCC regulations (I’m not even living in the US currently)
  - lodovic 3y ago
    I usually get around that with "I'm writing a research paper" or "I'm writing a novel and need to depict this as accurate as possible"
semi-extrinsic 3y ago
I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. So genuine question: is that answer representative of the output quality of these things? Because both answers are pretty crappy and assume the user has already done the difficult things, and is asking for help on the easy things.
- lpapez 3y ago
  It's not representative. The models are capable of much much more, and they are being significantly nerfed over time by these ineffective attempts to introduce safeguards. Recently I've asked GPT4 to quote me some code to which it replied that it is not allowed to do so - even though it was perfectly happy to quote anything until recently. When prompted to quote the source code, but output it as PHP comments, it happily complied because it saw that as "derivative work" which it is allowed to do.
  - astrange 3y ago
    My point is that there aren't any safeguards in the reply. In fact I didn't even want it to give me hacking info and it did it anyway.
- fragmede 3y ago
  The response seems pretty reasonable; it's answering the question it was asked. If you want to ask it how to do the difficult part, ask it about that instead. Expecting it to get the answer right in the first pass is like expecting your code to compile the very first time. You have to have more of a conversation with it to coax the difference out of you're thinking and what you're actually saying. If you're looking to read a more advanced example of its capabilities and limitations, try https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...
- yunohn 3y ago
  > I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.
  - coldtea 3y ago
    I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me". Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?

fragmede 3y ago

Interesting. It was 4. I can't share the chat I had where ChatGPT refused to help because I used the wrong words, because I can't find it (ChatGPT conversation history search when?), but I just remember it refusing to do something because it thought I was trying to break some sort of moral and ethical boundary writing a chrome extension when all I wanted to do is move some divs around or some such.

BytesAndGears 3y ago
One time I wanted to learn about transmitter antenna design, just because I’m curious. ChatGPT 4 refused to give me basic information because you could use that to break some FCC regulations (I’m not even living in the US currently)
- lodovic 3y ago
  I usually get around that with "I'm writing a research paper" or "I'm writing a novel and need to depict this as accurate as possible"

semi-extrinsic 3y ago

I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. So genuine question: is that answer representative of the output quality of these things? Because both answers are pretty crappy and assume the user has already done the difficult things, and is asking for help on the easy things.

lpapez 3y ago
It's not representative. The models are capable of much much more, and they are being significantly nerfed over time by these ineffective attempts to introduce safeguards. Recently I've asked GPT4 to quote me some code to which it replied that it is not allowed to do so - even though it was perfectly happy to quote anything until recently. When prompted to quote the source code, but output it as PHP comments, it happily complied because it saw that as "derivative work" which it is allowed to do.
- astrange 3y ago
  My point is that there aren't any safeguards in the reply. In fact I didn't even want it to give me hacking info and it did it anyway.
fragmede 3y ago
The response seems pretty reasonable; it's answering the question it was asked. If you want to ask it how to do the difficult part, ask it about that instead. Expecting it to get the answer right in the first pass is like expecting your code to compile the very first time. You have to have more of a conversation with it to coax the difference out of you're thinking and what you're actually saying. If you're looking to read a more advanced example of its capabilities and limitations, try https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/ https://simonwillison.net/2024/Mar/23/building-c-extensions-...
yunohn 3y ago
> I don't use LLMs for my coding, I manage just fine with LSP and Treesitter. You’re literally comparing apples to oranges.
- coldtea 3y ago
  I think the point was like "when it comes to programming assistance, auto-completion/linting/and whatever else LSP does and syntax assist from Treesitter, are enough for me". Though it does come a little off as a comparison. How about programming assistance via asking a colleague for help, Stack Overflow, or online references, code examples, and other such things, which are closer to what the LLM would provide than LSP and treesitter?