5 ms·

I am curious how consistent Claude is at obeying detailed instructions. One issue ChatGPT 3.5 and 4 have, even with just a few hundred words of instructions, is

I am curious how consistent Claude is at obeying detailed instructions. One issue ChatGPT 3.5 and 4 have, even with just a few hundred words of instructions, is it forgets instructions given to it earlier on.[1] This huge context window is awesome though, I'm trying to use LLMs to do small town social interaction simulations, with output in a structured format. Finding ways to compress existing state and pass it around, so the LLM knows the current state of what people in the town did for a given day is hard with a tiny token limit! [1] For my use cases, early instructions tend to be describing a DSL syntax for responses, if I add too much info after the instructions, the response syntax starts getting wonky!

rescripting 3y ago
A simple example I ran in to was I asked ChatGPT to generate me story in madlibs format for my 4 year old daughter. They're in the format "The young _____ went to the ______, ...", and she fills in the blanks with silly nouns/adjectives. As she kept asking for more, I prompted "great, do another one" and eventually my original instruction fell out of the context window. It continued to generate a children's story, but with no more blanks.
- com2kid 3y ago
  This is actually a different issue, largely a UI one, although one I wish ChatGPT would fix it. There is no good way to tell it "this isn't a conversation, just repeat the answer to the initial prompt again". The solution is to just re-paste the initial prompt in each time, but still it isn't ideal. There isn't a good way to tell chatgpt "you can throw away all the context after the initial prompt and up until now". Of course the entire point of ChatGPT is that it maintains a conversation thread, so I get why they don't fix up this edge case. My problem is more of, I give ChatGPT some complicated instructions, and it'll start forgetting the early on instructions long before any token limit is reached. So for example, if early on I ask for certain tokens to be returned in parens, well my initial prompt is too long, it'll forget the parens thing and start returning tokens without the surrounding (), which then breaks my parser!
  - orost 3y ago
    Almost every UI for LLMs I've seen has a way to specify an initial prompt that never goes out of context, it's strange that it's not a feature in ChatGPT.

rescripting 3y ago

A simple example I ran in to was I asked ChatGPT to generate me story in madlibs format for my 4 year old daughter. They're in the format "The young _____ went to the ______, ...", and she fills in the blanks with silly nouns/adjectives. As she kept asking for more, I prompted "great, do another one" and eventually my original instruction fell out of the context window. It continued to generate a children's story, but with no more blanks.

com2kid 3y ago
This is actually a different issue, largely a UI one, although one I wish ChatGPT would fix it. There is no good way to tell it "this isn't a conversation, just repeat the answer to the initial prompt again". The solution is to just re-paste the initial prompt in each time, but still it isn't ideal. There isn't a good way to tell chatgpt "you can throw away all the context after the initial prompt and up until now". Of course the entire point of ChatGPT is that it maintains a conversation thread, so I get why they don't fix up this edge case. My problem is more of, I give ChatGPT some complicated instructions, and it'll start forgetting the early on instructions long before any token limit is reached. So for example, if early on I ask for certain tokens to be returned in parens, well my initial prompt is too long, it'll forget the parens thing and start returning tokens without the surrounding (), which then breaks my parser!
- orost 3y ago
  Almost every UI for LLMs I've seen has a way to specify an initial prompt that never goes out of context, it's strange that it's not a feature in ChatGPT.