4 ms·
The documentation for system prompts is setting off all of my prompt injection red flags: https://docs.anthropic.com/claude/docs/how-to-use-system-prompts https
by simonw 3y ago
The documentation for system prompts is setting off all of my prompt injection red flags: https://docs.anthropic.com/claude/docs/how-to-use-system-prompts https://docs.anthropic.com/claude/docs/how-to-use-system-pro...
I guess that design is at least honest: OpenAI field the system prompt in a separate fragment of JSON, but it all gets concatenated back together (with some magic delimiter tokens) when it's fed to the underlying model.
- minimaxir 3y agoRelatedly, I checked and OpenAI deleted all references to their ChatML spec from their GitHub repo. This is what it said in an earlier commit: https://github.com/openai/openai-python/blob/2942bf4bb635b1e6722d6decb5e8619950fcddd1/chatml.md https://github.com/openai/openai-python/blob/2942bf4bb635b1e...
- simonw 3y agoSomething I never understood about ChatML: were those "<|im_start|>" things reserved sequences of text that mapped to specific integer tokens, but were not things you could include in your own text that you submitted to their API (or if you did try they would be tokenized differently)?
- minimaxir 3y agoChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken https://github.com/openai/tiktoken#extending-tiktoken In theory they could be added in normal input but it's possible OpenAI has safeguards against it.