> The sister thread https://news.ycombinator.com/item?id=34973654 https://news.ycombinator.com/item?id=34973654 points out that "completely separate models" are

> The sister thread https://news.ycombinator.com/item?id=34973654 https://news.ycombinator.com/item?id=34973654 points out that "completely separate models" are exactly what OpenAI is now selling for $250k/yr. The sister thread isn't describing a Microsoft model, it's describing an OpenAI model. > Actually, you can. That’s the purpose of RLHF. You reward the model for behaving the way you want. And in that context, it’s a matter of rewarding it for paying attention to [system]. You're overthinking how specific alignment is. ChatGPT went through alignment to train it to stay on topic during conversations. There's a difference between general alignment and the kind of hyper-specific training you're thinking of. But you're also kind of missing the scope of prompt injection attacks. Even if Microsoft did train the model to pay attention to specific prompt words, it doesn't mean that the model wouldn't be vulnerable to other prompt injections, because prompt injections are not a deliberate vulnerability that OpenAI added. They're an emergent property of the model. Look, the fact that prompt injections do work today with Bing chat that don't use [system][0] should cause you to think that maybe there's something more complicated going on here than just bad parsing rules. If I can't convince of that, then... I can't convince of that, it's fine; in terms of disagreements I've had on HN, this one is pretty low-consequence, it's a purely technical disagreement. But I'm going to throw out a prediction that Microsoft is not going to be able to easily guard against this attack. Check back in over time and see if that prediction holds true if you want to. Otherwise, similarly, I hope you have a great week. And honestly, I hope you're right, because if you're not right then it's going to be a significant challenge to wire any LLM that works with 3rd-party data to real-world systems. [0]: Read through the paper, there are examples they list that don't use [system], instead they emulate Basic code or a terminal prompt. Things that Microsoft almost certainly didn't train the model to pay attention to.