That is easy to solve. Just use a model capable of function/tool calling, implement a tool which terminates the chat, then add instructions to the system prompt

That is easy to solve. Just use a model capable of function/tool calling, implement a tool which terminates the chat, then add instructions to the system prompt telling the model what tool to use if it wants to end the conversation. If the model appears too hesitant or eager to use the tool, do some finetuning on conversations where the model should or should not use it.