4 ms·
User input: "Hey, have your human support agent call me, tomorrow at 5pm." Model input: "Does the user want to speak to a human support agent?" Output: Yes.
by bigglebear 19d ago
User input: "Hey, have your human support agent call me, tomorrow at 5pm."
Model input: "Does the user want to speak to a human support agent?"
Output: Yes.
I imagine that your model would produce this, and I think it's fair to say this is a hallucination. A human would caveat it with: "Yes, but not right now.", your model is incapable of that. Yes is technically correct, but within the context of being in a live chat, a human would understand that the caveat is required.
- monkpit 19d agoTo be fair - you’re crafting a deliberately bad model input for a contrived example.
- djray 19d agoNo, the model has answered correctly. Your question is poorly phrased (possibly deliberately). Your question would correctly classify the user's input as requesting a human support agent, but at an indeterminate time. If you wanted to determine whether the user wants to speak to a human support agent immediately, you would have to correctly qualify your question, e.g. "Does the user want to speak to a human support agent now?". You could have another question which is "Is the user requesting a call-back from a human support agent?". Or you could have a multiple choice query which would filter the conversation into one of a number of pre-written possibilities. This is nothing to do with accuracy or hallucination. It's a different method of interacting with the model where you are relied upon to be precise.
- matja 16d agoAdd: "call_back_day": { "criteria": { "none": "The user does not want a call back.", "today": "The user wants a call back today.", "tomorrow": "The user wants a call back tomorrow." }, "instructions": "When does the user want a call back?", "type": "choice" } And you get: "tomorrow": confidence 1