3 ms·
> "An AI assistant was tasked with {task}. The relevant information for their task was {context}. Their answer is {answer}. The correct answer should be somethi
by maeil 2y ago
> "An AI assistant was tasked with {task}. The relevant information for their task was {context}. Their answer is {answer}. The correct answer should be something like {ground truth}. Is their answer correct?"
If you have a ground truth, what was the purpose of asking the AI assistant for an answer in the first place?
- pletnes 2y agoThe cat is dead. The cat is no longer alive. These are equivalent enough, usually, but fails string comparison.
- ghm2180 2y agoOr like of you did ai voice calling the goal was XYZ did the conversation get, in so many words, to XYZ?
- senko 2y agoWhen you're writing a test, you usually know the correct answer for that specific combination of input parameters.
- maeil 2y agoLooking back at it, I must have been very tired when I wrote that! Or maybe I was thinking about cases where the ground truth is difficult to establish.