4 ms·
If we force content_contains_answer to false, we always re-run the evaluation step first to get a new justification for why the content doesn't have enough inf
by jasonwcfan 4y ago
If we force content_contains_answer to false, we always re-run the evaluation step first to get a new justification for why the content doesn't have enough information. Then we run that justification through another prompt to convert it to a user-facing response.
It happens pretty rarely, off the top of my head probably 5% of the time. However we do see a pretty wide range in binary token probabilities anywhere from low 50% to something that asymptotically approaches 100%.