3 ms·
If you swap an LLM, at the very least, you have to run your entire Eval set again. This will almost certainly lead to a prompt rebuild, to better accommodate n
by intended 3y ago
If you swap an LLM, at the very least, you have to run your entire Eval set again.
This will almost certainly lead to a prompt rebuild, to better accommodate new model idiosyncrasies.
If you are unlucky- your use case may be one where Evals require human review.
Unless you are YOLOing it without evals. In that case this is relevant.
- matusp 3y agoYou need to rerun eval after each LLM update. GPTs have a new version every couple of months and their capabilities can change quite drastically pretty much randomly. Maybe they will make it more robust in time, but I think this is the feature of the technology and people will have to adapt to these quirks
- intended 3y agoYup. Constant Eval.