4 ms·
The small model you trained, how did you annotate the dataset? Because most of the times output from big models will be subjective, and with not a drastic diffe
by ankit219 2y ago
The small model you trained, how did you annotate the dataset? Because most of the times output from big models will be subjective, and with not a drastic difference in quality. If there is a drastic difference, you dont need a model/classifier. For smaller differences in quality + cost, what would the annotated dataset look like?
- yoeven 2y agoWe run tons of evals with public datasets from LLM Arena on common categories such as law, financing task, code, maths, etc., and pair them up with the public benchmarks such as Natural2Code, GPQA, etc then tagging the benchmarks with relevant cost, speed benchmarks on a provider level. One of the weights of the Prompt Engine model is on the similarity of the outputs as we execute a user prompt on 5-6 different prompts. Higher quality outputs tend to be similar across at least 3 of the models.