2 ms·
Show HN: An LLM purpose built for data annotation, outperforms GPT-3.5
Try it out here: https://labs.refuel.ai/playground
Refuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets.
It is a Llama-v2-13b base model, trained on over 2500 unique datasets (5.24B tokens) spanning categories such as classification, entity resolution, matching, reading comprehension and information extraction.
- spdustin 3y agoIt seems to suffer from choice bias, favoring choices that come earlier. If I add “car components” to the end of the list, the last example reliably returns “car”. If I move “car” to the end of the list, the model reliably returns “car components”