ruby314

3 ms·

1. 1.▲
   by ruby314 2y ago

   Also wanted to address the confusion on the role of llama-3-8b and llama-3.1-8b. In the blog post we use these models as an example of an evaluator LLM. We select what is best for your custom eval under the hood. LSR is just one example of
2. 2.▲
   by ruby314 2y ago

   Re #3 - my bad, mixing terminology in my answer above. It’s the “base model” for the evaluator model (vs a fine tuned evaluator model). Just using the labeled Halubench dataset as the outputs to be evaluated, so no base model for the Haluev
3. 3.▲
   by ruby314 2y ago

   and yeah, we fully acknowledge that our list of references is not comprehensive :) thanks again for sharing!
4. 4.▲
   by ruby314 2y ago

   "An advantage of being aware of early papers is that it accumulates citations so you can often find good works in reverse citations." - absolutely. Thanks again for sharing interesting references. Cool that GRATH uses contrast pai
5. 5.▲
   by ruby314 2y ago

   Ty! - We generate contrast pairs (for this post, using gpt-4o) and do some post processing for quality (synthetic data). The impact of different types of contrast pairs is a continuing area of research for us. - We treat the evaluator model
6. 6.▲
   by ruby314 2y ago

   Thanks for sharing the reference. It's definitely related! But the approach in that paper is different in a couple of very significant ways: 1. They train a classifier on top of the activations of the base LLM. so you would have to hav
7. 7.▲
   by ruby314 2y ago

   yes that's correct! we project an evaluator LLM's internal activations onto meaningful linear directions, derived from contrasting examples. the strongest connection is to LLM interpretability (existence of meaningful linear direc