3 ms·

LLMs know more than what they say

148 points by nqnielsen 2y ago

nqnielsen 2y ago
And how to use LLM interpretability research for applied evaluation
autokad 2y ago
if what I understand is correct, that they project the LLM's internal activations into meaningful linear directions derived from contrasting examples, I guess this is similar to how we began to derive a lot more value from ebeddings by using the embedding values for various things.
- ruby314 2y ago
  yes that's correct! we project an evaluator LLM's internal activations onto meaningful linear directions, derived from contrasting examples. the strongest connection is to LLM interpretability (existence of meaningful linear directions) and steering research (computation from contrast pairs). This has been done with base model activations to understand base model behavior, but we show you can boost evaluation accuracy this way too, with a small number of human feedback
  - autokad 2y ago
    this is pretty interesting work, I am curious to what emerges from this kind of thing in the future. I have been working on something 'similar' that involves the same problem, how do you evaluate a response without a source of truth, and this gets at that. I was looking at it from the stand point of the embeddings of the output with different temperatures
sanxiyn 2y ago
I believe the paper to be cited is "The Internal State of an LLM Knows When It's Lying", published last year: https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734
- ruby314 2y ago
  Thanks for sharing the reference. It's definitely related! But the approach in that paper is different in a couple of very significant ways: 1. They train a classifier on top of the activations of the base LLM. so you would have to have access to the activations of the LLM you are using for the task. We use the activations of an open evaluator LLM (e.g. Llama-3.1-8B-Instruct) 2. Our features are projections - a few numbers that derive their predictive power from the linear directions we project onto. That allows us to fit our classifier with 10s of samples instead of 1000s to fit a feedforward neural network on top of the high dimensional activation space Would be interesting to compare the accuracy of the two approaches on the same benchmark, but our approach is broadly applicable even when you aren't working with an open LLM, and doesn't require a ton of training data
  - sanxiyn 2y ago
    I didn't mean to suggest it as a competition to the method presented (LSR: latent space readout). It is old after all. LSR's use in evaluator LLM and working with small samples (because it works with linear direction) does seem novel and useful to me. An advantage of being aware of early papers is that it accumulates citations so you can often find good works in reverse citations. I had a brief look and the following seems interesting: GRATH: Gradual Self-Truthifying for Large Language Models https://arxiv.org/abs/2401.12292 https://arxiv.org/abs/2401.12292 TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space https://arxiv.org/abs/2402.17811 https://arxiv.org/abs/2402.17811 In contrast, Lynx may be a better model and HaluBench may be a better benchmark, but the paper is too new so it has zero reverse citations on Google Scholar at the moment. Interestingly, Lynx paper does cite Azaria 2023 although in a very cursory way.
    - ruby314 2y ago
      "An advantage of being aware of early papers is that it accumulates citations so you can often find good works in reverse citations." - absolutely. Thanks again for sharing interesting references. Cool that GRATH uses contrast pairs in an iterative process with DPO and TruthX is steering (using the term broadly) with a creating architecture to determine the inference time edits. One thing about Lynx and HaluBench - as we understand it, Halubench is the test set for Lynx's training data. They do have a couple of held out data sources besides the four they train with, but as far as we could tell from their paper they use the same hallucination-inducing function. Be curious to hear your thoughts on that.
tarasglek 2y ago
This looks cool, but I'm confused as to how this is surfaced in your product, llama-8 is not present in your model list. I thought maybe you offer hallucination detection, but I also don't see that. RAG evals also not visible
- coffeephoenix 2y ago
  Thanks for the q. The LSR version of AutoFeedback is on our Pro tier. You can sign up for more info on that here: https://tally.so/r/w2XVQA https://tally.so/r/w2XVQA In the free trial of Log10.io you can use the ICL (self-improving) version of AutoFeedback for hallucination detection and RAG evals. In generalize, you can customize evals for a wide range of use cases using AutoFeedback: https://docs.log10.io/feedback/auto_feedback https://docs.log10.io/feedback/auto_feedback https://github.com/log10-io/log10-cookbook/tree/main https://github.com/log10-io/log10-cookbook/tree/main https://arjunbansal.substack.com/p/hybrid-evaluation-scaling-human-feedback https://arjunbansal.substack.com/p/hybrid-evaluation-scaling... As for models, we support evaluating a wide range of models including Llama - in addition we support OpenAI, Anthropic, Gemini, Mistral, MosaicML, Together, and self-hosted models. We are also compatible with frameworks such as Langchain, Magentic and LiteLLM. For how to integrate different models please see here: https://github.com/log10-io/log10 https://github.com/log10-io/log10
  - ruby314 2y ago
    Also wanted to address the confusion on the role of llama-3-8b and llama-3.1-8b. In the blog post we use these models as an example of an evaluator LLM. We select what is best for your custom eval under the hood. LSR is just one example of the research powering our custom evals
uiDevofNW 2y ago
This is a stupid arguement. I wish author understood an ounce of how LLMs works. Of course, they know more than whay they say. That's because LLMs are nothing but probabistic structures. They mix and match and provide probabilistic approach. Therefore, they are always making a choice between multiple options. I wish there was a global mandatory course before these substacky authors write for fame.

148 points by nqnielsen 2y ago

And how to use LLM interpretability research for applied evaluation

if what I understand is correct, that they project the LLM's internal activations into meaningful linear directions derived from contrasting examples, I guess this is similar to how we began to derive a lot more value from ebeddings by using the embedding values for various things.

ruby314 2y ago
yes that's correct! we project an evaluator LLM's internal activations onto meaningful linear directions, derived from contrasting examples. the strongest connection is to LLM interpretability (existence of meaningful linear directions) and steering research (computation from contrast pairs). This has been done with base model activations to understand base model behavior, but we show you can boost evaluation accuracy this way too, with a small number of human feedback
- autokad 2y ago
  this is pretty interesting work, I am curious to what emerges from this kind of thing in the future. I have been working on something 'similar' that involves the same problem, how do you evaluate a response without a source of truth, and this gets at that. I was looking at it from the stand point of the embeddings of the output with different temperatures

sanxiyn 2y ago

I believe the paper to be cited is "The Internal State of an LLM Knows When It's Lying", published last year: https://arxiv.org/abs/2304.13734 https://arxiv.org/abs/2304.13734

ruby314 2y ago
Thanks for sharing the reference. It's definitely related! But the approach in that paper is different in a couple of very significant ways: 1. They train a classifier on top of the activations of the base LLM. so you would have to have access to the activations of the LLM you are using for the task. We use the activations of an open evaluator LLM (e.g. Llama-3.1-8B-Instruct) 2. Our features are projections - a few numbers that derive their predictive power from the linear directions we project onto. That allows us to fit our classifier with 10s of samples instead of 1000s to fit a feedforward neural network on top of the high dimensional activation space Would be interesting to compare the accuracy of the two approaches on the same benchmark, but our approach is broadly applicable even when you aren't working with an open LLM, and doesn't require a ton of training data
- sanxiyn 2y ago
  I didn't mean to suggest it as a competition to the method presented (LSR: latent space readout). It is old after all. LSR's use in evaluator LLM and working with small samples (because it works with linear direction) does seem novel and useful to me. An advantage of being aware of early papers is that it accumulates citations so you can often find good works in reverse citations. I had a brief look and the following seems interesting: GRATH: Gradual Self-Truthifying for Large Language Models https://arxiv.org/abs/2401.12292 https://arxiv.org/abs/2401.12292 TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Space https://arxiv.org/abs/2402.17811 https://arxiv.org/abs/2402.17811 In contrast, Lynx may be a better model and HaluBench may be a better benchmark, but the paper is too new so it has zero reverse citations on Google Scholar at the moment. Interestingly, Lynx paper does cite Azaria 2023 although in a very cursory way.
  - ruby314 2y ago
    "An advantage of being aware of early papers is that it accumulates citations so you can often find good works in reverse citations." - absolutely. Thanks again for sharing interesting references. Cool that GRATH uses contrast pairs in an iterative process with DPO and TruthX is steering (using the term broadly) with a creating architecture to determine the inference time edits. One thing about Lynx and HaluBench - as we understand it, Halubench is the test set for Lynx's training data. They do have a couple of held out data sources besides the four they train with, but as far as we could tell from their paper they use the same hallucination-inducing function. Be curious to hear your thoughts on that.

tarasglek 2y ago

This looks cool, but I'm confused as to how this is surfaced in your product, llama-8 is not present in your model list. I thought maybe you offer hallucination detection, but I also don't see that. RAG evals also not visible

coffeephoenix 2y ago
Thanks for the q. The LSR version of AutoFeedback is on our Pro tier. You can sign up for more info on that here: https://tally.so/r/w2XVQA https://tally.so/r/w2XVQA In the free trial of Log10.io you can use the ICL (self-improving) version of AutoFeedback for hallucination detection and RAG evals. In generalize, you can customize evals for a wide range of use cases using AutoFeedback: https://docs.log10.io/feedback/auto_feedback https://docs.log10.io/feedback/auto_feedback https://github.com/log10-io/log10-cookbook/tree/main https://github.com/log10-io/log10-cookbook/tree/main https://arjunbansal.substack.com/p/hybrid-evaluation-scaling-human-feedback https://arjunbansal.substack.com/p/hybrid-evaluation-scaling... As for models, we support evaluating a wide range of models including Llama - in addition we support OpenAI, Anthropic, Gemini, Mistral, MosaicML, Together, and self-hosted models. We are also compatible with frameworks such as Langchain, Magentic and LiteLLM. For how to integrate different models please see here: https://github.com/log10-io/log10 https://github.com/log10-io/log10
- ruby314 2y ago
  Also wanted to address the confusion on the role of llama-3-8b and llama-3.1-8b. In the blog post we use these models as an example of an evaluator LLM. We select what is best for your custom eval under the hood. LSR is just one example of the research powering our custom evals

uiDevofNW 2y ago

This is a stupid arguement. I wish author understood an ounce of how LLMs works. Of course, they know more than whay they say. That's because LLMs are nothing but probabistic structures. They mix and match and provide probabilistic approach. Therefore, they are always making a choice between multiple options. I wish there was a global mandatory course before these substacky authors write for fame.