4 ms·

LLMs Are Interpretable

17 points by tkellogg 3y ago

gorenb 3y ago
must read for anyone interested in llms
sirwhinesalot 3y ago
If you consider your schizophrenic uncle to be "interpretable", sure.
- tkellogg 3y ago
  An alarming trend I see with LLM skeptics is this general direction, where whole categories of real humans are about to be classified as sub-human because an AI has now surpassed their abilities. Every time an LLM skeptic uses the "it's not actually intelligent" argument, it comes dangerously close to dehumanizing actual humans.
  - freejazz 3y ago
    This is just you end-running the intelligence debate by assuming LLMs outputs are comparable to humans in anyway beyond the facial
  - sirwhinesalot 3y ago
    Racist crazy conspiracy-theorist uncle was the example given in the article :)
lukev 3y ago
I like this point of view, and I'll even go a step further... If we're going to use LLMs as the basis for anything resembling general intelligence, it won't be through one-shot invocation of the model. It'll be through some kind of chain/tree/graph of thought where the model invokes itself recursively. In this scenario, we have an exact transcript of the model's thought process, exactly as it occurred, written for us in english. The model can't even have a private "thought", everything needs to be in the visible context. You can't get more interpretable than that.
imranq 3y ago
I think interpretable is a overloaded term. If you use RAG, sure you can look at the retrieved text and understand what is being used, but at that point you're just interpreting the retriever. LLMs need to be interpreted so that they can be edited have their biases understood in a systematic way. However just as people aren't "interpretable" these algorithms are not going to be able to display their inner workings with 100% confidence. It's going to remain probabilistic, which might be fine for the majority of use cases. I think we're coming from an age where everything was 100% interpretable because we knew what was going on inside the machine (e.g. in a knowledge graph). There needs to be some definition of what we want to achieve with interpretability for us to understand what standards we need to keep.
- tkellogg 3y ago
  yes! absolutely. > I think interpretable is a overloaded term. Author here, this is basically the tl;dr of the paper I kept referencing throughout the post. My take, I hope I was clear, is that understanding the inner workings isn't very helpful, except for ML engineers trying to debug a model. I think I'd break the terms down something like - debuggable: The traditional definition of interpretability - trustable: What I talk about here
zby 3y ago
A great article - just a technical nitpick for the author: "in-context learning" is not what you do with RAG. "In context learning" is a really confusing name for reasoning by analogy. In RAG you provide source information in the prompt - in ICL you provide examples of how the task should be accomplished. : """ In-context learning in language models, also known as few-shot learning or few-shot prompting, is a technique where the model is presented with prompts and responses as a context prior to performing a task. For example, to train a language model to generate imaginative and witty jokes. We can leverage in-context learning by exposing the model to a dataset of joke prompts and corresponding punchlines: Prompt 1: “Why don’t scientists trust atoms?” Response: “Because they make up everything! Prompt 2: “What do you call a bear with no teeth?” Response: “A gummy bear!” Prompt 3: “Why did the scarecrow win an award?” Response: “Because he was outstanding in his field!” By training in different types of jokes, the model develops an understanding of how humor works and becomes capable of creating its own clever and amusing punchlines. """ from https://www.techopedia.com/from-language-models-to-problem-solvers-the-rise-of-in-context-learning-in-ais-problem-solving-journey https://www.techopedia.com/from-language-models-to-problem-s...
- tkellogg 3y ago
  ah! that's true, I do wrongly conflate those a lot.

17 points by tkellogg 3y ago

gorenb 3y ago

must read for anyone interested in llms

sirwhinesalot 3y ago

If you consider your schizophrenic uncle to be "interpretable", sure.

tkellogg 3y ago
An alarming trend I see with LLM skeptics is this general direction, where whole categories of real humans are about to be classified as sub-human because an AI has now surpassed their abilities. Every time an LLM skeptic uses the "it's not actually intelligent" argument, it comes dangerously close to dehumanizing actual humans.
- freejazz 3y ago
  This is just you end-running the intelligence debate by assuming LLMs outputs are comparable to humans in anyway beyond the facial
- sirwhinesalot 3y ago
  Racist crazy conspiracy-theorist uncle was the example given in the article :)

lukev 3y ago

I like this point of view, and I'll even go a step further... If we're going to use LLMs as the basis for anything resembling general intelligence, it won't be through one-shot invocation of the model. It'll be through some kind of chain/tree/graph of thought where the model invokes itself recursively. In this scenario, we have an exact transcript of the model's thought process, exactly as it occurred, written for us in english. The model can't even have a private "thought", everything needs to be in the visible context. You can't get more interpretable than that.

imranq 3y ago

I think interpretable is a overloaded term. If you use RAG, sure you can look at the retrieved text and understand what is being used, but at that point you're just interpreting the retriever. LLMs need to be interpreted so that they can be edited have their biases understood in a systematic way. However just as people aren't "interpretable" these algorithms are not going to be able to display their inner workings with 100% confidence. It's going to remain probabilistic, which might be fine for the majority of use cases. I think we're coming from an age where everything was 100% interpretable because we knew what was going on inside the machine (e.g. in a knowledge graph). There needs to be some definition of what we want to achieve with interpretability for us to understand what standards we need to keep.

tkellogg 3y ago
yes! absolutely. > I think interpretable is a overloaded term. Author here, this is basically the tl;dr of the paper I kept referencing throughout the post. My take, I hope I was clear, is that understanding the inner workings isn't very helpful, except for ML engineers trying to debug a model. I think I'd break the terms down something like - debuggable: The traditional definition of interpretability - trustable: What I talk about here

zby 3y ago

A great article - just a technical nitpick for the author: "in-context learning" is not what you do with RAG. "In context learning" is a really confusing name for reasoning by analogy. In RAG you provide source information in the prompt - in ICL you provide examples of how the task should be accomplished. : """ In-context learning in language models, also known as few-shot learning or few-shot prompting, is a technique where the model is presented with prompts and responses as a context prior to performing a task. For example, to train a language model to generate imaginative and witty jokes. We can leverage in-context learning by exposing the model to a dataset of joke prompts and corresponding punchlines: Prompt 1: “Why don’t scientists trust atoms?” Response: “Because they make up everything! Prompt 2: “What do you call a bear with no teeth?” Response: “A gummy bear!” Prompt 3: “Why did the scarecrow win an award?” Response: “Because he was outstanding in his field!” By training in different types of jokes, the model develops an understanding of how humor works and becomes capable of creating its own clever and amusing punchlines. """ from https://www.techopedia.com/from-language-models-to-problem-solvers-the-rise-of-in-context-learning-in-ais-problem-solving-journey https://www.techopedia.com/from-language-models-to-problem-s...

tkellogg 3y ago
ah! that's true, I do wrongly conflate those a lot.