5 ms·

Can someone ELI5 the significance of this? (okay maybe not 5, but in basic language)

93po 2y ago
from chatgpt itself: The article discusses how researchers use sparse autoencoders to identify and interpret key features within complex language models like GPT-4, making their inner workings more understandable. This advancement helps improve AI safety and reliability by breaking down the models' decision-making processes into simpler, human-interpretable parts.
OtherShrezzing 2y ago
LLM based AIs have lots of "features" which are kind of synonymous with "concepts" - these can be anything from `the concept of an apostrophe in the word don't`, to `"George Wash" is usually followed by "ington" in the context of early American History`. Inside of the LLMs neural network, these are mapped to some circuitry-in-software-esque paths. We don't really have a good way of understanding how these features are generated inside of the LLMs or how their circuitry is activated when outputting them, or why the LLMs are following those circuits. Because of this, we don't have any way to debug this component of an LLM - which makes them harder to improve. Similarly, if LLMs/AIs ever get advanced enough, we'll want to be able to identify if they're being wilfully deceptive towards us, which we can't currently do. For these reasons, we'd like to understand what is actually happening in the neural network to produce & output concepts. This domain of research is usually referred to as "interpretability". OpenAI (and also DeepMind and Anthropic) have found a few ways to inspect the inner circuitry of the LLMs, and reveal a handful of these features. They do this by asking questions of the model, and then inspecting which parts of the LLM's inner circuitry "lights up". They then ablate (turn off) circuitry to see if those features become less frequently used in the AIs response as a verification step. The graphs and highlighted words are visual representations of concepts that they are reasonably certain about - for example, the concept of the word "AND" linking two parts of a sentence together highlights the word "AND". Neel Nanda is the best source for this info if you're interested in interpretability (IMO it's the most interesting software problem out there at the moment), but note that his approach is different to OpenAI's methodology discussed in the post: https://www.neelnanda.io/mechanistic-interpretability https://www.neelnanda.io/mechanistic-interpretability
- localfirst 2y ago
  hallucination solution?
  - OtherShrezzing 2y ago
    Solving this problem would be a step on the way to debugging (and then resolving, or at least highlighting) hallucinations.
    - skywhopper 2y ago
      I’m skeptical that it could ever be possible to tell the difference between a hallucination and a “fact” in terms of what’s going on inside the model. Because hallucinations aren’t really a bug in the usual sense. Ie, there’s not some logic wrong or something misfiring. Instead, it’s more appropriate to think of LLMs as always hallucinating. And sometimes that comes really close to reality because there’s a lot of reinforcement in the training data. And sometimes we humans infer meaning that isn’t there because that’s how humans work. And sometimes the leaps show clearly as “hallucinations” because the patterns the model is expressing don’t match the patterns that are meaningful to us. (Eg when they hallucinate strongly patterned things like URLs or academic citations, which don’t actually point to anything real. The model picked up the pattern of what such citations look like really well, but it didn’t and can’t make the leap to linking those patterns to reality.) Not to mention that a lot of use cases for LLMs we actually want “hallucination”. Eg when we ask it to do any creative task or make up stories or jokes or songs or pictures. It’s only a hallucination in the wrong context. But context is the main thing LLMs just don’t have.
      - cwalv 2y ago
        > Instead, it’s more appropriate to think of LLMs as always hallucinating. That matches my mental model as well. To get rid of hallucinations, "I don't know" would have to be an acceptable answer, and it would have to output that when 'appropriate' ... Which, it doesn't know (and to be fair, neither do we most of the time, without some way of checking/validating(.
orbital-decay 2y ago
High-level concepts stored inside the large models (diffusion models, transformers etc) are normally hard to separate from each other, and the model is more or less a black box. A lot of research is put into obtaining the insight into what model knows. This is another advancement in this direction; it allows for easy separation of the concepts. This can be used to analyze the knowledge inside the model, and potentially modify (add, erase, change the importance) certain concepts without affecting unrelated ones. The precision achievable with the particular technique is always in question though, and some concepts are just too close to separate from each other, so it's probably not perfect.
HarHarVeryFunny 2y ago
In general this is just copying work done by Anthropic, so there's nothing fundamentally new here. What they have done here is to identify patterns internal to GPT-4 that correspond to specific identifiable concepts. The work was done my OpenAI's mostly dismantled safety team (it has the names of this teams recently departed co-leads Ilya & Jan Leike on it), so this is nominally being done for safety reasons to be able to boost or suppress specific concepts from being activated when the model is running, such as Anthropic's demonstration of boosting their models fixation on the Golden Gate bridge: https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude This kind of work would also seem to have potential functional uses as well as safety ones, given that it allow you to control the model in specific ways.

93po 2y ago

from chatgpt itself: The article discusses how researchers use sparse autoencoders to identify and interpret key features within complex language models like GPT-4, making their inner workings more understandable. This advancement helps improve AI safety and reliability by breaking down the models' decision-making processes into simpler, human-interpretable parts.

OtherShrezzing 2y ago

LLM based AIs have lots of "features" which are kind of synonymous with "concepts" - these can be anything from `the concept of an apostrophe in the word don't`, to `"George Wash" is usually followed by "ington" in the context of early American History`. Inside of the LLMs neural network, these are mapped to some circuitry-in-software-esque paths. We don't really have a good way of understanding how these features are generated inside of the LLMs or how their circuitry is activated when outputting them, or why the LLMs are following those circuits. Because of this, we don't have any way to debug this component of an LLM - which makes them harder to improve. Similarly, if LLMs/AIs ever get advanced enough, we'll want to be able to identify if they're being wilfully deceptive towards us, which we can't currently do. For these reasons, we'd like to understand what is actually happening in the neural network to produce & output concepts. This domain of research is usually referred to as "interpretability". OpenAI (and also DeepMind and Anthropic) have found a few ways to inspect the inner circuitry of the LLMs, and reveal a handful of these features. They do this by asking questions of the model, and then inspecting which parts of the LLM's inner circuitry "lights up". They then ablate (turn off) circuitry to see if those features become less frequently used in the AIs response as a verification step. The graphs and highlighted words are visual representations of concepts that they are reasonably certain about - for example, the concept of the word "AND" linking two parts of a sentence together highlights the word "AND". Neel Nanda is the best source for this info if you're interested in interpretability (IMO it's the most interesting software problem out there at the moment), but note that his approach is different to OpenAI's methodology discussed in the post: https://www.neelnanda.io/mechanistic-interpretability https://www.neelnanda.io/mechanistic-interpretability

localfirst 2y ago
hallucination solution?
- OtherShrezzing 2y ago
  Solving this problem would be a step on the way to debugging (and then resolving, or at least highlighting) hallucinations.
  - skywhopper 2y ago
    I’m skeptical that it could ever be possible to tell the difference between a hallucination and a “fact” in terms of what’s going on inside the model. Because hallucinations aren’t really a bug in the usual sense. Ie, there’s not some logic wrong or something misfiring. Instead, it’s more appropriate to think of LLMs as always hallucinating. And sometimes that comes really close to reality because there’s a lot of reinforcement in the training data. And sometimes we humans infer meaning that isn’t there because that’s how humans work. And sometimes the leaps show clearly as “hallucinations” because the patterns the model is expressing don’t match the patterns that are meaningful to us. (Eg when they hallucinate strongly patterned things like URLs or academic citations, which don’t actually point to anything real. The model picked up the pattern of what such citations look like really well, but it didn’t and can’t make the leap to linking those patterns to reality.) Not to mention that a lot of use cases for LLMs we actually want “hallucination”. Eg when we ask it to do any creative task or make up stories or jokes or songs or pictures. It’s only a hallucination in the wrong context. But context is the main thing LLMs just don’t have.
    - cwalv 2y ago
      > Instead, it’s more appropriate to think of LLMs as always hallucinating. That matches my mental model as well. To get rid of hallucinations, "I don't know" would have to be an acceptable answer, and it would have to output that when 'appropriate' ... Which, it doesn't know (and to be fair, neither do we most of the time, without some way of checking/validating(.

orbital-decay 2y ago

High-level concepts stored inside the large models (diffusion models, transformers etc) are normally hard to separate from each other, and the model is more or less a black box. A lot of research is put into obtaining the insight into what model knows. This is another advancement in this direction; it allows for easy separation of the concepts. This can be used to analyze the knowledge inside the model, and potentially modify (add, erase, change the importance) certain concepts without affecting unrelated ones. The precision achievable with the particular technique is always in question though, and some concepts are just too close to separate from each other, so it's probably not perfect.

HarHarVeryFunny 2y ago

In general this is just copying work done by Anthropic, so there's nothing fundamentally new here. What they have done here is to identify patterns internal to GPT-4 that correspond to specific identifiable concepts. The work was done my OpenAI's mostly dismantled safety team (it has the names of this teams recently departed co-leads Ilya & Jan Leike on it), so this is nominally being done for safety reasons to be able to boost or suppress specific concepts from being activated when the model is running, such as Anthropic's demonstration of boosting their models fixation on the Golden Gate bridge: https://www.anthropic.com/news/golden-gate-claude https://www.anthropic.com/news/golden-gate-claude This kind of work would also seem to have potential functional uses as well as safety ones, given that it allow you to control the model in specific ways.