3 ms·

So many highlights from reading this. One that stood out for me is their discovery that refusal works by inhibition: > It turns out that, in Claude, refusal to

So many highlights from reading this. One that stood out for me is their discovery that refusal works by inhibition: > It turns out that, in Claude, refusal to answer is the default behavior: we find a circuit that is "on" by default and that causes the model to state that it has insufficient information to answer any given question. However, when the model is asked about something it knows well—say, the basketball player Michael Jordan—a competing feature representing "known entities" activates and inhibits this default circuit Many cellular processes work similarly ie. there will be a process that runs as fast as it can and one or more companion “inhibitors” doing a kind of “rate limiting”. Given both phenomena are emergent it makes you wonder if do-but-inhibit is a favored technique of the universe we live in, or just coincidence :)

colah3 2y ago
Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with excitation and inhibitory edges -- transcription networks, protein networks, networks of biological neurons -- and one can study recurring motifs that turn up in these networks and try to learn from them. It wouldn't be surprising if some lessons from that work may also transfer to artificial neural networks, although there are some technical things to consider.
- cadamsdotcom 2y ago
  Agreed! So many emergent systems in nature achieve complex outcomes without central coordination - from cellular level to ant colonies & beehives. There are bound to be implications for designed systems. Closely following what you guys are uncovering through interpretability research - not just accepting LLMs as black boxes. Thanks to you & the team for sharing the work with humanity. Interpretability is the most exciting part of AI research for its potential to help us understand what’s in the box. By way of analogy, centuries ago farmers’ best hope for good weather was to pray to the gods! The sooner we escape the “praying to the gods” stage with LLMs the more useful they become.
- scryers_bloom 2y ago
  This all feels familiar to the principle of least action found in physics.
rcxdude 2y ago
It does make a certain amount of sense, though. A specific 'I don't know' feature would need to be effectively the inverse of all of the features the model can recognise, which is going to be quite difficult to represent as anything other than the inverse of 'Some feature was recognised'. (imagine trying to recognise every possible form of nonsense otherwise)
gradascent 2y ago
Then why do I never get an “I don’t know” type response when I use Claude, even when the model clearly has no idea what it’s talking about? I wish it did sometimes.
- hun3 2y ago
  Quoting a paragraph from OP (https://www.anthropic.com/research/tracing-thoughts-language-model#hallucinations https://www.anthropic.com/research/tracing-thoughts-language...): > Sometimes, this sort of “misfire” of the “known answer” circuit happens naturally, without us intervening, resulting in a hallucination. In our paper, we show that such misfires can occur when Claude recognizes a name but doesn't know anything else about that person. In cases like this, the “known entity” feature might still activate, and then suppress the default "don't know" feature—in this case incorrectly. Once the model has decided that it needs to answer the question, it proceeds to confabulate: to generate a plausible—but unfortunately untrue—response.
  - trash_cat 2y ago
    Fun fact, "confabulation", not "hallucinating" is the correct term what LLMs actually do.
    - mystified5016 2y ago
      Fun fact, the "correct" term is the one in use. Dictionaries define language after the fact, they do not prescribe its usage in the future.
      - trash_cat 2y ago
        Confabulation means generating false memories without intent to deceive, which is what LLMs do. They can't hallucinate because they don't perceive. 'Hallucination' caught on, but it's more metaphor than precision.

colah3 2y ago

Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with excitation and inhibitory edges -- transcription networks, protein networks, networks of biological neurons -- and one can study recurring motifs that turn up in these networks and try to learn from them. It wouldn't be surprising if some lessons from that work may also transfer to artificial neural networks, although there are some technical things to consider.

cadamsdotcom 2y ago
Agreed! So many emergent systems in nature achieve complex outcomes without central coordination - from cellular level to ant colonies & beehives. There are bound to be implications for designed systems. Closely following what you guys are uncovering through interpretability research - not just accepting LLMs as black boxes. Thanks to you & the team for sharing the work with humanity. Interpretability is the most exciting part of AI research for its potential to help us understand what’s in the box. By way of analogy, centuries ago farmers’ best hope for good weather was to pray to the gods! The sooner we escape the “praying to the gods” stage with LLMs the more useful they become.
scryers_bloom 2y ago
This all feels familiar to the principle of least action found in physics.

rcxdude 2y ago

It does make a certain amount of sense, though. A specific 'I don't know' feature would need to be effectively the inverse of all of the features the model can recognise, which is going to be quite difficult to represent as anything other than the inverse of 'Some feature was recognised'. (imagine trying to recognise every possible form of nonsense otherwise)

gradascent 2y ago

Then why do I never get an “I don’t know” type response when I use Claude, even when the model clearly has no idea what it’s talking about? I wish it did sometimes.

hun3 2y ago
Quoting a paragraph from OP (https://www.anthropic.com/research/tracing-thoughts-language-model#hallucinations https://www.anthropic.com/research/tracing-thoughts-language...): > Sometimes, this sort of “misfire” of the “known answer” circuit happens naturally, without us intervening, resulting in a hallucination. In our paper, we show that such misfires can occur when Claude recognizes a name but doesn't know anything else about that person. In cases like this, the “known entity” feature might still activate, and then suppress the default "don't know" feature—in this case incorrectly. Once the model has decided that it needs to answer the question, it proceeds to confabulate: to generate a plausible—but unfortunately untrue—response.
- trash_cat 2y ago
  Fun fact, "confabulation", not "hallucinating" is the correct term what LLMs actually do.
  - mystified5016 2y ago
    Fun fact, the "correct" term is the one in use. Dictionaries define language after the fact, they do not prescribe its usage in the future.
    - trash_cat 2y ago
      Confabulation means generating false memories without intent to deceive, which is what LLMs do. They can't hallucinate because they don't perceive. 'Hallucination' caught on, but it's more metaphor than precision.