38 ms·

Language models can explain neurons in language models

688 points by mfiguiere 4y ago

TyrianPurple 4y ago
By analyzing the function of individual neurons, these guys were/are able to gain a deeper understanding of how language models process language, which could lead to improved model architecture and training methods.
ftxbro 4y ago
> "This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself." I feel like this isn't a Yud-approved approach to AI alignment.
- shadowgovt 4y ago
  "Yud-approved?"
  - circuit10 4y ago
    https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky
  - ftxbro 4y ago
    He's the one in the fedora who is losing patience that otherwise smart sounding people are seriously considering letting AI police itself https://www.youtube.com/watch?v=41SUp-TRVlg https://www.youtube.com/watch?v=41SUp-TRVlg
    - jack_riminton 4y ago
      That's not a fedora, that's his King of the Redditors crown
  - aitanabewa 4y ago
    Meaning is approved by Eliezer Yudkowsky. https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://twitter.com/ESYudkowsky https://twitter.com/ESYudkowsky https://www.youtube.com/watch?v=AaTRHFaaPG8 https://www.youtube.com/watch?v=AaTRHFaaPG8 (Lex Fridman Interview)
- FeepingCreature 4y ago
  Honestly, I think any foundational work on the topic is inherently Yud-favored, compared to the blithe optimism and surface-level analysis at best that is usually applied to the topic. Ie, I think it's not that this shouldn't be done. This should certainly be done. It's just that so many more things than it should be done before we move forward.
- Teever 4y ago
  Why does this matter?
- snapcaster 4y ago
  Agreed, Yud does seem to have been right about the course things will take but I'm not confident he actually has any solutions to the problem to offer
  - qumpis 4y ago
    Which things has he been right about and when, if you recall?
    - snapcaster 4y ago
      At least to me, this current generation of AI progress is looking a lot more "foomy" than a lot of people besides him predicted
  - causalmodels 4y ago
    His solution is a global regulatory regime to ban new large training runs. The tools required to accomplish this are, IMO, out of the question but I will give Yud credit for being honest about them while others who share his viewpoint try to hide the ball.
    - snapcaster 4y ago
      Good point, but I think if he was serious he would be looking to Serena Butler for inspiration not nuclear non-proliferation treaties
- bick_nyers 4y ago
  These were my thoughts exactly. On one hand, this can enable alignment research to catch up faster. On the other hand, if we are worried about homicidal AI, then putting it in charge of policing itself (and training it to find exploits in a way) is probably not ideal.
- ShamelessC 4y ago
  You mean Yudkowski? I saw him on Lex Fridman and he was entirely unconvincing. Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety?
  - ftxbro 4y ago
    > Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety? I'm not sure Yudkowski is an EA, but the EAs want him in their polycule.
    - tomjakubowski 4y ago
      He posts on the forum. I'm not sure what more evidence is needed that he's part of it. https://forum.effectivealtruism.org/users/eliezeryudkowsky https://forum.effectivealtruism.org/users/eliezeryudkowsky
      - ftxbro 4y ago
        I guess it's true, not just a rationalist but also effective altruist!
  - lubesGordi 4y ago
    I heard him on Lex too, and it seemed to be just a given that AI is going to be deceptive and want to kill us all. I don't think there was a single example of how that could be accomplished given. I'm open to hearing thoughts on this, maybe I'm not creative enough to see the 'obvious' ways this could happen.
    - ethanbond 4y ago
      This is also why I go into chess matches against 1400 elo players. I cannot conceive of the specific ways in which they will beat me (a 600 elo player), so I have good reason to suspect that I can win. I'm willing to bet the future of our species on my consistent victory in these types of matches, in fact.
- killthebuddha 4y ago
  DISCLAIMER: I think Yudkowsky is a serious thinker and his ideas should be taken seriously, regardless of whether not they are correct. Your comment triggered a random thought: A perfect name for Yudkowsky et al and the AGI doomers is... wait for it... the Yuddites :)
  - throwaway2137 4y ago
    Already used on 4chan :)
srajabi 4y ago
"This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations." On first look this is genius but it seems pretty tautological in a way. How do we know if the explainer is good?... Kinda leads to thinking about who watches the watchers...
- wongarsu 4y ago
  It also lags one iteration behind. Which is a problem because a misaligned model might lie to you, spoiling all future research with this method
  - regularfry 4y ago
    It doesn't have to lag, though. You could ask gpt-2 to explain gpt-2. The weights are just input data. The reason this wasn't done on gpt-3 or gpt-4 is just because a) they're much bigger, and b) they're deeper, so the roles of individual neurons are more attenuated.
- m1el 4y ago
  You're correct to have a suspicion here. Hypothetically the explainer could omit a neuron or give a wrong explanation for the role of a neuron. Imagine you're trying to understand a neural network, and you spend enormous amount of time generating hypotheses and validating them. Well the explainer might give you 90% correct hypotheses, it means you have 10 times less work to produce hypotheses. So if you have a solid way of testing an explanation, even if the explainer is evil, it's still useful.
- vhold 4y ago
  It produces examples that can be evaluated. https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/21/neurons/2932 https://openaipublic.blob.core.windows.net/neuron-explainer/...
cschmid 4y ago
Has anyone here found a link to the actual paper? If I click on 'paper', I only see what seems to be an awkward HTML version.
- simonw 4y ago
  You mean this? https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html https://openaipublic.blob.core.windows.net/neuron-explainer/... Would you prefer a PDF? (I'm always fascinated to hear from people who would rather read a PDF than a web-native paper like this one, especially given that web papers are actually readable on mobile devices. Do you do all of your reading on a laptop?)
  - bad_alloc 4y ago
    Nope, reading the printed paper on... paper. :)
  - kkylin 4y ago
    I personally prefer reading PDF on an iPad so I can mark it up.
  - probably_wrong 4y ago
    The equations look terrible on Firefox for Android, as they are really small - a two-line fraction is barely taller than a single line, forcing me to constantly zoom in and out. So yes, I would prefer a PDF and have a guarantee that it will look the same no matter where I read it.
  - hexomancer 4y ago
    > Would you prefer a PDF? Yes, I was just reading the paper and some of the javascript glitched and deleted all the contents of the document except the last section, making me lose all context and focus. Doesn't really happen with PDF files.
  - whimsicalism 4y ago
    If you want to draw on it, PDF is usually the best
fnovd 4y ago
LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own. I wonder how many new words we will come up with to describe concepts (or "node-activating clusters of meaning") that the AI finds salient that we don't yet have a singular word for. Or, for that matter, how many of those concepts we will find meaningful at all. What will this teach us about ourselves?
- mrcode007 4y ago
  If the Gödel incompleteness theorem applies here, then the explanations are likely … incomplete or self-referential.
  - PartiallyTyped 4y ago
    As long as lazy evaluation exists, self-reference is fine, no? Hofstadter talks about something similar in his books.
  - fnovd 4y ago
    So is the word "word" but that seems to have worked out OK so far. I can explain the meaning of "meaning" and that seems to work OK too. Being self-referential sounds a lot more like a feature than a bug. Given that the neurons in our own heads are connected to each other and not any ground truth, I think LLMs should do just fine.
  - AlexCoventry 4y ago
    The Goedel Incompleteness Theorem has no straightforward application to this question.
    - galaxyLogic 4y ago
      It would if the language model did reasoning according rules of logic. But they don't. They use Markov chains. To me it makes no sense to say that a LLM could explain its own reasoning if it does no (logical) reasoning at all. It might be able to explain how the neural network calculates its results. But there are no logical reasoning steps in there that could be explained, are there?
jablongo 4y ago
This isnt exactly building an understanding of LLMs from first principles... IMO we should broadly be following the (imperfect) example set forth by neuroscientists attempting to explain fMRI scans and assigning functionality to various subregions in the brain. It is circular and "unsafe" from an alignment perspective to use a complex model to understand the internals of a simpler model; in order to understand GPT4 then we need GPT5? These approaches are interesting, but we should primarily focus on building our understanding of these models from building blocks that we already understand.
- roddylindsay 4y ago
  I don't follow. Neuroscience imaging tools like fMRI are only used because it is impossible to measure the activations of each neuron in a brain in real time (unlike an artificial neural network). This research paper's attempt to understand the role of individual neurons or neuron clusters within a complete network gets much closer to "first principles" than fMRI.
  - jablongo 4y ago
    Right so it should be much easier w/ access to every neuron and activation. But the general approach is an experimental one where you try to use your existing knowledge about physics and biology to discern what is activating different structures (and neurons) in the brain. I agree w/ the approach of trying to assign some functionality to individual 'neurons', but I don't think that using GPT4 to do so is the most appealing way to go about that, considering GPT4 is the structure we are interested in decoding in the first place.
    - PeterisP 4y ago
      All of this seems to lead to something like this paper https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... On the other hand, I find it plausible that it's fundamentally impossible to assign some functionality to individual 'neurons' due to the following argument: 1. Let's assume that for a system calculating a specific function, there is a NN configuration (weights) so that at some fully connected NN layer there is a well-defined functionality for specific individual neurons - #1 represents A, #2 represents B, #3 represents C etc. 2. The exact same system outcome can be represented with infinitely many other weight combinations which effectively result in a linear transformation (i.e. every possible linear transformation) of the data vector at this layer, e.g. where #1 represents 0.1A + 0.3B + 0.6C, #2 represents 0.5B+0.5C, and #3 represents 0.4B+0.6C - in which case the functionality A (or B, or C) is not represented by any individual neurons; 3. When the system is trained, it's simply not likely that we just happen to get the best-case configuration where the theoretically separable functionality is actually separated among individual 'neurons'. Biological minds do get this separation because each connection has a metabolic cost; but the way we train our models (both older perceptron-like layers, and modern transfomer/attention ones) do allow linking everything to everything, so the natural outcome is that functionality simply does not get cleanly split out in individual 'neurons' and each 'neuron' tends to represent some mix of multiple functionalities.
bilsbie 4y ago
I’m so interested in this. Any ideas how I can get involved with only a 2014 laptop?
- Garrrrrr 4y ago
  https://www.w3schools.com/ai/default.asp https://www.w3schools.com/ai/default.asp
  - drBonkers 4y ago
    Wow— this page is superb.
- VictorLevoso 4y ago
  https://www.neelnanda.io/mechanistic-interpretability/getting-started https://www.neelnanda.io/mechanistic-interpretability/gettin...
rounakdatta 4y ago
I'm split between what's more impressive: - The software powering the research paper - The research itself (holy moly! They're showing the neurons!)
xthetrfd 4y ago
This blog post is not very informative. How did they prompt GPT4 to explain the neuron's behavior?
- samgriesemer 4y ago
  It's explained more in the "read paper" link, where they provide the actual prompts: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-algorithm-explain https://openaipublic.blob.core.windows.net/neuron-explainer/...
jerpint 4y ago
I’m most surprised by the approach they take of passing GPT tuples of (token, importance) and having the model reliably figure out the patterns. Nothing would suggest this should work in practice, yet it just… does. In more or less zero shot. With a completely different underlying model. That’s fascinating.
- bilsbie 4y ago
  They’re not looking at activations?
  - jerpint 4y ago
    No just at normalized log probabilities I think
jacooper 4y ago
For people overwhelmed by all the AI science speak, just spend a few minutes with bing or phind and it will explain everything surprisingly well. Imagine telling someone in the middle of 2020, that in three years a computer will be able to speak, reason and explain everything as if it was a human, absolutely incredible!
- teaearlgraycold 4y ago
  I agree it’s crazy good. But timeline-wise, GPT-3 was in beta and used by many companies in 2020.
- mr_toad 4y ago
  In 2020 we were headed for an AI winter, according to all the hot takes: https://www.bbc.com/news/technology-51064369 https://www.bbc.com/news/technology-51064369 https://link.springer.com/article/10.1007/s13347-020-00396-6 https://link.springer.com/article/10.1007/s13347-020-00396-6 https://blog.re-work.co/ai-experts-discuss-the-possibility-of-another-ai-winter/ https://blog.re-work.co/ai-experts-discuss-the-possibility-o...
whimsicalism 4y ago
I think this is a generous usage of "can." As the article admits, these explanations are 'imperfect' and I think that is definitely true.
- sebzim4500 4y ago
  It depends how you parse it. It is clearly true that they 'can' explain neurons, in the sense that at least some of the neurons are quite well explained. On the other hand, it's also the case that the vast majority of neurons are not well explained at all by this method (or likely any method). I̵t̵'̵s̵ ̵o̵n̵l̵y̵ ̵b̵e̵c̵a̵u̵s̵e̵ ̵o̵f̵ ̵a̵ ̵q̵u̵i̵r̵k̵ ̵o̵f̵ ̵A̵d̵a̵m̵W̵ ̵t̵h̵a̵t̵ ̵t̵h̵i̵s̵ ̵i̵s̵ ̵p̵o̵s̵s̵i̵b̵l̵e̵ ̵a̵t̵ ̵a̵l̵l̵,̵ ̵i̵f̵ ̵G̵P̵T̵-̵2̵ ̵w̵a̵s̵ ̵t̵r̵a̵i̵n̵e̵d̵ ̵w̵i̵t̵h̵ ̵S̵G̵D̵ ̵a̵l̵m̵o̵s̵t̵ ̵n̵o̵ ̵n̵e̵u̵r̵o̵n̵s̵ ̵w̵o̵u̵l̵d̵ ̵b̵e̵ ̵i̵n̵t̵e̵r̵p̵r̵e̵t̵a̵b̵l̵e̵.̵ EDIT: This last part isn't true. I think they are only looking at the intermediate layer of the FFN which does have a privileged basis.
  - whimsicalism 4y ago
    > EDIT: This last part isn't true. I think they are only looking at the intermediate layer of the FFN which does have a privileged basis. it does?
    - sebzim4500 4y ago
      Yeah, that's where they apply the activation function and that happens per neuron so you can't rotate everything and expect the same result.
ftxbro 4y ago
I wonder will someone please check the neurons associated to the petertodd and other anomalous glitch tokens (https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petertodd-phenomenon https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...)? I can see the github and I see that for any given neuron you can see associated tokens but I don't see how to do an inverse search.
- ShamelessC 4y ago
  Those were discovered by finding strings that OpenAI’s tokenizer didn’t properly split up. Because of this, they are treated as singular tokens, and since these don’t occur frequently in the training data, you get what are effectively random outputs when using them. The author definitely tries to up the mysticism knob to 11 though, and the post itself is so long, you can hardly finish it before seeing this obvious critique made in the comments.
  - throwuwu 4y ago
    The ironic thing about lesswrong is that it’s quite the opposite in some fantastically oblivious ways.
    - ShamelessC 4y ago
      Yeah, it’s quite strange indeed. Clearly people with decent educations but zero background in applied research/peer review. More concerned with the sound of their own voice than with whether or not their findings are actually useful (or even true). Perhaps they are all on stimulants!
  - ftxbro 4y ago
    Thank you for your opinion on the post that I linked! I'm still curious about the associated neurons though.
    - ShamelessC 4y ago
      Fair enough. You would need to use an open model or work at OpenAI. I assume this work could be used on the llama models - although I’m not aware of anyone has found these glitchy phrases for those models yet.
kobe_bryant 4y ago
its interesting that layer 0 is a bunch of different things like upper case letters and symbols and types of verbs. it would be great to see all the things theyve found for different layers
SpaceManNabs 4y ago
Automating safeguards and interpretability decisions seems circular and likely to detach to policy.
bilsbie 4y ago
Can anyone explain what they did? I’m not understanding from the webpage or the paper. What role does gpt4 play? I’m seeing they had gpt4 label every neuron but how?
- redconfetti 4y ago
  I got the impression that it mentioned that the complexity of what's going on in GPT is so complex that we should use GPT to explain/summarize/graph what is going on. We should ask AI, how are you doing this?
  - redconfetti 4y ago
    Operator: Skynet, are you doing good thing? Skynet: Yes.
Imnimo 4y ago
To me the value here is not that GPT4 has some special insight into explaining the behavior of GPT2 neurons (they say it's comparable to "human contractors" - but human performance on this task is also quite poor). The value is that you can just run this on every neuron if you're willing to spend the compute, and having a very fuzzy, flawed map of every neuron in a model is still pretty useful as a research tool. But I would be very cautious about drawing conclusions from any individual neuron explanation generated in this way - even if it looks plausible by visual inspection of a few attention maps.
- mcguire 4y ago
  They also mention they got a score above 0.8 for 1000 neurons out of GPT2 (which has 1.5B (?)).
  - oofsa 4y ago
    I thought they had only applied the technique to 307,200 neurons. 1,000 / 307,200 = 0.33% is still low, but considering that not all neurons would be useful since they are initialized randomly, it's not too bad.
  - sebzim4500 4y ago
    1.5B parameters, only 300k neurons. The number of connections is roughly quadratic with the number of neurons.
int_19h 4y ago
Of note: "... our technique works poorly for larger models, possibly because later layers are harder to explain." And even for GPT-2, which is what they used for the paper: "... the vast majority of our explanations score poorly ..." Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. This may be the first step towards that, but as they also note, the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; I would expect the former in particular to be much more common in larger models, which is perhaps why they're harder to analyze in this manner.
- rvz 4y ago
  > Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. Exactly. Especially: > ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks. It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.
  - jahewson 4y ago
    > It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Like say, in court to detect if someone is lying? Or at an airport to detect drugs?
    - int_19h 4y ago
      You don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.
sebastianconcpt 4y ago
Great. Going meta with an introspective feedback loop. Let's see if that's the last requisite for exponential AGI growth... Singoolaretee here we go..............
- mcguire 4y ago
  There is no introspection here.
  - sebastianconcpt 4y ago
    ...our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations. The distance between "better explanations" and using that as input of prompts that would automate self-improve is very small, yes?
slowhadoken 4y ago
Language models can also tell you they’re not AI.
- 0xdeadbeefbabe 4y ago
  With dubious confidence too!
davesque 4y ago
Based on my skimming the paper, am I correct in understanding that they came up with an elaborate collection of prompts that embed the text generated by GPT-2 as well as a representation of GPT-2's internal state? Then, in effect, they simply asked GPT-4, "What do you think about all this?" If so, they're acting on a gigantic assumption that GPT-4 actually correctly encodes a reasonable model of the body of knowledge that went into the development of LLMs. Help me out. Am I missing something here?
- MacsHeadroom 4y ago
  >If so, they're acting on a gigantic assumption Yes the initial hypothesis that GPT-4 would know was a gigantic assumption. But a falsifiable one which we can easily generate reproducible tests for. The idea that simulated neurons could learn anything useful at all was once a gigantic assumption too.
  - jxmorris12 4y ago
    Neural networks aren’t simulated neurons…
- sebzim4500 4y ago
  After GPT-4 generates the hypothesis for a neuron they test it by comparing GPT-4's expectation for where the neuron should fire against where it actually fires. If you squint it's train/test separation.
- lennxa 4y ago
  Why does it have to understand how the LLMs are built? They have used gpt-4 to just build a classifier for each neuron's activition, and given the nlp abilities of gpt-4, the hope is that it can describe the nature of activation of the neurons.
nico 4y ago
Now grab the list of labels/explanations for each neuron, and train a small LLM only with data for that neuron. Then you get a dictionary/index of LLMs Could this be used to parallelize training? Or create lighter overall language models? The above would be like doing a “map”, how would we do a “reduce”?
andrewprock 4y ago
At some point did we change the name from perceptron to neuron? Neural networks don't have neurons.
thomastjeffery 4y ago
OpenAI need to hear an explanation of the word "explain".
shrimpx 4y ago
Seems like OpenAI is grasping at straws trying to make GPT "go meta". Reminds me of this Sam Altman quote from 2019: "We have made a soft promise to investors that once we build this sort-of generally intelligent system, basically we will ask it to figure out a way to generate an investment return." https://youtu.be/TzcJlKg2Rc0?t=1886 https://youtu.be/TzcJlKg2Rc0?t=1886
- ChatGTP 4y ago
  I have a similar feeling, they’ve potentially built the most amazing but commercially useless thing in history. I don’t mean it’s not useful entirely, but I mean. It’s not useful in that it’s not deterministic enough to be trustworthy, it’s dangerous and really hard to scale therefore it’s more of an academic project than something that will make Altman as famous as Sergey Brin. I personally take people like Hinton seriously too and think people playing with these things need more oversight themselves.
  - d_sem 4y ago
    Time will tell. Anecdotally, I know several professional who find ChatGPT3.5 & 4 to be valuable and willing to pay for access. I certainly save more than $20 per month for my work by using ChatGPT to accelerate my day to day activities.
    - ChatGTP 4y ago
      That’s why I was careful to say it’s not useless.
ccvannorman 4y ago
My take: Regurgitation of trained-on information about LLMs does not come anywhere close to "conscious brain knows it's conscious."
jokoon 4y ago
I wish there was insightful explanations on why AI cannot think, and if there are researchers trying to explore this topic, and if yes what they do.
- sebzim4500 4y ago
  Any discussion on this topic would be nothing but arguments over the definition of the word 'think'.
tschumacher 4y ago
Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.
- ChatGTP 4y ago
  I have to agree. I often think, “maybe I should use ChatGPT for this” then I realise I have very little way to verify what it tells me and as someone working in engineering, If I don’t understand the black box, I just can’t do it. I’m attracted to open source, because I can look at the code understand it.
  - famouswaffles 4y ago
    Open source doesn't mean you can explain the black box any better. and humans are black boxes that don't understand their mental processes either. We're currently better than LLMs at it i suppose but we're still very poor at it. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact.
    - ChatGTP 4y ago
      Humans can be held accountable so it’s not the same. Even if we’re a black box, we share common traits with other humans. We’re trained in similar ways. So we mostly understand what we will and won’t do. I think this constant degradation of humans is really foolish and harmful personally. “We’re just black boxes etc”, we might not know how brains work but we do and can understand each other. On the other hand I’m starting to feel like “AI researchers” are the greatest black box I’ve ever seen, the logic of what they’re trying to create and their hopes for it really baffle me. By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one?
Ameo 4y ago
I built a toy neural network that runs in the browser[1] to model 2D functions with the goal of doing something similar to this research (in a much more limited manner, ofc). Since the input space is so much more limited than language models or similar, it's possible to examine the outputs for each neuron for all possible inputs, and in a continuous manner. In some cases, you can clearly see neurons that specialize to different areas of the function being modeled, like this one: https://i.ameo.link/b0p.png https://i.ameo.link/b0p.png This OpenAI research seems to be feeding lots of varied input text into the models they're examining and keeping track of the activations of different neurons along the way. Another method I remember seeing used in the past involves using an optimizer to generate inputs that maximally activate particular neurons in vision models[2]. I'm sure that's much more difficult or even impossible for transformers which operate on sequences of tokens/embeddings rather than single static input vectors, but maybe there's a way to generate input embeddings and then use some method to convert them back into tokens. [1] https://nn.ameo.dev/ https://nn.ameo.dev/ [2] https://www.tensorflow.org/tutorials/generative/deepdream https://www.tensorflow.org/tutorials/generative/deepdream
- rmorey 4y ago
  this is wonderful, thanks for sharing!
- newhouseb 4y ago
  This tool is really lovely, great work! I'd be curious to see Softmax Linear Units [1] integrated into the possible activation functions since they seem to improve interpretability. PS: I share your curiosity with respect to things like deep dream. My brief summary of this paper is that you can use GPT4 to summarize what's similar about a set of highlighted words in context which is clever but doesn't fundamentally inform much that we didn't already know about how these models work. I wonder if there's some diffusion based approach that could be used to diffuse from noise in the residual stream towards a maximized activation at a particular point. [1] https://transformer-circuits.pub/2022/solu/index.html https://transformer-circuits.pub/2022/solu/index.html
sudoapps 4y ago
This is really interesting. Could this lead to eventually being able to deconstruct these "black-boxes" to remove proprietary data or enforce legal issues?
budoso 4y ago
How long until we copyright the Marvel Neuron?
nl 4y ago
Commenters here seem a little fixated on the fact that the technique scores poorly. This is true and somewhat problematic but exploring the data it looks to me like this could be more a problem with a combination of the scoring function and the limits of language rather than the methodology. For example, look at https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/17/neurons/3218 https://openaipublic.blob.core.windows.net/neuron-explainer/... It's described as "expressions of completion or success" with a score of 0.38. But going through the examples, they are very consistently a sort of colloquial expression of "completion/success" with a touch of surprise and maybe challenge. Examples are like: "Nuff said", "voila!", "Mission accomplished", "Game on!", "End of story", "enough said", "nailed it" etc. If they expressed it as a basket of words instead of a sentence, and could come up with words that express it better I'd score it much higher.
thedudeabides5 4y ago
Matrices, all the way down
alrex021 4y ago
IMO this is akin to how scientists developed thermodynamics to comprehend steam engines. Except now, AI could be on the forefront of forming conjectures and explanatory theories!
mxschll 4y ago
> We are open-sourcing our datasets and visualization tools for GPT-4-written explanations of all 307,200 neurons in GPT-2, as well as code for explanation and scoring using publicly available models on the OpenAI API. We hope the research community will develop new techniques for generating higher-scoring explanations and better tools for exploring GPT-2 using explanations. Aww, that's so nice of them to let the community do the work they can use for free. I might even forget that most of OpenAI is closed source.
- posterboy 4y ago
  Surely they are willing to shell a out a few bucks
vivegi 4y ago
Reminds me of the boo Blink, by Malcolm Gladwell -- We feel like we have to come up with a reason why we came up with an answer (paraphrasing)
call_me_g0d 4y ago
Can't see how using AI to explain AI could ever go wrong
- rexsteroxxy 4y ago
  Skynet anybody?
cnity 4y ago
LLMs are a strange loop.
deleted 4y ago
[deleted]
spallas 4y ago
More in detail: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html https://openaipublic.blob.core.windows.net/neuron-explainer/...
greenhearth 4y ago
Can't wait for West World style sex robots!
ljw1001 4y ago
We don’t really know how harmful this stuff will be, but we’ll keep on making it more powerful (and keep on bragging about it) until we find out.

688 points by mfiguiere 4y ago

By analyzing the function of individual neurons, these guys were/are able to gain a deeper understanding of how language models process language, which could lead to improved model architecture and training methods.

ftxbro 4y ago

> "This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself." I feel like this isn't a Yud-approved approach to AI alignment.

shadowgovt 4y ago
"Yud-approved?"
- circuit10 4y ago
  https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.m.wikipedia.org/wiki/Eliezer_Yudkowsky
- ftxbro 4y ago
  He's the one in the fedora who is losing patience that otherwise smart sounding people are seriously considering letting AI police itself https://www.youtube.com/watch?v=41SUp-TRVlg https://www.youtube.com/watch?v=41SUp-TRVlg
  - jack_riminton 4y ago
    That's not a fedora, that's his King of the Redditors crown
- aitanabewa 4y ago
  Meaning is approved by Eliezer Yudkowsky. https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://twitter.com/ESYudkowsky https://twitter.com/ESYudkowsky https://www.youtube.com/watch?v=AaTRHFaaPG8 https://www.youtube.com/watch?v=AaTRHFaaPG8 (Lex Fridman Interview)
FeepingCreature 4y ago
Honestly, I think any foundational work on the topic is inherently Yud-favored, compared to the blithe optimism and surface-level analysis at best that is usually applied to the topic. Ie, I think it's not that this shouldn't be done. This should certainly be done. It's just that so many more things than it should be done before we move forward.
Teever 4y ago
Why does this matter?
snapcaster 4y ago
Agreed, Yud does seem to have been right about the course things will take but I'm not confident he actually has any solutions to the problem to offer
- qumpis 4y ago
  Which things has he been right about and when, if you recall?
  - snapcaster 4y ago
    At least to me, this current generation of AI progress is looking a lot more "foomy" than a lot of people besides him predicted
- causalmodels 4y ago
  His solution is a global regulatory regime to ban new large training runs. The tools required to accomplish this are, IMO, out of the question but I will give Yud credit for being honest about them while others who share his viewpoint try to hide the ball.
  - snapcaster 4y ago
    Good point, but I think if he was serious he would be looking to Serena Butler for inspiration not nuclear non-proliferation treaties
bick_nyers 4y ago
These were my thoughts exactly. On one hand, this can enable alignment research to catch up faster. On the other hand, if we are worried about homicidal AI, then putting it in charge of policing itself (and training it to find exploits in a way) is probably not ideal.
ShamelessC 4y ago
You mean Yudkowski? I saw him on Lex Fridman and he was entirely unconvincing. Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety?
- ftxbro 4y ago
  > Why is everyone deferring to a bunch of effective altruism advocates when it comes to AI safety? I'm not sure Yudkowski is an EA, but the EAs want him in their polycule.
  - tomjakubowski 4y ago
    He posts on the forum. I'm not sure what more evidence is needed that he's part of it. https://forum.effectivealtruism.org/users/eliezeryudkowsky https://forum.effectivealtruism.org/users/eliezeryudkowsky
    - ftxbro 4y ago
      I guess it's true, not just a rationalist but also effective altruist!
- lubesGordi 4y ago
  I heard him on Lex too, and it seemed to be just a given that AI is going to be deceptive and want to kill us all. I don't think there was a single example of how that could be accomplished given. I'm open to hearing thoughts on this, maybe I'm not creative enough to see the 'obvious' ways this could happen.
  - ethanbond 4y ago
    This is also why I go into chess matches against 1400 elo players. I cannot conceive of the specific ways in which they will beat me (a 600 elo player), so I have good reason to suspect that I can win. I'm willing to bet the future of our species on my consistent victory in these types of matches, in fact.
killthebuddha 4y ago
DISCLAIMER: I think Yudkowsky is a serious thinker and his ideas should be taken seriously, regardless of whether not they are correct. Your comment triggered a random thought: A perfect name for Yudkowsky et al and the AGI doomers is... wait for it... the Yuddites :)
- throwaway2137 4y ago
  Already used on 4chan :)

srajabi 4y ago

"This work is part of the third pillar of our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations." On first look this is genius but it seems pretty tautological in a way. How do we know if the explainer is good?... Kinda leads to thinking about who watches the watchers...

wongarsu 4y ago
It also lags one iteration behind. Which is a problem because a misaligned model might lie to you, spoiling all future research with this method
- regularfry 4y ago
  It doesn't have to lag, though. You could ask gpt-2 to explain gpt-2. The weights are just input data. The reason this wasn't done on gpt-3 or gpt-4 is just because a) they're much bigger, and b) they're deeper, so the roles of individual neurons are more attenuated.
m1el 4y ago
You're correct to have a suspicion here. Hypothetically the explainer could omit a neuron or give a wrong explanation for the role of a neuron. Imagine you're trying to understand a neural network, and you spend enormous amount of time generating hypotheses and validating them. Well the explainer might give you 90% correct hypotheses, it means you have 10 times less work to produce hypotheses. So if you have a solid way of testing an explanation, even if the explainer is evil, it's still useful.
vhold 4y ago
It produces examples that can be evaluated. https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/21/neurons/2932 https://openaipublic.blob.core.windows.net/neuron-explainer/...

cschmid 4y ago

Has anyone here found a link to the actual paper? If I click on 'paper', I only see what seems to be an awkward HTML version.

simonw 4y ago
You mean this? https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html https://openaipublic.blob.core.windows.net/neuron-explainer/... Would you prefer a PDF? (I'm always fascinated to hear from people who would rather read a PDF than a web-native paper like this one, especially given that web papers are actually readable on mobile devices. Do you do all of your reading on a laptop?)
- bad_alloc 4y ago
  Nope, reading the printed paper on... paper. :)
- kkylin 4y ago
  I personally prefer reading PDF on an iPad so I can mark it up.
- probably_wrong 4y ago
  The equations look terrible on Firefox for Android, as they are really small - a two-line fraction is barely taller than a single line, forcing me to constantly zoom in and out. So yes, I would prefer a PDF and have a guarantee that it will look the same no matter where I read it.
- hexomancer 4y ago
  > Would you prefer a PDF? Yes, I was just reading the paper and some of the javascript glitched and deleted all the contents of the document except the last section, making me lose all context and focus. Doesn't really happen with PDF files.
- whimsicalism 4y ago
  If you want to draw on it, PDF is usually the best

fnovd 4y ago

LLMs are quickly going to be able to start explaining their own thought processes better than any human can explain their own. I wonder how many new words we will come up with to describe concepts (or "node-activating clusters of meaning") that the AI finds salient that we don't yet have a singular word for. Or, for that matter, how many of those concepts we will find meaningful at all. What will this teach us about ourselves?

mrcode007 4y ago
If the Gödel incompleteness theorem applies here, then the explanations are likely … incomplete or self-referential.
- PartiallyTyped 4y ago
  As long as lazy evaluation exists, self-reference is fine, no? Hofstadter talks about something similar in his books.
- fnovd 4y ago
  So is the word "word" but that seems to have worked out OK so far. I can explain the meaning of "meaning" and that seems to work OK too. Being self-referential sounds a lot more like a feature than a bug. Given that the neurons in our own heads are connected to each other and not any ground truth, I think LLMs should do just fine.
- AlexCoventry 4y ago
  The Goedel Incompleteness Theorem has no straightforward application to this question.
  - galaxyLogic 4y ago
    It would if the language model did reasoning according rules of logic. But they don't. They use Markov chains. To me it makes no sense to say that a LLM could explain its own reasoning if it does no (logical) reasoning at all. It might be able to explain how the neural network calculates its results. But there are no logical reasoning steps in there that could be explained, are there?

jablongo 4y ago

This isnt exactly building an understanding of LLMs from first principles... IMO we should broadly be following the (imperfect) example set forth by neuroscientists attempting to explain fMRI scans and assigning functionality to various subregions in the brain. It is circular and "unsafe" from an alignment perspective to use a complex model to understand the internals of a simpler model; in order to understand GPT4 then we need GPT5? These approaches are interesting, but we should primarily focus on building our understanding of these models from building blocks that we already understand.

roddylindsay 4y ago
I don't follow. Neuroscience imaging tools like fMRI are only used because it is impossible to measure the activations of each neuron in a brain in real time (unlike an artificial neural network). This research paper's attempt to understand the role of individual neurons or neuron clusters within a complete network gets much closer to "first principles" than fMRI.
- jablongo 4y ago
  Right so it should be much easier w/ access to every neuron and activation. But the general approach is an experimental one where you try to use your existing knowledge about physics and biology to discern what is activating different structures (and neurons) in the brain. I agree w/ the approach of trying to assign some functionality to individual 'neurons', but I don't think that using GPT4 to do so is the most appealing way to go about that, considering GPT4 is the structure we are interested in decoding in the first place.
  - PeterisP 4y ago
    All of this seems to lead to something like this paper https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268 https://journals.plos.org/ploscompbiol/article?id=10.1371/jo... On the other hand, I find it plausible that it's fundamentally impossible to assign some functionality to individual 'neurons' due to the following argument: 1. Let's assume that for a system calculating a specific function, there is a NN configuration (weights) so that at some fully connected NN layer there is a well-defined functionality for specific individual neurons - #1 represents A, #2 represents B, #3 represents C etc. 2. The exact same system outcome can be represented with infinitely many other weight combinations which effectively result in a linear transformation (i.e. every possible linear transformation) of the data vector at this layer, e.g. where #1 represents 0.1A + 0.3B + 0.6C, #2 represents 0.5B+0.5C, and #3 represents 0.4B+0.6C - in which case the functionality A (or B, or C) is not represented by any individual neurons; 3. When the system is trained, it's simply not likely that we just happen to get the best-case configuration where the theoretically separable functionality is actually separated among individual 'neurons'. Biological minds do get this separation because each connection has a metabolic cost; but the way we train our models (both older perceptron-like layers, and modern transfomer/attention ones) do allow linking everything to everything, so the natural outcome is that functionality simply does not get cleanly split out in individual 'neurons' and each 'neuron' tends to represent some mix of multiple functionalities.

bilsbie 4y ago

I’m so interested in this. Any ideas how I can get involved with only a 2014 laptop?

Garrrrrr 4y ago
https://www.w3schools.com/ai/default.asp https://www.w3schools.com/ai/default.asp
- drBonkers 4y ago
  Wow— this page is superb.
VictorLevoso 4y ago
https://www.neelnanda.io/mechanistic-interpretability/getting-started https://www.neelnanda.io/mechanistic-interpretability/gettin...

rounakdatta 4y ago

I'm split between what's more impressive: - The software powering the research paper - The research itself (holy moly! They're showing the neurons!)

xthetrfd 4y ago

This blog post is not very informative. How did they prompt GPT4 to explain the neuron's behavior?

samgriesemer 4y ago
It's explained more in the "read paper" link, where they provide the actual prompts: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-algorithm-explain https://openaipublic.blob.core.windows.net/neuron-explainer/...

jerpint 4y ago

I’m most surprised by the approach they take of passing GPT tuples of (token, importance) and having the model reliably figure out the patterns. Nothing would suggest this should work in practice, yet it just… does. In more or less zero shot. With a completely different underlying model. That’s fascinating.

bilsbie 4y ago
They’re not looking at activations?
- jerpint 4y ago
  No just at normalized log probabilities I think

jacooper 4y ago

For people overwhelmed by all the AI science speak, just spend a few minutes with bing or phind and it will explain everything surprisingly well. Imagine telling someone in the middle of 2020, that in three years a computer will be able to speak, reason and explain everything as if it was a human, absolutely incredible!

teaearlgraycold 4y ago
I agree it’s crazy good. But timeline-wise, GPT-3 was in beta and used by many companies in 2020.
mr_toad 4y ago
In 2020 we were headed for an AI winter, according to all the hot takes: https://www.bbc.com/news/technology-51064369 https://www.bbc.com/news/technology-51064369 https://link.springer.com/article/10.1007/s13347-020-00396-6 https://link.springer.com/article/10.1007/s13347-020-00396-6 https://blog.re-work.co/ai-experts-discuss-the-possibility-of-another-ai-winter/ https://blog.re-work.co/ai-experts-discuss-the-possibility-o...

whimsicalism 4y ago

I think this is a generous usage of "can." As the article admits, these explanations are 'imperfect' and I think that is definitely true.

sebzim4500 4y ago
It depends how you parse it. It is clearly true that they 'can' explain neurons, in the sense that at least some of the neurons are quite well explained. On the other hand, it's also the case that the vast majority of neurons are not well explained at all by this method (or likely any method). I̵t̵'̵s̵ ̵o̵n̵l̵y̵ ̵b̵e̵c̵a̵u̵s̵e̵ ̵o̵f̵ ̵a̵ ̵q̵u̵i̵r̵k̵ ̵o̵f̵ ̵A̵d̵a̵m̵W̵ ̵t̵h̵a̵t̵ ̵t̵h̵i̵s̵ ̵i̵s̵ ̵p̵o̵s̵s̵i̵b̵l̵e̵ ̵a̵t̵ ̵a̵l̵l̵,̵ ̵i̵f̵ ̵G̵P̵T̵-̵2̵ ̵w̵a̵s̵ ̵t̵r̵a̵i̵n̵e̵d̵ ̵w̵i̵t̵h̵ ̵S̵G̵D̵ ̵a̵l̵m̵o̵s̵t̵ ̵n̵o̵ ̵n̵e̵u̵r̵o̵n̵s̵ ̵w̵o̵u̵l̵d̵ ̵b̵e̵ ̵i̵n̵t̵e̵r̵p̵r̵e̵t̵a̵b̵l̵e̵.̵ EDIT: This last part isn't true. I think they are only looking at the intermediate layer of the FFN which does have a privileged basis.
- whimsicalism 4y ago
  > EDIT: This last part isn't true. I think they are only looking at the intermediate layer of the FFN which does have a privileged basis. it does?
  - sebzim4500 4y ago
    Yeah, that's where they apply the activation function and that happens per neuron so you can't rotate everything and expect the same result.

ftxbro 4y ago

I wonder will someone please check the neurons associated to the petertodd and other anomalous glitch tokens (https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petertodd-phenomenon https://www.lesswrong.com/posts/jkY6QdCfAXHJk3kea/the-petert...)? I can see the github and I see that for any given neuron you can see associated tokens but I don't see how to do an inverse search.

ShamelessC 4y ago
Those were discovered by finding strings that OpenAI’s tokenizer didn’t properly split up. Because of this, they are treated as singular tokens, and since these don’t occur frequently in the training data, you get what are effectively random outputs when using them. The author definitely tries to up the mysticism knob to 11 though, and the post itself is so long, you can hardly finish it before seeing this obvious critique made in the comments.
- throwuwu 4y ago
  The ironic thing about lesswrong is that it’s quite the opposite in some fantastically oblivious ways.
  - ShamelessC 4y ago
    Yeah, it’s quite strange indeed. Clearly people with decent educations but zero background in applied research/peer review. More concerned with the sound of their own voice than with whether or not their findings are actually useful (or even true). Perhaps they are all on stimulants!
- ftxbro 4y ago
  Thank you for your opinion on the post that I linked! I'm still curious about the associated neurons though.
  - ShamelessC 4y ago
    Fair enough. You would need to use an open model or work at OpenAI. I assume this work could be used on the llama models - although I’m not aware of anyone has found these glitchy phrases for those models yet.

kobe_bryant 4y ago

its interesting that layer 0 is a bunch of different things like upper case letters and symbols and types of verbs. it would be great to see all the things theyve found for different layers

SpaceManNabs 4y ago

Automating safeguards and interpretability decisions seems circular and likely to detach to policy.

bilsbie 4y ago

Can anyone explain what they did? I’m not understanding from the webpage or the paper. What role does gpt4 play? I’m seeing they had gpt4 label every neuron but how?

redconfetti 4y ago
I got the impression that it mentioned that the complexity of what's going on in GPT is so complex that we should use GPT to explain/summarize/graph what is going on. We should ask AI, how are you doing this?
- redconfetti 4y ago
  Operator: Skynet, are you doing good thing? Skynet: Yes.

Imnimo 4y ago

To me the value here is not that GPT4 has some special insight into explaining the behavior of GPT2 neurons (they say it's comparable to "human contractors" - but human performance on this task is also quite poor). The value is that you can just run this on every neuron if you're willing to spend the compute, and having a very fuzzy, flawed map of every neuron in a model is still pretty useful as a research tool. But I would be very cautious about drawing conclusions from any individual neuron explanation generated in this way - even if it looks plausible by visual inspection of a few attention maps.

mcguire 4y ago
They also mention they got a score above 0.8 for 1000 neurons out of GPT2 (which has 1.5B (?)).
- oofsa 4y ago
  I thought they had only applied the technique to 307,200 neurons. 1,000 / 307,200 = 0.33% is still low, but considering that not all neurons would be useful since they are initialized randomly, it's not too bad.
- sebzim4500 4y ago
  1.5B parameters, only 300k neurons. The number of connections is roughly quadratic with the number of neurons.

int_19h 4y ago

Of note: "... our technique works poorly for larger models, possibly because later layers are harder to explain." And even for GPT-2, which is what they used for the paper: "... the vast majority of our explanations score poorly ..." Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. This may be the first step towards that, but as they also note, the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; I would expect the former in particular to be much more common in larger models, which is perhaps why they're harder to analyze in this manner.

rvz 4y ago
> Which is to say, we still have no clue as to what's going on inside GPT-4 or even GPT-3, which I think is the question many want an answer to. Exactly. Especially: > ...the technique is already very computationally intensive, and the focus on individual neurons as a function of input means that they can't "reverse engineer" larger structures composed of multiple neurons nor a neuron that has multiple roles; This paper just brings us no closer to explainability in black box neural networks and is just another excuse piece by OpenAI to try to please the explainability situation that has been missing for decades in neural networks. It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently.
- jahewson 4y ago
  > It is also the reason why they cannot be trusted in the most serious of applications which such decision making requires lots of transparency rather than a model regurgitating nonsense confidently. Like say, in court to detect if someone is lying? Or at an airport to detect drugs?
  - int_19h 4y ago
    You don't even have to look that far ahead. Apparently, people are already using ChatGPT to compile custom diet plans for themselves, and they expect it to take into account the information they supply regarding their allergies etc. But, yes, those are also good examples of what we shouldn't be doing, but are going to do anyway.

sebastianconcpt 4y ago

Great. Going meta with an introspective feedback loop. Let's see if that's the last requisite for exponential AGI growth... Singoolaretee here we go..............

mcguire 4y ago
There is no introspection here.
- sebastianconcpt 4y ago
  ...our approach to alignment research: we want to automate the alignment research work itself. A promising aspect of this approach is that it scales with the pace of AI development. As future models become increasingly intelligent and helpful as assistants, we will find better explanations. The distance between "better explanations" and using that as input of prompts that would automate self-improve is very small, yes?

slowhadoken 4y ago

Language models can also tell you they’re not AI.

0xdeadbeefbabe 4y ago
With dubious confidence too!

davesque 4y ago

Based on my skimming the paper, am I correct in understanding that they came up with an elaborate collection of prompts that embed the text generated by GPT-2 as well as a representation of GPT-2's internal state? Then, in effect, they simply asked GPT-4, "What do you think about all this?" If so, they're acting on a gigantic assumption that GPT-4 actually correctly encodes a reasonable model of the body of knowledge that went into the development of LLMs. Help me out. Am I missing something here?

MacsHeadroom 4y ago
>If so, they're acting on a gigantic assumption Yes the initial hypothesis that GPT-4 would know was a gigantic assumption. But a falsifiable one which we can easily generate reproducible tests for. The idea that simulated neurons could learn anything useful at all was once a gigantic assumption too.
- jxmorris12 4y ago
  Neural networks aren’t simulated neurons…
sebzim4500 4y ago
After GPT-4 generates the hypothesis for a neuron they test it by comparing GPT-4's expectation for where the neuron should fire against where it actually fires. If you squint it's train/test separation.
lennxa 4y ago
Why does it have to understand how the LLMs are built? They have used gpt-4 to just build a classifier for each neuron's activition, and given the nlp abilities of gpt-4, the hope is that it can describe the nature of activation of the neurons.

nico 4y ago

Now grab the list of labels/explanations for each neuron, and train a small LLM only with data for that neuron. Then you get a dictionary/index of LLMs Could this be used to parallelize training? Or create lighter overall language models? The above would be like doing a “map”, how would we do a “reduce”?

andrewprock 4y ago

At some point did we change the name from perceptron to neuron? Neural networks don't have neurons.

thomastjeffery 4y ago

OpenAI need to hear an explanation of the word "explain".

shrimpx 4y ago

Seems like OpenAI is grasping at straws trying to make GPT "go meta". Reminds me of this Sam Altman quote from 2019: "We have made a soft promise to investors that once we build this sort-of generally intelligent system, basically we will ask it to figure out a way to generate an investment return." https://youtu.be/TzcJlKg2Rc0?t=1886 https://youtu.be/TzcJlKg2Rc0?t=1886

ChatGTP 4y ago
I have a similar feeling, they’ve potentially built the most amazing but commercially useless thing in history. I don’t mean it’s not useful entirely, but I mean. It’s not useful in that it’s not deterministic enough to be trustworthy, it’s dangerous and really hard to scale therefore it’s more of an academic project than something that will make Altman as famous as Sergey Brin. I personally take people like Hinton seriously too and think people playing with these things need more oversight themselves.
- d_sem 4y ago
  Time will tell. Anecdotally, I know several professional who find ChatGPT3.5 & 4 to be valuable and willing to pay for access. I certainly save more than $20 per month for my work by using ChatGPT to accelerate my day to day activities.
  - ChatGTP 4y ago
    That’s why I was careful to say it’s not useless.

ccvannorman 4y ago

My take: Regurgitation of trained-on information about LLMs does not come anywhere close to "conscious brain knows it's conscious."

jokoon 4y ago

I wish there was insightful explanations on why AI cannot think, and if there are researchers trying to explore this topic, and if yes what they do.

sebzim4500 4y ago
Any discussion on this topic would be nothing but arguments over the definition of the word 'think'.

tschumacher 4y ago

Even if we can explain the function of a single neuron what do we gain? If the goal is to reason about safety of computer vision in automated driving as an example, we would need to understand the system as a whole. The whole point of neural networks is to solve nuanced problems we can't clearly define. The fuzziness of the problems those systems solve is fundamentally at odds with the intent to reason about them.

ChatGTP 4y ago
I have to agree. I often think, “maybe I should use ChatGPT for this” then I realise I have very little way to verify what it tells me and as someone working in engineering, If I don’t understand the black box, I just can’t do it. I’m attracted to open source, because I can look at the code understand it.
- famouswaffles 4y ago
  Open source doesn't mean you can explain the black box any better. and humans are black boxes that don't understand their mental processes either. We're currently better than LLMs at it i suppose but we're still very poor at it. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://pure.uva.nl/ws/files/25987577/Split_Brain.pdf https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4204522 We can't recreate previous mental states, we just do a pretty good job (usually) of rationalizing decisions after the fact.
  - ChatGTP 4y ago
    Humans can be held accountable so it’s not the same. Even if we’re a black box, we share common traits with other humans. We’re trained in similar ways. So we mostly understand what we will and won’t do. I think this constant degradation of humans is really foolish and harmful personally. “We’re just black boxes etc”, we might not know how brains work but we do and can understand each other. On the other hand I’m starting to feel like “AI researchers” are the greatest black box I’ve ever seen, the logic of what they’re trying to create and their hopes for it really baffle me. By the way, I have infinitely more hope of understanding an open source black box compared to a closed source one?

Ameo 4y ago

I built a toy neural network that runs in the browser[1] to model 2D functions with the goal of doing something similar to this research (in a much more limited manner, ofc). Since the input space is so much more limited than language models or similar, it's possible to examine the outputs for each neuron for all possible inputs, and in a continuous manner. In some cases, you can clearly see neurons that specialize to different areas of the function being modeled, like this one: https://i.ameo.link/b0p.png https://i.ameo.link/b0p.png This OpenAI research seems to be feeding lots of varied input text into the models they're examining and keeping track of the activations of different neurons along the way. Another method I remember seeing used in the past involves using an optimizer to generate inputs that maximally activate particular neurons in vision models[2]. I'm sure that's much more difficult or even impossible for transformers which operate on sequences of tokens/embeddings rather than single static input vectors, but maybe there's a way to generate input embeddings and then use some method to convert them back into tokens. [1] https://nn.ameo.dev/ https://nn.ameo.dev/ [2] https://www.tensorflow.org/tutorials/generative/deepdream https://www.tensorflow.org/tutorials/generative/deepdream

rmorey 4y ago
this is wonderful, thanks for sharing!
newhouseb 4y ago
This tool is really lovely, great work! I'd be curious to see Softmax Linear Units [1] integrated into the possible activation functions since they seem to improve interpretability. PS: I share your curiosity with respect to things like deep dream. My brief summary of this paper is that you can use GPT4 to summarize what's similar about a set of highlighted words in context which is clever but doesn't fundamentally inform much that we didn't already know about how these models work. I wonder if there's some diffusion based approach that could be used to diffuse from noise in the residual stream towards a maximized activation at a particular point. [1] https://transformer-circuits.pub/2022/solu/index.html https://transformer-circuits.pub/2022/solu/index.html

sudoapps 4y ago

This is really interesting. Could this lead to eventually being able to deconstruct these "black-boxes" to remove proprietary data or enforce legal issues?

budoso 4y ago

How long until we copyright the Marvel Neuron?

nl 4y ago

Commenters here seem a little fixated on the fact that the technique scores poorly. This is true and somewhat problematic but exploring the data it looks to me like this could be more a problem with a combination of the scoring function and the limits of language rather than the methodology. For example, look at https://openaipublic.blob.core.windows.net/neuron-explainer/neuron-viewer/index.html#/layers/17/neurons/3218 https://openaipublic.blob.core.windows.net/neuron-explainer/... It's described as "expressions of completion or success" with a score of 0.38. But going through the examples, they are very consistently a sort of colloquial expression of "completion/success" with a touch of surprise and maybe challenge. Examples are like: "Nuff said", "voila!", "Mission accomplished", "Game on!", "End of story", "enough said", "nailed it" etc. If they expressed it as a basket of words instead of a sentence, and could come up with words that express it better I'd score it much higher.

thedudeabides5 4y ago

Matrices, all the way down

alrex021 4y ago

IMO this is akin to how scientists developed thermodynamics to comprehend steam engines. Except now, AI could be on the forefront of forming conjectures and explanatory theories!

mxschll 4y ago

> We are open-sourcing our datasets and visualization tools for GPT-4-written explanations of all 307,200 neurons in GPT-2, as well as code for explanation and scoring using publicly available models on the OpenAI API. We hope the research community will develop new techniques for generating higher-scoring explanations and better tools for exploring GPT-2 using explanations. Aww, that's so nice of them to let the community do the work they can use for free. I might even forget that most of OpenAI is closed source.

posterboy 4y ago
Surely they are willing to shell a out a few bucks

vivegi 4y ago

Reminds me of the boo Blink, by Malcolm Gladwell -- We feel like we have to come up with a reason why we came up with an answer (paraphrasing)

call_me_g0d 4y ago

Can't see how using AI to explain AI could ever go wrong

rexsteroxxy 4y ago
Skynet anybody?

cnity 4y ago

LLMs are a strange loop.

deleted 4y ago

[deleted]

spallas 4y ago

More in detail: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html https://openaipublic.blob.core.windows.net/neuron-explainer/...

greenhearth 4y ago

Can't wait for West World style sex robots!

ljw1001 4y ago

We don’t really know how harmful this stuff will be, but we’ll keep on making it more powerful (and keep on bragging about it) until we find out.