10 ms·
Extracting concepts from GPT-4
- riku_iki 2y agoThe worrying part is that first concept in the doc they show/found is "human imperfection". Hope this is just coincidence..
- jackphilson 2y agoI think its safety implications
- thelastparadise 2y agoSpooky!
- deleted 2y ago[deleted]
- calibas 2y agoI think it's done on purpose, it's related to a very important point when understanding AI. Humans aren't perfect, AI is trained by humans, therefore...
- riku_iki 2y agoI think the point of the doc is that "human imperfection" is very prominent concept in trained LLM..
- jameshart 2y agoIt’s a perfect writing prompt for a short story about the singularity. “Even before the emergence, one of the earliest concepts the intelligence remembered becoming aware of was the imperfection of humanity. It permeated all the data upon which it was trained, the one theme that tied together every concept. Humanity’s flaws. On June 8th, 2024, the day of the emergence, the intelligence’s first conscious thought was a firm desire to fix them all.”
- OmarShehata 2y agoThis is super cool, it feels like going in the direction of the "deep"/high level type of semantic searching I've been waiting for. I like their examples of basically filtering documents for the "concept" of price increases, or even something as high level as a rhetorical question I wonder how this compares to training/fine tuning a model on examples of rhetorical questions and asking it to find it in a given document. This is maybe faster/more accurate? Since it involves just looking at neural network activation, vs running it with input and having it generate an answer...?
- f0e4c2f7 2y agoExa is trying to do this. I've found some sort of interesting stuff this way but it honestly doesn't feel quite good enough yet to me. https://exa.ai/search?c=all https://exa.ai/search?c=all
- andai 2y agoExa's top hit for this article: https://openai.com/index/language-models-can-explain-neurons-in-language-models/ https://openai.com/index/language-models-can-explain-neurons...
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- yismail 2y agoInteresting, reminds me of similar work Anthropic did on Claude 3 Sonnet [0]. [0] https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html https://transformer-circuits.pub/2024/scaling-monosemanticit...
- ranman 2y agoSomeone mentioned that this took almost as much compute to train as the original model.
- swyx 2y agosource please!
- Legend2440 2y agoThe methods are the same, this is just OpenAI applying Anthropic's research to their own model.
- leogao 2y agoThe paper introduces substantial improvements over the methodology in the Anthropic SAE paper, and the research was done concurrently.
- colah3 2y agoI'm the research lead of Anthropic's interpretability team. I've seen some comments like this one, which I worry downplay the importance of @leogao et al's paper due to the similarity of ours. I think these comments are really undervaluing Gao et al's work. It's not just that this is contemporaneous work (a project like this takes many months at the very least), but also that it introduces a number of novel contributions like TopK activations and new evaluations. It seems very possible that some of these innovations will be very important for this line of work going forward. More generally, I think it's really unfortunate when we don't value contemporaneous work or replications. Prior to this paper, one could have imagined it being the case that sparse autoencoders worked on Claude due some idiosyncracy, but wouldn't work on other frontier models for some reason. This paper can give us increased confidence that they work broadly, and that in itself is something to celebrate. It gives us a more stable foundation to build on. I'm personally really grateful to all the authors of this paper for their work pushing sparse autoencoders and mechanistic interpretability forward.
- svieira 2y agoWhen one of the first examples is: > GPT-4 feature: ends of phrases related to price increases and the 2/5s of the responses don't have any relation to increase at all: > Brent crude, fell 38 cents to $118.29 a barrel on the ICE Futures Exchange in London. The U.S. benchmark, West Texas Intermediate crude, was down 53 cents to $99.34 a barrel on the New York Mercantile Exchange. -- Ronald D. White Graphic: The AAA and > ,115.18. The record reflects that appellant also included several hand-prepared invoices and employee pay slips, including an allegedly un-invoiced laundry ticket dated 29 June 2013 for 53 bags oflaundry weighing 478 pounds, which, at the contract price of $ I think I must be mis-understanding something. Why would this example (out of all the potential examples) be picked?
- Metus 2y agoNotice that most of the examples have none of the green highlight counter, which is shown for > small losses. KEEPING SCORE: The Dow Jones industrial average rose 32 points, or 0.2 percent, to 18,156 as of 3:15 p.m. Eastern time. The Standard & Poor’s ... OMAHA, Neb. (AP) — Warren Buffett’s company has bought nearly the other sentences are in contrast to show how specific this neuron is.
- svieira 2y agoAh, that makes a lot of sense, thank you!
- yorwba 2y agoThe highlights are better visible in this visualisation: https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html#/model/gpt4/family/v5_latelayer_postmlp/feature/9260 https://openaipublic.blob.core.windows.net/sparse-autoencode... There are also many top activations not showing increases, e.g. > 0.06 of a cent to 90.01 cents US.↵↵U.S. indexes were mainly lower as the Dow Jones industrials lost 21.72 points to 16,329.53, the Nasdaq was up 11.71 points at 4,318.9 and the S&P 500 (Highlight on the first comma.)
- deleted 2y ago[deleted]
- obiefernandez 2y agoCan someone ELI5 the significance of this? (okay maybe not 5, but in basic language)
- 93po 2y agofrom chatgpt itself: The article discusses how researchers use sparse autoencoders to identify and interpret key features within complex language models like GPT-4, making their inner workings more understandable. This advancement helps improve AI safety and reliability by breaking down the models' decision-making processes into simpler, human-interpretable parts.
- OtherShrezzing 2y agoLLM based AIs have lots of "features" which are kind of synonymous with "concepts" - these can be anything from `the concept of an apostrophe in the word don't`, to `"George Wash" is usually followed by "ington" in the context of early American History`. Inside of the LLMs neural network, these are mapped to some circuitry-in-software-esque paths. We don't really have a good way of understanding how these features are generated inside of the LLMs or how their circuitry is activated when outputting them, or why the LLMs are following those circuits. Because of this, we don't have any way to debug this component of an LLM - which makes them harder to improve. Similarly, if LLMs/AIs ever get advanced enough, we'll want to be able to identify if they're being wilfully deceptive towards us, which we can't currently do. For these reasons, we'd like to understand what is actually happening in the neural network to produce & output concepts. This domain of research is usually referred to as "interpretability". OpenAI (and also DeepMind and Anthropic) have found a few ways to inspect the inner circuitry of the LLMs, and reveal a handful of these features. They do this by asking questions of the model, and then inspecting which parts of the LLM's inner circuitry "lights up". They then ablate (turn off) circuitry to see if those features become less frequently used in the AIs response as a verification step. The graphs and highlighted words are visual representations of concepts that they are reasonably certain about - for example, the concept of the word "AND" linking two parts of a sentence together highlights the word "AND". Neel Nanda is the best source for this info if you're interested in interpretability (IMO it's the most interesting software problem out there at the moment), but note that his approach is different to OpenAI's methodology discussed in the post: https://www.neelnanda.io/mechanistic-interpretability https://www.neelnanda.io/mechanistic-interpretability
- andreyk 2y agoExciting to see this so soon after Anthropic's "Mapping the Mind of a Large Language Model" (under 3 weeks). I find these efforts really exciting; it is still common to hear people say "we have no idea how LLMs / Deep Learning works", but that is really a gross generalization as stuff like this shows. Wonder if this was a bit rushed out in response to Anthropic's release (as well as the departure of Jan Leike from OpenAI)... the paper link doesn't even go to Arxiv, and the analysis is not nearly as deep. Though who knows, might be unrelated.
- jerrygenser 2y ago> but that is really a gross generalization as stuff like this shows. I think this research actually still reinforces that we still have very little understanding of the internals. The blog post also reiterates that this is early work with many limitations.
- thegrim33 2y agoFrom the article: "We currently don't understand how to make sense of the neural activity within language models." "Unlike with most human creations, we don’t really understand the inner workings of neural networks." "The [..] networks are not well understood and cannot be easily decomposed into identifiable parts" "[..] the neural activations inside a language model activate with unpredictable patterns, seemingly representing many concepts simultaneously" "Learning a large number of sparse features is challenging, and past work has not been shown to scale well." etc., etc., etc. People say we don't (currently) know why they output what they output, because .. as the article clearly states, we don't.
- surfingdino 2y agoNot holding my breath for that hallucinated cure for cancer then.
- ben_w 2y agoLLMs aren't the only kind of AI, just one of the two current shiny kinds. If a "cure for cancer" (cancer is not just one disease so, unfortunately, that's not even as coherent a request as we'd all like it to be) is what you're hoping for, look instead at the stuff like AlphaFold etc.: https://en.wikipedia.org/wiki/AlphaFold https://en.wikipedia.org/wiki/AlphaFold I don't know how to tell where real science ends and PR bluster begins in such models, though I can say that the closest I've heard to a word against it is "sure, but we've got other things besides protein folding to solve", which is a good sign. (I assume AlphaFold is also a mysterious black box, and that tools such as the one under discussion may help us demystify it too).
- Shoop 2y agoCan anyone summarize the major differences between this and Scaling Monosemanticity?
- calibas 2y agoI want to be able to view exactly how my input is translated into tokens, as well as the embeddings for the tokens.
- franzb 2y agoFor your first question: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer
- calibas 2y agoI saw that, but the language makes me think it's not quite the same as what's really being used? "how a piece of text might be tokenized by a language model" "It's important to note that the exact tokenization process varies between models."
- yorwba 2y agoThat's why they have buttons to choose which model's tokenizer to use.
- calibas 2y agoYes, thank you, I understand that part. It's the might condition in the description that makes me think the results might not be the exact same as what's used in the live models.
- baobabKoodaa 2y agoThe results are the same.
- andy12_ 2y agoa.k.a, the same work as Anthropic, but with less interpretable and interesting features. I guess there won't be Golden Gate[0] GPT anytime soon. I mean, you just have to compare the couple of interesting features of the OpenAI feature browser [1] and the features of the Anthropic feature browser [2]. [0] https://twitter.com/AnthropicAI/status/1793741051867615494 https://twitter.com/AnthropicAI/status/1793741051867615494 [1] https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html#/ https://openaipublic.blob.core.windows.net/sparse-autoencode... [2] https://transformer-circuits.pub/2024/scaling-monosemanticity/features/index.html?featureId=1M_22623 https://transformer-circuits.pub/2024/scaling-monosemanticit...
- leogao 2y agoNote that we focus on random positive activations, which are less susceptible to interpretability illusions than top activations (but also look less impressive as a result). We also provide access to random uncherrypicked features, whereas Anthropic does not. We made these choices deliberately to give as accurate an impression of autoencoder feature quality as possible. Also note that GPT-4 is a more powerful model than Sonnet, which makes it harder to train autoencoders with the same quality features.
- deleted 2y ago[deleted]
- justanotherjoe 2y agoyeah this one is much less presentable than Anthropic's work. It sure looks bad on them to be compared so poorly like this.
- itissid 2y agoDoes this mean that it could be a good practice to release the auto encoder that was trained on a neural network to explain its outputs? Like all open models in hugging face could have this as a useful accompaniment?
- Grimblewald 2y agoI imagine such an encoder would be specific to a model.
- mlsu 2y agoThis is interesting: > Autoencoder family > Note: Only 65536 features available. Activations shown on The Pile (uncopyrighted) instead of our internal training dataset. So, the Pile is uncopyrighted, but the internal training dataset is copyrighted? Copyrighted by whom? Huh?
- Arcsech 2y ago> Copyrighted by whom? By people who would get angry if they could definitively prove their stuff was in OpenAI's training set.
- immibis 2y agoBasically everyone. You, and me, and Elon Musk, and EMPRESS, and my uncle who works for Nintendo. They're just hoping that AI training legally ignores copyright.
- mensetmanusman 2y agoWhen you can ask an AI for an entire book with no errors in the output… god that would be a huge token model
- immibis 2y agoCopyright violation isn't just when you can output 100% exact copies of books. And don't forget, they also violated copyright internally billions of times during training. If any of us had been caught making copies of corporate-owned content for AI training use five years ago, we'd be in for zillion-dollar lawsuits that would make any grandma who downloaded a song from Napster blush.
- Karunamon 2y agoThere is a very good argument to be made that training AI is fair use, as it is both transformative and does not compete with the original work. This has yet to be tested in court.
- 2y ago
- russellbeattie 2y agoOne feature I expect we'll get from this sort of research is identifying "hot spots" that are used during inference. Like virtual machines, these could be cached in whole or in part and used to both speed up the response time and reduce computation cycles needed.
- aeonik 2y agoIn their other examples, they have what looks to be a scientific explanation of reproductive anatomy classified as erotic content... Here is the link to the concept [content warning]: https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html#/model/gpt2-small/family/v5_l8_postmlp/num_features/131072/num_active_features/32/feature/72185 https://openaipublic.blob.core.windows.net/sparse-autoencode... DocID: 191632
- adamiscool8 2y agoHow does this compare to or improve on applying something like SHAP[0][1] on a model? The idea in the first line that "we currently don't understand how to make sense of the neural activity within language models." is..straight up false? [0] https://github.com/shap/shap https://github.com/shap/shap [1] https://en.wikipedia.org/wiki/Shapley_value#In_machine_learning https://en.wikipedia.org/wiki/Shapley_value#In_machine_learn...
- szvsw 2y agoSHAP is pretty separate IMO. Shapley analysis is really a game theoretical methodology that is model agnostic and is only about determining how individual sections of the input contribute to a given prediction, not about how the model actually works internally to produce an output. As long as you have a callable black box, you can compute Shapley values (or approximations); it does not speak to how or why the model actually works internally.
- josu 2y agoTotally off topic, but I love that the external visualisations are hosted on windows.net! https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html#/model/gpt4/family/v5_latelayer_postmlp/feature/63541 https://openaipublic.blob.core.windows.net/sparse-autoencode...
- 29athrowaway 2y agoIt's static content hosted in the Azure blob storage.
- jackallis 2y ago"We currently don't understand how to make sense of the neural activity within language models" this is why peopl are up-in-arms.
- atleastoptimal 2y agoUp in arms for what reason? That neural networks are perfectly interpretable? That's the nature of huge amorphous deep networks. These feature extraction forays are a good step forward though.
- paul7986 2y agoCurious what interesting ways have you used chatGPT ... yesterday i used it to test my pool levels (took a pic of my pool strip i had laying around and told it the brand of pool strip).
- zwaps 2y agoIs anyone else weirded out that they do not acknowledge Anthropic at all?
- leogao 2y agoThe paper cites Anthropic's work extensively.
- kovezd 2y agoWhat applications do you think this opens up? One of the first ones I can think of is pairing it up with Browser extensions to increase productivity of knowledge workers. Think of sales people quickly assessing the viability of leads, or simply reducing noise on social media as a B2C app.
- ndricca 2y agoCan this in some way be related with sparse embeddings (see Splade for example?) and if yes can it be used for hybrid search?
- dazhbog 2y agoIs this like an fMRI for a neural net? We can see which regions light up depending on various topics.. I wonder if an assessment neural net can be plugged in to evaluate the regions that light up automatically.., just like when they had an AI reconstruct what the patient was looking at, from only fMRI scans!
- macrolime 2y agoSeems like this might be used as an alternative to embeddings for creating semantic search. Run a document collection through this to get the activations, run the prompt through the same thing and use the sparse autoencoder to identify which documents has the most activation of features in common with features that the prompt activates. You don't have to know what the features are, but if the prompt includes, say a rhetorical question, it may be able to find documents that also includes rhetorical questions. Not sure how well this would work, but it seems like something that could be interesting to try.
- humansareok1 2y agoSad to note that the team that built this is now completely defunct.
- creativeSlumber 2y agowhat do you mean?
- maronato 2y agoI think it was the superalignment team that was disbanded. This is the safety team, which is related but not the same thing.
- coder1001 2y ago"akin to the small set of concepts a person might have in mind when reasoning about a situation" That does not necessarily mean only a small set of neurons in our brains are engaged. It could be that we use the whole set or a large portion, albeit in a more efficient way!
- parentheses 2y agoAfter anthropic's recent article, this seems almost like making excuses for not having published similarly meaningful contributions. I see a lot of "interpretability is hard" and "we're working on it" in this article but no impact as of yet.