12 ms·
Let's try to understand AI monosemanticity
- turtleyacht 3y agoBy the same token, thinking in memes all the time may be a form of impoverished cognition. Or, is it enhanced cognition, on the part of the interpreter having to unpack much from little?
- aatd86 3y agoSome kind of single context abstract interpretation maybe.
- throwanem 3y agoDarmok and Jalad at Mar-a-Lago.
- kibwen 3y agoShaka, paying to build the walls.
- dekhn 3y ago>By the same token, thinking in memes all the time may be a form of impoverished cognition. I would recast this: any thinking is a linear superposition of weighted tropes. If you read TVTropes enough you'll start to realize that the site doesn't just describe TV plots, but basically all human interaction and thought, nicely clustered into nearly orthogonal topics. Almost anything you can say can be expressed by taking a few tropes and combining them with weights.
- turtleyacht 3y agoThat's what I wonder, though: mimetic thinking as a viral phenomenon. It may reshape one's perceptions, forcing a certain conformance, like prions. If there's any room for alternatives or unconventional thought, it must now be assigned within an established hierarchy. Or, it must be shoehorned into invisible guardrails. This is not bad in itself. Of course, useful ideas must transmit successfully to realize progression over decades. But if one has a choice to focus attention, there also ought to exist the option to ignore it. I don't mean to "ignore all frameworks" or "everything original must come from the self." I just mean I avoid TVTropes--yet I also embrace Dramatica. No doubt TVTropes is a form of design patterns for writing, for which it has incredible value. It's just not something I want to "shape my mind around." As I write this, I know it doesn't make much sense. It may just be a silly thing, and I have yet to "mature to realize the concept of" a shared, tropic mindview.
- erikerikson 3y agoBefore finishing my read, I need to register an objection to the opening which reads to me so as to imply it is the only means: > Researchers simulate a weird type of pseudo-neural-tissue, “reward” it a little every time it becomes a little more like the AI they want, and eventually it becomes the AI they want. This isn't the only way. Back propagation is a hack around the oversimplification of neural models. By adding a sense of location into the network, you get linearly inseparable functions learned just fine. Hopfield networks with Hebbian learning are sufficient and are implemented by the existing proofs of concept we have.
- janalsncm 3y agoThis is true. We use backpropagation not because it’s the only way or because it’s biologically plausible (the brain doesn’t have any backward passes) but because it works. Neural networks aren’t special because of any sort of connection to the brain, we use them because we have hardware (GPUs) which can train them pretty quickly. I feel the same way about transformers vs RNNs: even if RNNs are more “correct” in some sense of having theoretically infinite memory it takes forever to train them so transformers won. And then we developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant.
- bilsbie 3y ago> developed techniques like Long LoRA which make theoretical disadvantages functionally irrelevant. How’s that?
- janalsncm 3y agoContext windows used to be tiny e.g. 768 or 1024. That meant sliding windows of limited context if you had any reasonably sized input. If your context window is 32k or even 100k, a lot of inputs will fit into the context window entirely. The reason huge context windows weren't possible in the past is that memory requirements were quadratic with input length. Long LoRA lets us use less memory for our context windows, or use the same memory footprint for larger context windows.
- s1gnp0st 3y ago> Shouldn’t the AI be keeping the concept of God, Almighty Creator and Lord of the Universe, separate from God- This seems wrong. God-zilla is using the concept of God as a superlative modifier. I would expect a neuron involved in the concept of godhood to activate whenever any metaphorical "god-of-X" concept is being used.
- Sniffnoy 3y agoI mean, it's not actually. It's just a somewhat unusual transcription (well, originally somewhat unusual, now obviously it's the official English name) of what might be more usually transcribed as "Gojira".
- s1gnp0st 3y agoAh, I thought the Japanese word was just "jira". My mistake.
- postmodest 3y agoThat's an entirely different monster.
- eichin 3y agoIndeed, but not an entirely unrelated one though - per https://en.wikipedia.org/wiki/Jira_(software)#Naming https://en.wikipedia.org/wiki/Jira_(software)#Naming the inspiration path was Bugzilla -> Godzilla -> Gojira -> Jira (which is why Confluence keeps correcting me when I try to spell it JIRA)
- VinLucero 3y agoI see what you did there.
- lukev 3y agoThere's actually a somewhat reasonable analogy to human cognitive processes here, I think, in the sense that humans tend to form concepts defined by their connectivity to other concepts (c.f. Ferdinand de Saussure & structuralism). Human brains are also a "black box" in the sense that you can't scan/dissect one to build a concept graph. Neural nets do seem to have some sort of emergent structural concept graph, in the case of LLMs it's largely informed by human language (because that's what they're trained on.) To an extent, we can observe this empirically through their output even if the first principles are opaque.
- wahern 3y ago> Neural nets do seem to have some sort of emergent structural concept graph, in the case of LLMs it's largely informed by human language (because that's what they're trained on.) To an extent, we can observe this empirically through their output even if the first principles are opaque. Alternatively, what you're seeing are the structures inherent within human culture as manifested through its literature[1], with LLMs simply being a new and useful tool which makes these structures more apparent. [1] And also its engineers' training choices
- hprotagonist 3y agoi buy this. cf the “lossy jpeg of the web” analogy; sure there’s structure: we put it there!
- pixl97 3y agoThen why wouldn't any LLM trained on real data have structure? I mean we don't live in the chaos realm, atoms make molecules, those molecules make structures, those structures make cells, organs, lifeforms.
- hprotagonist 3y agothe degrees to which the (non)deterministic and (non)atomistic models of the universe have predictive power aside — yes, sure: you should expect a well trained NN to have an internal structure that approximates that of the statistical distribution of encoded features of the training data into the latent space of the model. that’s very nearly the definition of “trained”.
- _as_text 3y agoI just skimmed through it for now, but it has seemed kinda natural to me for a few months now that there would be a deep connection between neural networks and differential or algebraic geometry. Each ReLU layer is just a (quasi-)linear transformation, and a pass through two layers is basically also a linear transformation. If you say you want some piece of information to stay (numerically) intact as it passes through the network, you say you want that piece of information to be processed in the same way in each layer. The groups of linear transformations that "all process information in the same way, and their compositions do, as well" are basically the Lie groups. Anyone else ever had this thought? I imagine if nothing catastrophic happens we'll have a really beautiful theory of all this someday, which I won't create, but maybe I'll be able to understand it after a lot of hard work.
- mathematicaster 3y agoI did a little spelunking some time ago reacting to the same urge. Tropical geometry appears to be where the math talk is at. Just dropping the reference here, I don't grok the literature.
- KhoomeiK 3y agoYou might be interested in this workshop: https://www.neurreps.org/ https://www.neurreps.org/ And a possibly relevant paper from it: https://openreview.net/forum?id=Ag8HcNFfDsg https://openreview.net/forum?id=Ag8HcNFfDsg
- zozbot234 3y agoReLU is quite far from linear, adding ReLU activations to a linear layer amounts to fitting a piecewise-segmented model of the underlying data.
- drdeca 3y agoWell, at all but a finite number of points (specifically all but one point), there is a neighborhood of that point at which ReLU matches a linear function... In one sense, that seems rather close to being linear. If you take a random point (according to a continuous probability distribution) , then with probability 1, if look in a small enough neighborhood of the selected point, it will be indistinguishable from linear within that neighborhood. And, for a network made of ReLU gates and affine maps, still get that it looks indistinguishable from affine on any small enough region around any point outside of a set of measure zero. So... Depends what we mean by “almost linear” I think. I think one can make a reasonable case for saying that, in a sense it is “almost linear”. But yes, of course I agree that in another important sense, it is far from linear. (E.g. it is not well approximated by any linear function)
- shermantanktop 3y agoAs described in the post, this seems quite analogous to the operation of a bloom filter, except each "bit" is more than a single bit's worth of information, and the match detection has to do some thresholding/ranking to select a winner. That said, the post is itself clearly summarizing much more technical work, so my analogy is resting on shaky ground.
- gmuslera 3y agoAt least the first part reminded me of Hyperion and how AIs evolved there (I think the actual explanation is in The Fall of Hyperion), smaller but more interconnected "code". Not sure about actual implementation, but at least for us concepts or words are not pure nor isolated, they have multiple meanings that collapse into specific ones as you put several together
- daveguy 3y agoAll this anthropomorphizing of activation networks strikes me as very odd. None of these neurons "want" to do anything. They respond to specific input. Maybe humans are the same, but in the case of artificial neural networks we at least know it's a simple mathematical function. Also, an artificial neuron is nothing like a biological neuron. At the most basic -- artificial neurons don't "fire" except in direct response to inputs. Biological neurons fire because of their internal state, state which is modified by biological signaling chemicals. It's like comparing apples to gorillas.
- lainga 3y agoWait till you find out how we talk about evolutionary adaptation... a couple analogies and elided concepts here and there and you'd swear Lamarck had smothered Darwin in his cradle
- jjw1414 3y agoYes, Lamarck is alive and well. I frequently hear colleagues in biology with years of experience making lazy statements such as, "Humans evolved to walk because..." as if evolution is a conscious act. In many cases, these people are using the term evolution as shorthand for, "Random mutations gave rise to new gene variants (alleles), and over numerous generations, natural selection leads to an increase in the prevalence of beneficial variants, specifically those that provide a reproductive advantage. The genetic variants that combined to enable bipedal locomotion provided such an advantage to the human population". Evolution is a passive, natural process driven by genetic variation, environmental changes, and the differential reproductive success of individuals with advantageous traits. This in turn results in population changes. Evolution doesn't involve a conscious choice or direction. Evolution is the result of the cumulative effects of these factors over long periods of time.
- error9348 3y ago> No one knows how it works. Researchers simulate a weird type of pseudo-neural-tissue, “reward” it a little every time it becomes a little more like the AI they want, and eventually it becomes the AI they want. There is a distinction to be made in "knowing how it works" on architecture vs weights themselves.
- asylteltine 3y agoI’ve never understood the hidden layers argument. Ultimately these models are executing code. You can examine the code. Why can’t that be done?
- afthonos 3y agoThink of it this way: looking at the neurons is like looking at the assembly and trying to understand how a video is played on YouTube. Yes, in theory it’s possible, but realistically no one would be able to do that. At least not in a reasonable amount of time.
- astrange 3y agoVideo decoding works by reading and writing pixels to the same buffers you use for displaying them, so you could see a lot of how it works by watching the accesses there.
- afthonos 3y agoYeah, but imagine you don’t know what “decoding”, “buffers”, and “display” mean in a software context.
- OkayPhysicist 3y agoFundamentally, artificial neural networks boil down to some code that can be written, mathematically, as σ(C*σ(B*σ(A*x)))... = y repeated matrix-vector multiplication, separated by nonlinear functions, σ(). A,B,C etc here are your weights, x is your input vector, y is your output vector. Any nonlinear function will work, some behave better than others, but they must be present, otherwise you could simplify the problem by apply the associative property to the weights side of things, and you'd effectively have a single layer that can only simulate linear functions. A hidden layer is just any vector that is intermediate to that calculation, for example the result of σ(A*x) would be the first hidden layer. So right off the bat, you obviously have a problem: "Examining the code" means examining the weights, which are just gigabyte/terabyte sized matrices. Opaque is putting it mildly. The only sane approach is to start from one of our known-meaningful vectors (the input or output), and work our way inwards from there, either seeing what elements of hidden layer vectors have the most significant value when a certain input is applied, or determining what hidden layer vector values produce values closest resembling a desired output. Then you start running into the problems described in the article.
- adamnemecek 3y agoSuperposition makes sense when you understand all of ML as a convolution.
- blovescoffee 3y agoYou can just note the fact that two bits can represent four values.
- adamnemecek 3y agoCan you explain machine learning through this lens?
- kevdozer1 3y agosuch a wonderful article, really enjoyed reading it
- nl 3y agoPersonally I find the original paper much better written and easier to understand: https://transformer-circuits.pub/2023/monosemantic-features/index.html https://transformer-circuits.pub/2023/monosemantic-features/...
- zone411 3y agoIf you have a background in ML, then yes, a paper is almost always better. I've recommended papers over sources like Towards Data Science etc. here. However, for laypeople, I doubt it would be as effective - they'd need to look up terms like MLP, ReLU, UMAP, Logit, or even what an activation function is, and they are the target audience of this post.
- FeepingCreature 3y agoThere's also a difference in goals. The point of a paper is to convince an expert that the claim is correct. With an article, you can just assume that the claim is correct (or give it any level of credence you want), because the point is to inform more than convince. So the article can skip or sketch all the showing-your-work and steps and proofs and just present the conclusions.
- csours 3y agoI feel like we're a few more paradigm shifts away from self-driving cars, and this is one of them - being able to actually understand neural nets and modify them in a constructive way more directly - aka engineering. Some more: cheaper sensors (happening now) better sensor integration (happening now, kind of) better tools for ml grokking and intermediate engineering (this article, kind of) better tools for layering ml (probably the same thing as above) a new model for insurance/responsibility/something like this (unsure) better communication with people inside and outside the car (barely on the radar)
- Merrill 3y agoWhen LLMs are trained on text, are the words annotated to indicate the semantic meaning, or is the LLM training process expected to disambiguate the possibly hundreds of semantic meanings of an individual common word such as "run"?
- OkayPhysicist 3y agoThe task LLMs are trained on is "predict the next word", which elegantly is included for free in your training set of text. Typically no annotation is provided, since that would involve a ton of human labor doing the annotations.
- Merrill 3y agoCould you ask a specialized AI to define each of the words in a block of text to automate the meaning extraction?
- IanCal 3y agoProbably. There's good work showing impressively performing small models that were trained on more "text book" like data rather than just loads of text - but where the "text books" were either wholly or largely created by another AI model. Using models to generate/score/rank/modify data to be more useful as training data is a very interesting angle.
- DonsDiscountGas 3y agoIsn't this also(just?) a description of how high-dimensional embedding spaces work? Putting every kind of concept all in the same space is going to lead to some weird stuff. Different regions of the latent space will cover different concepts, with very uneven volumes, and local distances will generally be meaningful (red vs green) but long-distances won't (red vs. ennui). I guess we could also look at it the other way; embedding spaces work this way because the underlying neurons work this way.
- bjackman 3y agoI had the same feeling, I have always seen it like: 1. Neurons encode a concept and activate when it shows up. 2. No, it's way more complicated and mysterious than that. And now this seems to add: 3. Actually, it's only more complicated than that in a fairly straightforward mathematical sense. It's not that mysterious at all. I suspect this means that either I'm not picking up on subtleties in the article, or Scott is representing it in a way that slightly oversimplified the situation! On the other hand, the last quote in the article from the researchers does seem to be hitting the "it's not that mysterious" note. A simple matter of very hard engineering. So, I dunno. Cool!
- ggm 3y agoAs long as you read it with the skeptics "yes, but it's not intelligence" its a good read. It's when you read it with the "at last, I can understand how reasoning and inference with meaning is going to emerge from this" you have a problem. It's a great read but what you bring to it, informs what you take from it.
- lngnmn2 3y agoEvery training set will produce a different set of weights, even the same training set with produce different weights with different initialization, leave alone slightly different architectures. So what exactly is the point, except "look at us, we are so clever"?
- lngnmn2 3y ago[dead]
- chrissnow2023 3y agoCOOL~ interpreting a big AI with a bigger is like interpreting 42 with Earth~
- keithalewis 3y ago[disclaimer: after talking to many people much smarter than me, I might, just barely, sort of understand this. Any mistakes below are my own] Thanks for the heads up! AI is better at making up stories than humans already. Hodl my beer while I go buy some BSitCoins.
- Nelkins 3y agoThis reminds of research done on category-specific semantic deficits, where there can be neurodegeneration that impacts highly specific or related knowledge (for example, brain trauma that affects a person's ability to understand living things like zebras and carrots, but not non-living things like helicopters or pliers). https://academic.oup.com/brain/article/130/4/1127/278057 https://academic.oup.com/brain/article/130/4/1127/278057
- dmd 3y agoThe idea of a smaller NN outperforming what you might think it could do by simulating a larger one reminds me of something I read about Portia spiders (going on a deep dive into them after reading the excellent 'Children of Time' by Adrian Tchaikovsky). The idea is that they're able to do things with their handful - on the order of tens of thousands - of neurons that you'd think would require 4 or 5 orders of magnitude more, by basically time-sharing them; do some computation, store it somehow, then reuse the same neurons in a totally different way.
- pas 3y agosee https://www.nature.com/articles/nature12160 https://www.nature.com/articles/nature12160 "mixed selectivity neurons"
- brindlejim 3y agoThis is egregiously false. ML models are math and code. They are not tissue; there are no neurons in an ML model. None. The analogy is false. And it is used here to make you think that ML models are more human than they are, or more superhuman. There is no "bigger AI waiting to get out". Shoggoth anyone? These are scare tactics from hyperventilating EAs (both Scott and the Anthropic team).
- makeworld 3y agoThere are neurons. https://en.wikipedia.org/wiki/Artificial_neuron https://en.wikipedia.org/wiki/Artificial_neuron
- brindlejim 3y agoThe repetition of a false analogy does not make it true. They are no more neurons than transistors are "little brain cells". This is just shorthand for outsiders who can't hope to understand. But on HN, we have technologists. The nodes of an ML model are made of math and code. There is no neuron there, and those nodes work very different than neurons in their action and structure. The analogy breaks down.
- reexpressionist 3y ago[dead]
- drc500free 3y agoThe original is quite long, but quite interesting.[1] Reading it makes me feel like I did reading A Brief History of Time as a middle schooler - concepts that are mainly just out of reach, with a few flashes that I actually understand. One particularly interesting topic is the "theories of superposition" section, which gets into how LLMs categorize concepts. Are concepts all distinct or indistinct? Are they independent or do they cluster? It seems that the answer is all of the above. This ties into linguistic theories of categorization[2] that I saw referenced in (of all places) a book about the partition of Judaeo-Christianity in the first centuries CE. Some categories have hard lines - something is a "bird" or it is not. Some categories have soft lines - like someone being "tall." Some categories work on prototypes, making them have different intensities within the space - A sparrow, swallow, or robin is more "birdy" than a chicken, emu, or turkey. Apparently Wittgenstein was the first to really explore with Family Resemblances that a category might not have hard boundaries, according to people who study these things.[3] These sorts of "manifolds" seem to appear, where some concepts are not just distinct points that are or aren't. It's exciting to see that LLMs may give us insights into how our brains store concepts. I've heard people criticize them as "just predicting the next most likely token," but I've found myself lost when speaking in the middle of a garden path sentence many times. I don't know how a sentence will end before I start saying it, and it's certainly plausible that LLMs actually do match they way we speak. Probably the most exciting piece is seeing how close they seem to get to mimicking how we communicate and think, while being fully limited to language with no other modeling behind it - no concept of the physical world, no understanding of counting or math, just words. It's clear when you scratch the surface that LLM outputs are bullshit with no thought underneath them, but it's amazing how much is covered by linking concepts with no logic other than how you've heard them linked before. [1] https://transformer-circuits.pub/2023/monosemantic-features/index.html https://transformer-circuits.pub/2023/monosemantic-features/... [2] https://www.sciencedirect.com/science/article/abs/pii/001002857690013X https://www.sciencedirect.com/science/article/abs/pii/001002... [3] https://en.wikipedia.org/wiki/Family_resemblance https://en.wikipedia.org/wiki/Family_resemblance