10 ms·

LLMs use a surprisingly simple mechanism to retrieve some stored knowledge

408 points by CharlesW 3y ago

leobg 3y ago
> In one experiment, they started with the prompt “Bill Bradley was a” and used the decoding functions for “plays sports” and “attended university” to see if the model knows that Sen. Bradley was a basketball player who attended Princeton. Why not just change the prompt? Name, University attended, Sport played Bill Bradley,
- numeri 3y ago
  This is research, trying to understand the fundamentals of how these models work. They weren't actually trying to find out where Bill Bradley went to university.
  - leobg 3y ago
    Of course. But weren’t they trying to find out whether or not that fact was represented in the model’s parameters?
    - wnoise 3y ago
      No, they were trying to figure out if they had isolated where facts like that were represented.
vsnf 3y ago
> Linear functions, equations with only two variables and no exponents, capture the straightforward, straight-line relationship between two variables Is this definition considering the output to be included in the set of variables? What a strange way to phrase it. Under this definition, I wonder what an equation with one variable is. Is a single constant an equation?
- 01HNNWZ0MV43FF 3y ago
  Yeah I guess they mean one independent variable and one dependent variable It rarely matters because if you had 2 dependent variables, you can just express that as 2 equations, so you might as well assume there's exactly 1 dependent and then only discuss the number of independent variables.
- olejorgenb 3y ago
  I would think `x = 4` is considered an equation, yes?
  - pessimizer 3y ago
    And linear at that: x = 0y + 4
- ksenzee 3y ago
  I think they're trying to say "equations in the form y = mx + b" without getting too technical.
- hansvm 3y ago
  It's just a change in perspective. Consider a vertical line. To have an "output" variable you have to switch the ordinary `y=mx+b` formulation to `x=c`. The generalization `ax+by=c` accommodates any shifted line you can draw. Adding more variables increases the dimension of the space in consideration (`ax+by+cz=d` could potentially define a plane). Adding more equations potentially reduces the size of the space in consideration (e.g., if `x+y=1` then also knowing `2x+2y=2` wouldn't reduce the solution space, but `x-y=0` would, and would imply `x=y=1/2`, and further adding `x+2y=12` would imply a lack of solutions). Mind you, the "two variable" statement in this news piece is a red-herring. The paper describes higher-dimension linear relationships, of the form `Mv=c` for some constant matrix `M`, some constant vector `c`, and some variable vector `v`. On some level, the result isn't _that_ surprising. The paper only examines one layer (not the whole network), after the network has done a huge amount of embedding work. In that layer, they find that under half the time they're able to get over 60% of the way there with a linear approximation. Another interpretation is that the single layer does some linear work and shoves it through some nonlinear transformations, and more than half the time that nonlinearity does something very meaningful (and even in that under half the time where the linear approximation is "okay", the metrics are still bad). I'm not super impressed, but I don't have time to full parse the thing right now. It is a bit surprising; if memory serves, one of the authors on this paper had a much better result in terms of neural network fact editing in the last year or two. This looks like a solid research idea, solid work, it didn't pan out, and to get it published they heavily overstated the conclusions (and then the university press release obviously bragged as much as it could).
whatever1 3y ago
Llms seem like a good compression mechanism. It blows my mind that I can have a copy of llama locally on my PC and have access to virtually the entire internet
- Culonavirus 3y ago
  Yea except it's a lossy compression. With the lost part being hallucinated in at inference time.
  - Kuinox 3y ago
    If you've read the article, the LLM hallucinations aren't due to the model not knowing the information but a function that choose to remember the wrong thing.
    - sinemetu11 3y ago
      From the paper: > Finally, we use our dataset and LRE-estimating method to build a visualization tool we call an attribute lens. Instead of showing the next token distribution like Logit Lens (nostalgebraist, 2020) the attribute lens shows the object-token distribution at each layer for a given relation. This lets us visualize where and when the LM finishes retrieving knowledge about a specific relation, and can reveal the presence of knowledge about attributes even when that knowledge does not reach the output. They're just looking at what lights up in the embedding when they feed something in, and whatever lights up is "knowing" about that topic. The function is an approximation they added on top of the model. It's important to not conflate this with the actual weights of the model. You can't separate the hallucinations from the model -- they exist precisely because of the lossy compression.
      - deleted 3y ago
        [deleted]
    - ewild 3y ago
      even this place has people not reading the articles. we are doomed
aia24Q1 3y ago
I thought "fact" means truth.
MuffinFlavored 3y ago
I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects. edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32? > Quantization is the process of mapping 32-bit floating-point numbers (which are the weights in the neural network) to a much smaller bit representation, like 4-bit values, for storage and memory efficiency. > Dequantization happens when the model is used (during inference or even training, if applicable). The 4-bit quantized weights are converted back into floating-point numbers that the model's computations are actually performed with. This is done using the scale and zero-point determined during the initial quantization, or through more sophisticated mapping functions that aim to preserve as much information as possible despite the reduced precision. so what is the relationship to "parameters" and "# of unique tokens the model knows about (vocabulary size)"? > At first glance, LLAMa only has a 32,000 vocabulary size and 65B parameters as compared to GPT-3, > The 65 billion parameters in a model like LLAMA (or any large language model) essentially function as a highly intricate mapping system that determines how to respond to a given input based on the learned relationships between tokens in its training data.
- Filligree 3y ago
  It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
  - MuffinFlavored 3y ago
    say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its learned statistical patterns to predict the probability of what comes next in a sequence of text. based on what inputs? 1. previous tokens in the sequence from immediate context 2. tokens summarizing the overall topic/subject matter from the extended context 3. scoring of learned patterns from training 4. what else?
estebarb 3y ago
I find this similar to what relation vectors do in word2vec: you can add a vector of "X of" and often get the correct answer. It could be that the principle is still the same, and transformers "just" build a better mapping of entities into the embedding space?
- PaulHoule 3y ago
  I think so. It’s hard for me to believe that the decision surfaces inside those models are really curved enough (like the folds of your brain) to really take advantage of FP32 numbers inside vectors: that is I just don’t believe it is x = 0 means “fly” x = 0.01 means “drive” x = 0.02 means “purple” but rather more like x < 1.5 means “cold” x > 1.5 means “hot” which is one reason why quantization (often 1 bit) works. Also it is a reason why you can often get great results feeding text or images through a BERT or CLIP-type model and then applying classical ML models that frequently involve linear decision surfaces.
  - taneq 3y ago
    Are you conflating nonlinear embedding spaces with the physical curvature of the cerebellum? I don't think there's a direct mapping.
    - PaulHoule 3y ago
      My mental picture is that violently curved decision surfaces could look like the convolutions of the brain even though they have nothing to do with how the brain actually works. I think of how tSNE and other algorithms sometimes produce projections that sometimes look like that (maybe that’s just what you get when you have to bend something complicated to fit into a 2-d space) and frequently show cusps that to me look like a sign of trouble (took me a while in my PhD work to realize how Poincaré sections from 4 or 6 dimensions can look messed up when a part of the energy surface tilts perpendicularly to the projection surface.) I still find it hard to believe that dense vectors are the right way to deal with text despite the fact that they work so well. For images it is one thing because changing one pixel a little doesn’t change the meaning of an image, but changing a single character of a text can completely change the meaning of the text. Also there’s the reality that if you randomly stick together tokens you get something meaningless, so it seems almost all of the representation space covers ill formed texts and only a low dimensional manifold holds the well formed texts. Now the decision surfaces really have to be nonlinear and crumpled over all but I think there’s a definitely a limit on how crumpled those surfaces can be.
derefr 3y ago
Help me understand: when they say that the facts are stored as a linear function… are they saying that the LLM has a sort of N-dimensional “fact space” encoded into the model in some manner, where facts are embedded into the space as (points / hyperspheres / Voronoi manifolds / etc); and where recalling a fact is — at least in an abstract sense — the NN computing / remembering a key to use, and then doing a key-value lookup in this space? If so: how do you embed a KV-store into an edge-propagated graphical model? Are there even any well-known techniques for doing that “by hand” right now? (Also, fun tangent: isn't the "memory palace" memory technique, an example of human brains embedding facts into a linear function for easier retrieval?)
- bionhoward 3y ago
  [Layer] Normalization constrains huge vectors representing tokens (input fragments) to positions on a unit ball (I think), and the attention mechanism operates by rotating the unconstrained ones based on the sum of their angles relative to all the others. I only skimmed the paper but believe the point here is that there are relatively simple functions hiding in or recoverable from the bigger network which specifically address certain categories of relationships between concepts. Since it would, in theory, be possible to optimize such functions more directly if they are possible to isolate, could this enable advances in the way such models are trained? Absolutely. After all, one of the best criticisms of “modern” AI is the notion we’re just mixing around a soup of linear algebra. Allowing some sense of modularity (reductionism) could make them less of a black box and more of a component driven approach (in the lagging concept space and not just the leading layer space)
- jacobn 3y ago
  The fundamental operation done by the transformer, softmax(Q.K^T).V, is essentially a KV-store lookup. The Query is dotted with the Key, then you take the softmax to pick mostly one winning Key (the Key closest to the Query basically), and then use the corresponding Value. That is really, really close to a KV lookup, except it's a little soft (i.e. can hit multiple Keys), and it can be optimized using gradient descent style methods to find the suitable QKV mappings.
  - 3y ago
wslh 3y ago
Can we roughly say that LLMs produces (training mode) a lot of IF-THENs in an automatic way from a vast quantity of information (nor techniques) that was not available before?
mike_hearn 3y ago
This is really cool. My mind goes immediately to what sort of functions are being used to encode programming knowledge, and if they are also simple linear functions whether the standard library or other libraries can be directly uploaded into an LLMs brain as it evolves, without needing to go through a costly training or performance-destroying fine-tune. That's still a sci-fi ability today but it seems to be getting closer.
- Animats 3y ago
  That's a good point. It may be possible to directly upload predicate-type info into a LLM. This could be especially useful if you need to encode tabular data. Somewhere, someone probably read this and is thinking about how to export Excel or databases to an LLM. It's encouraging to see people looking inside the black box successfully. The other big result in this area was that paper which found a representation of a game board inside a LLM after the LLM had trained to play a game. Any other good results in that area? The authors point out that LLMs are doing more than encoding predicate-type info. That's just part of what they are doing.
  - AaronFriel 3y ago
    It indeed is. An attention mechanism's key and value matrices grow linearly with context length. With PagedAttention[1], we could imagine an external service providing context. The hard part is the how, of course. We can't load our entire database in every conversation, and I suspect there are also challenges around training (perhaps addressed via LandmarkAttention[2]) and building a service efficiently retrieve additional key-value matrices. The external service vector database may require tight timings necessary to avoid stalling LLMs. To manage 20-50 tokens/sec, answers must arrive within 50-20ms. And we cannot do this in real-time, pausing the transformer when a layer produces a query vector stalls the batch, so we need a way to predict queries (or embeddings) several tokens ahead of where they'd be useful and inject the context in when it's needed, and to know when to page it out. [1] https://arxiv.org/abs/2309.06180 https://arxiv.org/abs/2309.06180 [2] https://arxiv.org/abs/2305.16300 https://arxiv.org/abs/2305.16300
  - wongarsu 3y ago
i5heu 3y ago
So it is entirely possible to decouple the reasoning part from the information part? This is like absolutely mind blowing if this is true.
- learned 3y ago
  A big caveat mentioned in the article is that this experiment was done with a small set (N=47) of specific questions that they expected to have relatively simple relational answers: > The researchers developed a method to estimate these simple functions, and then computed functions for 47 different relations, such as “capital city of a country” and “lead singer of a band.” While there could be an infinite number of possible relations, the researchers chose to study this specific subset because they are representative of the kinds of facts that can be written in this way. About 60% of these relations were retrieved using a linear function in the model. The remaining appeared to have nonlinear retrieval and is still a subject of investigation: > Functions retrieved the correct information more than 60 percent of the time, showing that some information in a transformer is encoded and retrieved in this way. “But not everything is linearly encoded. For some facts, even though the model knows them and will predict text that is consistent with these facts, we can’t find linear functions for them. This suggests that the model is doing something more intricate to store that information,” he says.
deleted 3y ago
[deleted]
retrofrost 3y ago
This is amazing work, but to me it highlights some of the biggest problems in the current AI zeitgeist, we are not really trying to work on any neuron or ruleset that isnt much different from the perceptron thats just a sumnation function. Is it really that suprising that we just see this same structure repeated in the models. Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? We have all sorts of unique training methods and encoding schemes that don't ever get used because the big libraries don't support them. Until, we start seeing real varation in the fundamental rulesets of neuralnets we are always just going to be fighting against the fact these are just perceptrons with extra steps.
- visarga 3y ago
  > Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.
  - dartos 3y ago
    I mean RWKV seems promising and isn’t a transformer model. Transformers have first mover advantage. They were the first models that scaled to large parameter counts. That doesn’t mean they’re the best or that they’ve won, just that they were the first to get big (literally and metaphorically)
    - tkellogg 3y ago
      Yeah, I'd argue that transformers created such capital saturation that there's a ton of opportunity for alternative approaches to emerge.
      - dartos 3y ago
        Speak of the devil. Jamba just hit the front page.
zyklonix 3y ago
This reminds me of the famous "King - Man + Woman = Queen" embedding example. The fact that embeddings have semantic properties in them explains why simple linear functions would work as well.
robertclaus 3y ago
I think this paper is cool and I love that they ran these experiments to validate these ideas. However, I'm having trouble reconciling the novelty of the ideas themselves. Isn't this result expected given that LLM's naturally learn simple statistical trends between words? To me it's way cooler that they clearly demonstrated not all LLM behavior can be explained this simply.
mikewarot 3y ago
I wonder if this relation still holds with newer models that have have even more compute thrown at them? My intuition is that the structure inherent to language makes Word2Vec possible. Then training on terabytes of human text encoded with Word2Vec + Positional Encoding makes it possible to then have the ability to predict the next encoding at superhuman levels of cognition (while training!). It's my sense that the bag of words (as input/output method) combined with limited context windows (to make Positional Encoding work) is a huge impedance mismatch to the internal cognitive structure. Thus I think that given the orders of magnitude more compute thrown at GPT-4 et al, it's entirely possible new forms of representation evolved and remain to be discovered by humans probing through all the weights. I also think that MemGPT could, eventually, become an AGI because of the unlimited long term memory. More likely, though, I think it would be like the protagonist in Memento[1]. [1] https://en.wikipedia.org/wiki/Memento_(film) https://en.wikipedia.org/wiki/Memento_(film) [edit - revise to address question]
- autokad 3y ago
  sorry if I misread your comment, but you seem to be indicating that LLMs such as chat gpt (which use gpt 3+) are bag of words models? they are sequence models.
  - mikewarot 3y ago
    I edited my response... I hope it helps... my understanding is that the output gives probabilities for all the words, then one is chosen with some random thrown in (via the #temperature) then fed back in... which to me seems to equate to bag of words. Perhaps I mis-understood the term.
    - smaddox 3y ago
      Bag of words models use a context that is a "bag" (i.e. an unorder map from elements to their counts) of words/tokens. GPT's use a context that is a sequence (i.e. an ordered list) of words/tokens.
uoaei 3y ago
This is the "random linear projections as memorization technique" perspective on Transformers. It's not a new idea per se, but nice to see it fleshed out. If you dig into this perspective, it does temper any claims of "cognitive behavior" quite strongly, if only because Transformers have such a large capacity for these kinds of "memories".
- tel 3y ago
  Do you have a reference on “random linear projections as memorization”? I know random projections quite well but haven’t seen that connection.
seydor 3y ago
Does this point to a way to compress entire LLMs by selecting a set of relations?

408 points by CharlesW 3y ago

> In one experiment, they started with the prompt “Bill Bradley was a” and used the decoding functions for “plays sports” and “attended university” to see if the model knows that Sen. Bradley was a basketball player who attended Princeton. Why not just change the prompt? Name, University attended, Sport played Bill Bradley,

numeri 3y ago
This is research, trying to understand the fundamentals of how these models work. They weren't actually trying to find out where Bill Bradley went to university.
- leobg 3y ago
  Of course. But weren’t they trying to find out whether or not that fact was represented in the model’s parameters?
  - wnoise 3y ago
    No, they were trying to figure out if they had isolated where facts like that were represented.

vsnf 3y ago

> Linear functions, equations with only two variables and no exponents, capture the straightforward, straight-line relationship between two variables Is this definition considering the output to be included in the set of variables? What a strange way to phrase it. Under this definition, I wonder what an equation with one variable is. Is a single constant an equation?

01HNNWZ0MV43FF 3y ago
Yeah I guess they mean one independent variable and one dependent variable It rarely matters because if you had 2 dependent variables, you can just express that as 2 equations, so you might as well assume there's exactly 1 dependent and then only discuss the number of independent variables.
olejorgenb 3y ago
I would think `x = 4` is considered an equation, yes?
- pessimizer 3y ago
  And linear at that: x = 0y + 4
ksenzee 3y ago
I think they're trying to say "equations in the form y = mx + b" without getting too technical.
hansvm 3y ago
It's just a change in perspective. Consider a vertical line. To have an "output" variable you have to switch the ordinary `y=mx+b` formulation to `x=c`. The generalization `ax+by=c` accommodates any shifted line you can draw. Adding more variables increases the dimension of the space in consideration (`ax+by+cz=d` could potentially define a plane). Adding more equations potentially reduces the size of the space in consideration (e.g., if `x+y=1` then also knowing `2x+2y=2` wouldn't reduce the solution space, but `x-y=0` would, and would imply `x=y=1/2`, and further adding `x+2y=12` would imply a lack of solutions). Mind you, the "two variable" statement in this news piece is a red-herring. The paper describes higher-dimension linear relationships, of the form `Mv=c` for some constant matrix `M`, some constant vector `c`, and some variable vector `v`. On some level, the result isn't _that_ surprising. The paper only examines one layer (not the whole network), after the network has done a huge amount of embedding work. In that layer, they find that under half the time they're able to get over 60% of the way there with a linear approximation. Another interpretation is that the single layer does some linear work and shoves it through some nonlinear transformations, and more than half the time that nonlinearity does something very meaningful (and even in that under half the time where the linear approximation is "okay", the metrics are still bad). I'm not super impressed, but I don't have time to full parse the thing right now. It is a bit surprising; if memory serves, one of the authors on this paper had a much better result in terms of neural network fact editing in the last year or two. This looks like a solid research idea, solid work, it didn't pan out, and to get it published they heavily overstated the conclusions (and then the university press release obviously bragged as much as it could).

whatever1 3y ago

Llms seem like a good compression mechanism. It blows my mind that I can have a copy of llama locally on my PC and have access to virtually the entire internet

Culonavirus 3y ago
Yea except it's a lossy compression. With the lost part being hallucinated in at inference time.
- Kuinox 3y ago
  If you've read the article, the LLM hallucinations aren't due to the model not knowing the information but a function that choose to remember the wrong thing.
  - sinemetu11 3y ago
    From the paper: > Finally, we use our dataset and LRE-estimating method to build a visualization tool we call an attribute lens. Instead of showing the next token distribution like Logit Lens (nostalgebraist, 2020) the attribute lens shows the object-token distribution at each layer for a given relation. This lets us visualize where and when the LM finishes retrieving knowledge about a specific relation, and can reveal the presence of knowledge about attributes even when that knowledge does not reach the output. They're just looking at what lights up in the embedding when they feed something in, and whatever lights up is "knowing" about that topic. The function is an approximation they added on top of the model. It's important to not conflate this with the actual weights of the model. You can't separate the hallucinations from the model -- they exist precisely because of the lossy compression.
    - deleted 3y ago
      [deleted]
  - ewild 3y ago
    even this place has people not reading the articles. we are doomed

aia24Q1 3y ago

I thought "fact" means truth.

MuffinFlavored 3y ago

I don't understand how a "CSV file/database/model" of 70,000,000,000 (70B) "parameters" of 4-bit weights (a 4 bit value can be 1 of 16 unique numbers) gets us an interactive LLM/GPT that is near-all-knowledgable on all topics/subjects. edit: did research, the 4-bit is just a "compression method", the model ends up seeing f32? > Quantization is the process of mapping 32-bit floating-point numbers (which are the weights in the neural network) to a much smaller bit representation, like 4-bit values, for storage and memory efficiency. > Dequantization happens when the model is used (during inference or even training, if applicable). The 4-bit quantized weights are converted back into floating-point numbers that the model's computations are actually performed with. This is done using the scale and zero-point determined during the initial quantization, or through more sophisticated mapping functions that aim to preserve as much information as possible despite the reduced precision. so what is the relationship to "parameters" and "# of unique tokens the model knows about (vocabulary size)"? > At first glance, LLAMa only has a 32,000 vocabulary size and 65B parameters as compared to GPT-3, > The 65 billion parameters in a model like LLAMA (or any large language model) essentially function as a highly intricate mapping system that determines how to respond to a given input based on the learned relationships between tokens in its training data.

Filligree 3y ago
It doesn't, is the simple answer. The slightly more complicated one is that a compressed text dump of Wikipedia isn't even 70GB, and this is lossy compression of the internet.
- MuffinFlavored 3y ago
  say the average LLM these days has a unique token (vocabulary) size of ~32,000 (not its context size, # of unique tokens it can pick between in a response. English words, punctuation, math, code, etc.) the 60-70B parameters of models is basically like... just stored patterns of "if these 10 tokens in a row input, then these 10 tokens in a row output score the highest" Is that a good summary? > The model uses its learned statistical patterns to predict the probability of what comes next in a sequence of text. based on what inputs? 1. previous tokens in the sequence from immediate context 2. tokens summarizing the overall topic/subject matter from the extended context 3. scoring of learned patterns from training 4. what else?

estebarb 3y ago

I find this similar to what relation vectors do in word2vec: you can add a vector of "X of" and often get the correct answer. It could be that the principle is still the same, and transformers "just" build a better mapping of entities into the embedding space?

PaulHoule 3y ago
I think so. It’s hard for me to believe that the decision surfaces inside those models are really curved enough (like the folds of your brain) to really take advantage of FP32 numbers inside vectors: that is I just don’t believe it is x = 0 means “fly” x = 0.01 means “drive” x = 0.02 means “purple” but rather more like x < 1.5 means “cold” x > 1.5 means “hot” which is one reason why quantization (often 1 bit) works. Also it is a reason why you can often get great results feeding text or images through a BERT or CLIP-type model and then applying classical ML models that frequently involve linear decision surfaces.
- taneq 3y ago
  Are you conflating nonlinear embedding spaces with the physical curvature of the cerebellum? I don't think there's a direct mapping.
  - PaulHoule 3y ago
    My mental picture is that violently curved decision surfaces could look like the convolutions of the brain even though they have nothing to do with how the brain actually works. I think of how tSNE and other algorithms sometimes produce projections that sometimes look like that (maybe that’s just what you get when you have to bend something complicated to fit into a 2-d space) and frequently show cusps that to me look like a sign of trouble (took me a while in my PhD work to realize how Poincaré sections from 4 or 6 dimensions can look messed up when a part of the energy surface tilts perpendicularly to the projection surface.) I still find it hard to believe that dense vectors are the right way to deal with text despite the fact that they work so well. For images it is one thing because changing one pixel a little doesn’t change the meaning of an image, but changing a single character of a text can completely change the meaning of the text. Also there’s the reality that if you randomly stick together tokens you get something meaningless, so it seems almost all of the representation space covers ill formed texts and only a low dimensional manifold holds the well formed texts. Now the decision surfaces really have to be nonlinear and crumpled over all but I think there’s a definitely a limit on how crumpled those surfaces can be.

derefr 3y ago

Help me understand: when they say that the facts are stored as a linear function… are they saying that the LLM has a sort of N-dimensional “fact space” encoded into the model in some manner, where facts are embedded into the space as (points / hyperspheres / Voronoi manifolds / etc); and where recalling a fact is — at least in an abstract sense — the NN computing / remembering a key to use, and then doing a key-value lookup in this space? If so: how do you embed a KV-store into an edge-propagated graphical model? Are there even any well-known techniques for doing that “by hand” right now? (Also, fun tangent: isn't the "memory palace" memory technique, an example of human brains embedding facts into a linear function for easier retrieval?)

bionhoward 3y ago
[Layer] Normalization constrains huge vectors representing tokens (input fragments) to positions on a unit ball (I think), and the attention mechanism operates by rotating the unconstrained ones based on the sum of their angles relative to all the others. I only skimmed the paper but believe the point here is that there are relatively simple functions hiding in or recoverable from the bigger network which specifically address certain categories of relationships between concepts. Since it would, in theory, be possible to optimize such functions more directly if they are possible to isolate, could this enable advances in the way such models are trained? Absolutely. After all, one of the best criticisms of “modern” AI is the notion we’re just mixing around a soup of linear algebra. Allowing some sense of modularity (reductionism) could make them less of a black box and more of a component driven approach (in the lagging concept space and not just the leading layer space)
jacobn 3y ago
The fundamental operation done by the transformer, softmax(Q.K^T).V, is essentially a KV-store lookup. The Query is dotted with the Key, then you take the softmax to pick mostly one winning Key (the Key closest to the Query basically), and then use the corresponding Value. That is really, really close to a KV lookup, except it's a little soft (i.e. can hit multiple Keys), and it can be optimized using gradient descent style methods to find the suitable QKV mappings.
- 3y ago

wslh 3y ago

Can we roughly say that LLMs produces (training mode) a lot of IF-THENs in an automatic way from a vast quantity of information (nor techniques) that was not available before?

mike_hearn 3y ago

This is really cool. My mind goes immediately to what sort of functions are being used to encode programming knowledge, and if they are also simple linear functions whether the standard library or other libraries can be directly uploaded into an LLMs brain as it evolves, without needing to go through a costly training or performance-destroying fine-tune. That's still a sci-fi ability today but it seems to be getting closer.

Animats 3y ago
That's a good point. It may be possible to directly upload predicate-type info into a LLM. This could be especially useful if you need to encode tabular data. Somewhere, someone probably read this and is thinking about how to export Excel or databases to an LLM. It's encouraging to see people looking inside the black box successfully. The other big result in this area was that paper which found a representation of a game board inside a LLM after the LLM had trained to play a game. Any other good results in that area? The authors point out that LLMs are doing more than encoding predicate-type info. That's just part of what they are doing.
- AaronFriel 3y ago
  It indeed is. An attention mechanism's key and value matrices grow linearly with context length. With PagedAttention[1], we could imagine an external service providing context. The hard part is the how, of course. We can't load our entire database in every conversation, and I suspect there are also challenges around training (perhaps addressed via LandmarkAttention[2]) and building a service efficiently retrieve additional key-value matrices. The external service vector database may require tight timings necessary to avoid stalling LLMs. To manage 20-50 tokens/sec, answers must arrive within 50-20ms. And we cannot do this in real-time, pausing the transformer when a layer produces a query vector stalls the batch, so we need a way to predict queries (or embeddings) several tokens ahead of where they'd be useful and inject the context in when it's needed, and to know when to page it out. [1] https://arxiv.org/abs/2309.06180 https://arxiv.org/abs/2309.06180 [2] https://arxiv.org/abs/2305.16300 https://arxiv.org/abs/2305.16300
- wongarsu 3y ago

i5heu 3y ago

So it is entirely possible to decouple the reasoning part from the information part? This is like absolutely mind blowing if this is true.

learned 3y ago
A big caveat mentioned in the article is that this experiment was done with a small set (N=47) of specific questions that they expected to have relatively simple relational answers: > The researchers developed a method to estimate these simple functions, and then computed functions for 47 different relations, such as “capital city of a country” and “lead singer of a band.” While there could be an infinite number of possible relations, the researchers chose to study this specific subset because they are representative of the kinds of facts that can be written in this way. About 60% of these relations were retrieved using a linear function in the model. The remaining appeared to have nonlinear retrieval and is still a subject of investigation: > Functions retrieved the correct information more than 60 percent of the time, showing that some information in a transformer is encoded and retrieved in this way. “But not everything is linearly encoded. For some facts, even though the model knows them and will predict text that is consistent with these facts, we can’t find linear functions for them. This suggests that the model is doing something more intricate to store that information,” he says.

deleted 3y ago

[deleted]

retrofrost 3y ago

This is amazing work, but to me it highlights some of the biggest problems in the current AI zeitgeist, we are not really trying to work on any neuron or ruleset that isnt much different from the perceptron thats just a sumnation function. Is it really that suprising that we just see this same structure repeated in the models. Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? We have all sorts of unique training methods and encoding schemes that don't ever get used because the big libraries don't support them. Until, we start seeing real varation in the fundamental rulesets of neuralnets we are always just going to be fighting against the fact these are just perceptrons with extra steps.

visarga 3y ago
> Just because feedforward topologies with single neuron steps are the easiest to train and run on graphics cards does that really make them the actual best at accomplishing tasks? You are ignoring a mountain of papers trying all conceivable approaches to create models. It is evolution by selection, in the end transformers won.
- dartos 3y ago
  I mean RWKV seems promising and isn’t a transformer model. Transformers have first mover advantage. They were the first models that scaled to large parameter counts. That doesn’t mean they’re the best or that they’ve won, just that they were the first to get big (literally and metaphorically)
  - tkellogg 3y ago
    Yeah, I'd argue that transformers created such capital saturation that there's a ton of opportunity for alternative approaches to emerge.
    - dartos 3y ago
      Speak of the devil. Jamba just hit the front page.

zyklonix 3y ago

This reminds me of the famous "King - Man + Woman = Queen" embedding example. The fact that embeddings have semantic properties in them explains why simple linear functions would work as well.

robertclaus 3y ago

I think this paper is cool and I love that they ran these experiments to validate these ideas. However, I'm having trouble reconciling the novelty of the ideas themselves. Isn't this result expected given that LLM's naturally learn simple statistical trends between words? To me it's way cooler that they clearly demonstrated not all LLM behavior can be explained this simply.

mikewarot 3y ago

I wonder if this relation still holds with newer models that have have even more compute thrown at them? My intuition is that the structure inherent to language makes Word2Vec possible. Then training on terabytes of human text encoded with Word2Vec + Positional Encoding makes it possible to then have the ability to predict the next encoding at superhuman levels of cognition (while training!). It's my sense that the bag of words (as input/output method) combined with limited context windows (to make Positional Encoding work) is a huge impedance mismatch to the internal cognitive structure. Thus I think that given the orders of magnitude more compute thrown at GPT-4 et al, it's entirely possible new forms of representation evolved and remain to be discovered by humans probing through all the weights. I also think that MemGPT could, eventually, become an AGI because of the unlimited long term memory. More likely, though, I think it would be like the protagonist in Memento[1]. [1] https://en.wikipedia.org/wiki/Memento_(film) https://en.wikipedia.org/wiki/Memento_(film) [edit - revise to address question]

autokad 3y ago
sorry if I misread your comment, but you seem to be indicating that LLMs such as chat gpt (which use gpt 3+) are bag of words models? they are sequence models.
- mikewarot 3y ago
  I edited my response... I hope it helps... my understanding is that the output gives probabilities for all the words, then one is chosen with some random thrown in (via the #temperature) then fed back in... which to me seems to equate to bag of words. Perhaps I mis-understood the term.
  - smaddox 3y ago
    Bag of words models use a context that is a "bag" (i.e. an unorder map from elements to their counts) of words/tokens. GPT's use a context that is a sequence (i.e. an ordered list) of words/tokens.

uoaei 3y ago

This is the "random linear projections as memorization technique" perspective on Transformers. It's not a new idea per se, but nice to see it fleshed out. If you dig into this perspective, it does temper any claims of "cognitive behavior" quite strongly, if only because Transformers have such a large capacity for these kinds of "memories".

tel 3y ago
Do you have a reference on “random linear projections as memorization”? I know random projections quite well but haven’t seen that connection.

seydor 3y ago

Does this point to a way to compress entire LLMs by selecting a set of relations?