10 ms·

Language models are injective and hence invertible

231 points by mazsa 1y ago

luc_ 1y ago
I wonder how these pieces of understanding can be applied to neuroscience.
- tgv 1y ago
  Real brains have a very different structure and composition. Neurons aren't monotonic, there are loops, etc.
- raincole 1y ago
  These pieces of "understanding" mostly don't even applied to other LLMs.
- simiones 1y ago
  They can't. Biological neural networks have no resemblance to the artificial neural networks of the kind used in LLMs. The only similarity is based on a vague computational abstraction of the very primitive understanding of how brains and nerve cells worked we had in the 50s when the first "neural network" was invented.
fxwin 1y ago
I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606 https://x.com/GladiaLab/status/1983812121713418606
- jascha_eng 1y ago
  Yeh it would be fun if we could reverse engineer the prompts from auto generated blog posts. But this is not quite the case.
- lou1306 1y ago
  Still, it is technically correct. The model produces a next-token likelihood distribution, then you apply a sampling strategy to produce a sequence of tokens.
  - KeplerBoy 1y ago
    Depends on your definition of the model. Most people would be pretty upset with the usual LLM providers if they drastically changed the sampling strategy for the worse and claimed to not have changed the model at all.
    - blackbear_ 1y ago
      Tailoring the message to the audience is really a fundamental principle of good communication. Scientists and academics demand an entirely different level of rigor compared to customers of LLM providers.
      - KeplerBoy 1y ago
        Sure, but they went slightly overboard with that headline and they knew it. But oh well, they have a lot of eyes and discussion on their paper so it's a success.
        mattlutze 1y ago
        I feel like, if the feedback to your paper is "this is over-done / they claim more than they prove / it's kinda hype-ish" you're going to get less references in future papers. That would seem to be counter to the "impact" goal for research.
        KeplerBoy 1y ago
        Fair enough, that might be more my personal opinion instead of sound advice for successful research. Also I understand that you have a very limited amount of time to get your research noticed in this topic. Who knows if it's relevant two years down the line.
- weinzierl 1y ago
  Spoiler but for those who do not want to open the paper, the contribution statement is: "Equal contribution; author order settled via Mario Kart." If only more conflicts in life would be settled via Mario Kart.
  - Lord-Jobo 1y ago
    Envisioning our most-elderly world leaders throwing down in Mario Kart, fighting for 23rd and 24th place as they bump into walls over and over and struggle to hold their controllers properly… well it’s a very pleasant thought.
- causal 1y ago
  But I bet you could reconstruct a plausible set of distributions by just rerunning the autoregression on a given text with the same model. You won't invert the exact prompt but it could give you a useful approximation.
kobelb 1y ago
Are the weights invertible, or are the prompts being fed into the model invertible?
- singularity2001 1y ago
  The prompt is reconstructable from the (latent) representation vector but obviously not from the output
sigmoid10 1y ago
>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny patch on the surface of the unit sphere. Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small (like less than one in 10^10000 or something). Even worse than your chances of finding a hash collision in sha256. Their claim certainly does not sound like something you could verify by testing a few billion examples. Although I'd love to see a detailed calculation. The paper is certainly missing one.
- sebastianmestre 1y ago
  I think that the latent space that GPT-2 uses has 768 dimensions (i.e. embedding vectors have that many components).
  - sigmoid10 1y ago
    It doesn't really matter which vector you are looking at, since they are using a tiny constraint in a high dimensional continuous space. There's gotta be an unfathomable amount of vectors you can fit in there. Certainly more than a few billion.
    - sebastianmestre 1y ago
      No, yeah, totally. Even assuming binary vectors 2^768 is a ridiculously huge number. The probability of collision even assuming a bad sampling that discards 75% of dimensions is still vanishingly small.
- gnfargbl 1y ago
  The nature of high-dimensional spaces kind of intuitively supports the argument for invertability though, no? In the sense that: > I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small.
fatherrhyme 1y ago
Am I misunderstanding this? Any stateful system that exposes state in a flexible way has risk to data exposure. Does anyone actually think a stateful system wouldn’t release state? Why not just write a paper “The sky may usually be blue”?
- MattRix 1y ago
  yes, you’re misunderstanding it
patrick0d 1y ago
paper looks nice! i think what they found was that they can recover the input sequence by trying all tokens from the vocab and finding a unique state. they do a forward pass to check each possible token at a given depth. i think this is since the model will encode the sequence in the mid flight token so this encoding is revealed to be unique by their paper. so one prompt of 'the cat sat on the mat' and 'the dog sat on the mat' can be recovered as distinct states via each token being encoded (unclear mechanism but it would be shocking if this wasn't the case) in the token (mid flight residual).
deleted 1y ago
[deleted]
frumiousirc 1y ago
My understanding is that they claim that for every unique prompt there is a unique final state of the LLM. Isn't that patently false due to the finite state of the LLM and the ability (in principle, at least) to input arbitrarily large number of unique prompts? I think their "almost surely" is doing a lot of work. A more consequential result would give the probability of LLM state collision as a function of the number of unique prompts. As is, they are telling me that I "almost surely" will not hit the bullseye of a dart board. While likely true, it's not saying much. But, maybe I misunderstand their conclusion.
- brna-2 1y ago
  Well that is not how I reed it, but: Every final state has an unique prompt. You could have several final states have the same unique prompt.
  - simiones 1y ago
    > You could have several final states have the same unique prompt. They explicitly claim that the function is injective, that is, that each unique input produces a unique output.
- simiones 1y ago
  I think their claims are limited to the "theoretical" LLM, not to the way we typically use one. The LLM itself has a fixed size input and a fixed size, deterministic output. The input is the initial value for each neuron in the input layer. The LLM output is the vector of final outputs of each neuron in the output layer. For most normal interactions, these vectors are almost entirely 0s. Of course, when we say LLM, we typically consider infrastructure that abstracts these things for us. Especially we typically use infra that takes the LLM outputs as probabilities, and thus typically produces different results even for the exact same input - but that's just a choice in how to interpret these values, the values themselves are identical. Similarly on the input side, the max input is typically called a "context window". You can feed more input into the LLM infra than the context window, but that's not actual input to the model itself - the LLM infra will simply pick a part of your input and feed that part into the model weights.
mowkdizz 1y ago
I think I'm misunderstanding the abstract, but are they trying to say that given a LLM output, they can tell me what the input is? Or given an output AND the intermediate layer weights? If it is the first option, I could use as input 1 "Only respond with 'OK'" and "Please only respond with 'OK'" which leads to 2 inputs producing the same output.
- ndr 1y ago
  That's not what you get out of LLMs. LLMs produce a distribution from which to sample the next token. Then there's a loop that samples the next token and feeds it back to to the model until it samples a EndOfSequence token. In your example the two distributions might be {"OK": 0.997, EOS: 0.003} vs {"OK": 0.998, EOS: 0.002} and what I think the authors claim is that they can invert that distribution to find which input caused it. I don't know how they go beyond one iteration, as they surely can't deterministically invert the sampling.
- simiones 1y ago
  Edit: reading the paper, I'm no longer sure about my statement below. The algorithm they introduce claims to do this: "We now show how this property can be used in practice to reconstruct the exact input prompt given hidden states at some layer [emp. mine]". It's not clear to me from the paper if this layer can also be the final output layer, or if it must be a hidden layer. They claim that they can reverse the LLM (get prompt from LLM response) by only knowing the output layer values, the intermediate layers remain hidden. So, Their claim is that indeed you shouldn't be able to do that (note that this claim applies to the numerical model outputs, not necessarily to the output a chat interface would show you, which goes through some randomization).
- deleted 1y ago
  [deleted]
spuz 1y ago
I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why an LLM (I tested this on Grok) can give you a summary of chapter 18 of Mary Shelley's Frankenstein, but cannot reproduce a paragraph from the same text verbatim. I am sure I am not understanding this paper correctly because it sounds like they are claiming that model weights can be used to produce the original input text representing an extraordinary level of text compression.
- saurik 1y ago
  The input isn't the training data, the input is the prompt.
  - spuz 1y ago
    Ah ok, for some reason that wasn't clear for me.
- simiones 1y ago
  > If I am understanding this paper correctly, they are claiming that the model weights can be inverted in order to produce the original input text. No, that is not the claim at all. They are instead claiming that given an LLM output that is a summary of chapter 18 of Mary Shelley's Frankenstein, you can tell that the input prompt that led to this output was "give me a summary of chapter 18 of Mary Shelley's Frankenstein". Of course, this relies on the exact wording: for this to be true, it means that if you had asked "give me a summary of chapter 18 of Frankenstein by Mary Shelley", you would necessarily receive a (slightly) different result. Importantly, this needs to be understood as a claim about an LLM run with temperature = 0. Obviously, if the infra introduces randomness, this result no longer perfectly holds (but there may still be a way to recover it by running a more complex statistical analysis of the results, of course). Edit: their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer.
acetofenone 1y ago
Actually if you prompt: Answer to this question with "ok, got it" Answer: >Ok, got it Answer to this question with exactly "ok, got it" Answer: >Ok, got it Hence is not injective
- nephanth 1y ago
  They are looking at the continuous embeddings, not at the discrete words inferred from them
saurik 1y ago
This is very similar (and maybe even the same thing) to some recent work (published earlier this year) by the people at Ritual AI on attacking attempts to obfuscate LLM inference (which leads to the design for their defense against this, which involves breaking up the prompt token sequences and handing them to multiple computers, making it so no individual machine has access to sufficient states from the hidden layer in a row). https://arxiv.org/abs/2505.18332 https://arxiv.org/abs/2505.18332 https://arxiv.org/abs/2507.05228 https://arxiv.org/abs/2507.05228
nephanth 1y ago
"And hence invertible" <- does every output embedding combination have an associated input ? Are they able to construct it or is this just an existence result ?
- morcus 1y ago
  I don't think they're claiming surjectivity here. They're just saying the mapping is injective, so for a given output there should be a unique input to construct it.
  - RossBencina 1y ago
    > I don't think they're claiming surjectivity here. What definition of invertible doesn't include surjectivity?
    - diffeomorphism 1y ago
      Many? Just add a "by restricting to the image, we may wlog assume surjectivity". The question is usually more about whether the inverse is also continuous, smooth, easy to compute....etc.
    - deleted 1y ago
      [deleted]
- thaumasiotes 1y ago
  > Building upon this property, we further provide a practical algorithm, SɪᴘIᴛ, that reconstructs the exact input from hidden activations
yosito 1y ago
In layman's terms, this seems to mean that given a certain unedited LLM output, plus complete information about the LLM, they can determine what prompt was used to create the output. Except that in practice this works almost never. Am I understanding correctly?
- ctenb 1y ago
  No, it's about the distribution being injective, not a single sampled response. So you need a lot of outputs of the same prompt, and know the LLM, and then you should in theory be able to reconstruct the original prompt.
  - deleted 1y ago
    [deleted]
- eapriv 1y ago
  No, it says nothing about LLM output being invertible.
stared 1y ago
It reminded me of "Text embeddings reveal almost as much as text" from 2023 (https://news.ycombinator.com/item?id=37867635 https://news.ycombinator.com/item?id=37867635) - and yes, they do cite it. It has a huge implication for privacy. There is some "mental model" that embedding vectors are like hash - so you can store them in database, even though you would not store plain text. It is an incorrect assumption - as a good embedding stores ALL - not just the general gist, but dates, names, passwords. There is an easy fix to that - a random rotation; preserves all distances.
- FeepingCreature 1y ago
  This mental model is also in direct contradiction to the whole purpose of the embedding, which is that the embedding describes the original text in a more interpretable form. If a piece of content in the original can be used for search, comparison etc., p much by definition it has to be stored in the embedding. Similarly, this result can be rephrased as "Language Models process text." If the LLM wasn't invertible with regards to a piece of input text, it couldn't attend to this text either.
- gowld 1y ago
  > There is an easy fix to that - a random rotation; preserves all distances. Is that like homomorphic encryption, in a sense, where you can calculate the encryption of a function on the plaintext, without ever seeing the input or calculated function of plaintext.
adamddev1 1y ago
Could this be a way to check for AI plagiarism? Given a chunk of text would you be able to (almost) prove that it came from a prompt saying "Write me a short essay on ___" ?
- ape4 1y ago
  Or maybe paste the essay into the question and ask what prompt was used to produce it.
- pu_pe 1y ago
  I first had this impression as well, but I can think of several complications: 1) You would have to know the exact model used, and you would need to have access to their weights 2) System prompt and temperature, not sure how this handles those 3) If anyone changes even a word in the output, this method would fall apart
Marshferm 1y ago
Quoting Timos Moraitis a Neuromorphic PhD "For reasons like this, "in-context learning" is not an accurate term for transformers. It's projection and storage, nothing is learnt. This new paper has attracted a lot of interest, and it's nice that it proves things formally and empirically, but it looks like people are surprised by it, even though it was clear." https://x.com/timos_m/status/1983625714202010111 https://x.com/timos_m/status/1983625714202010111
RA_Fisher 1y ago
This claim's so big that it requires theoretical proof, empirical analysis isn't convincing (given the size of the claim). Causal inference experts have long known that many inputs map to outputs (that's why identification of the inputs that actually caused a given output is a never-ending task).
spacecadet 1y ago
I find this interesting. I have tools that attempt to reverse engineer black box models through auto-prompting and analysis of the outputs/tokens. I have used this to develop prompt injection attacks that "steer" output, but have never tried to use the data to recreate an exact input...
usgroup 1y ago
Injective doesn’t mean bijective, and that seems obvious. That is, presumably very many inputs will map to the output “Yes”.
- fxwin 1y ago
  Afaict surjectivity was already a given before this paper, their contribution is the injectivity part (and thus invertibility)
ghm2180 1y ago
There is actually a good analytical result on how vector similarity can easily fail to recover relevant information https://arxiv.org/pdf/2403.05440 https://arxiv.org/pdf/2403.05440 > For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization. I am not strong in mathematics but if this paper claims run opposite to each other.
realitydrift 1y ago
[flagged]
CGMthrowaway 1y ago
Summary from the authors: -Different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space - Injectivity is not accidental, but a structural property of language models - Across billions of prompt pairs and several model sizes, we find no collisions: no two prompts are mapped to the same hidden states - We introduce SipIt, an algorithm that exactly reconstructs the input from hidden states in guaranteed linear time. - This impacts privacy, deletion, and compliance: once data enters a Transformer, it remains recoverable.
- orbital-decay 1y ago
  > - This impacts privacy, deletion, and compliance Surely that's a stretch... Typically, the only thing that leaves a transformer is its output text, which cannot be used to recover the input.
  - Kostchei 1y ago
    remains recoverable... for less than a training run of compute .It's a lot, but it is doable
    - orbital-decay 1y ago
      Here's an output text: "Yes." Recover the exact input that led to it. (you can't, because the hidden state is already irreversibly collapsed during the sampling of each token) The paper doesn't claim this to be possible either, they prove the reversibility of the mapping between the input and the hidden state, not the output text. Or rather "near-reversibility", i.e. collisions are technically possible but they have to be very precisely engineered during the model training and don't normally happen.
      - Nydhal 1y ago
        if you generate a lot of output text you can approximate the hidden state.
  - 1y ago
davidhyde 1y ago
42 In less than 7.5 million years please. https://simple.wikipedia.org/wiki/42_(answer) https://simple.wikipedia.org/wiki/42_(answer)
D-Machine 1y ago
Authors: Giorgos Nikolaou‡*, Tommaso Mencattini†‡*, Donato Crisostomi†, Andrea Santilli†, Yannis Panagakis§¶, Emanuele Rodolà† †Sapienza University of Rome ‡EPFL §University of Athens ¶Archimedes RC *Equal contribution; author order settled via Mario Kart.
WhitneyLand 1y ago
tldr: Seeing what happens internally in an LLM lets you reconstruct the original prompt exactly. Maybe not surprising if you logged all internal activity, but it can be done from only a single snapshot of hidden activations from the standard forward pass.
mattfinlayson 1y ago
Author of related work here. This is very cool! I was hoping that they would try to invert layer by layer from the output to the input but it seems that they do a search process at the input layer instead. They rightly point out the residual connections make a layer by layer approach difficult. I may point out though that an rmsnorm layer should be invertible due to the epsilon term in the denominator which can be used to recover the input magnitude
- causal 1y ago
  What is meant by "residual connections" here?
csense 1y ago
A few critiques: - If you have a feature detector function (f(x) = 0 when feature is not present, f(x) = 1 when feature is present) and you train a network to compute f(x), or some subset of the network "decides on its own during training" to compute f(x), doesn't that create a zero set of non-zero measure if training continues long enough? - What happens when the middle layers are of much lower dimension than the input? - Real analyticity means infinitely many derivatives (according to Appendix A). Does this mean the results don't apply to functions with corners (e.g. ReLU)?
TZubiri 1y ago
Isn't a requirement for injectivity that each input must map to 1 output? Where LLMs can result in the same output given multiple different inputs?
breppp 1y ago
Does that mean that all these embeddings in all those vector databases can be used to extract all these secret original documents?
- sidnb13 1y ago
  I don't think vector databases are intended to be secure, encrypted forms of data storage in the first place.
skybrian 1y ago
I'm wondering how this might be summarized in simple terms? It sounds like, after processing some text, the entire prompt is included in the in-memory internal state of the program that's doing inference. But it seems like it would need to remember the prompt to answer questions about it. How does this interact with the attention mechanism?
- krackers 1y ago
  I wonder why this is such a surprise, this is in fact what you would naively expect given the way the residual stream is structured no? Each attention block adds to the residual stream. And we already know from logit-lens type work that the residual stream roughly remains in the same "basis" [1], which I vaguely remember is something that resnet architectures explicitly try to achieve. So maybe it's my armchair naivety but in order for both of these to hold while the LLM being able to do some sort of "abstraction", it seems like it is natural for the initial token embedding to be projected into some high-dimensional subspace and then as it passes through different attention blocks you get added "deltas" on top of that, filling out other parts of that subspace. And looking at the overall attention network from an information passing perspective, encoding and having access to the input tokens is certainly a nice thing to have. Now maybe it could be argued that the original input could be lossily "converted" into some other more abstract representation, but if the latent space is large enough to not force this, then there's probably no strict reason to do so. And in fact we do know that traditional BPE token embeddings don't even form a subspace (there's a fixed vocab size and embeddings are just a lookup table, so it's only just a bunch of scattered points). I wonder if this work is repeated with something like vision tokens, whether you will get the same results. [1] https://nitter.poast.org/khoomeik/status/1920620258353610896 https://nitter.poast.org/khoomeik/status/1920620258353610896
carruc 1y ago
[dead]
danielmarkbruce 1y ago
tldr ~ for dense decoder‑only transformers, the last‑token hidden state almost certainly identifies the input and you can invert it in practice from internal activations..
kouru225 1y ago
Wait so is it possible to pass a message using AI and does this matter? Like let’s imagine I have an AI model and only me and my friend have it. I write a prompt and get back the vectors only. No actual output. Then I send my friend those vectors and they use this algorithm to reconstruct my message at the endpoint. Does this method of messaging protect against a MITM attack? Can this be used in cryptography?
vhxs 1y ago
Did the authors post code for their algorithm anywhere?

231 points by mazsa 1y ago

luc_ 1y ago

I wonder how these pieces of understanding can be applied to neuroscience.

tgv 1y ago
Real brains have a very different structure and composition. Neurons aren't monotonic, there are loops, etc.
raincole 1y ago
These pieces of "understanding" mostly don't even applied to other LLMs.
simiones 1y ago
They can't. Biological neural networks have no resemblance to the artificial neural networks of the kind used in LLMs. The only similarity is based on a vague computational abstraction of the very primitive understanding of how brains and nerve cells worked we had in the 50s when the first "neural network" was invented.

fxwin 1y ago

I don't like the title of this paper, since most people in this space probably think of language models not as producing a distribution (wrt which they are indeed invertible, which is what the paper claims) but as producing tokens (wrt which they are not invertible [0]). Also the author contribution statement made me laugh. [0] https://x.com/GladiaLab/status/1983812121713418606 https://x.com/GladiaLab/status/1983812121713418606

jascha_eng 1y ago
Yeh it would be fun if we could reverse engineer the prompts from auto generated blog posts. But this is not quite the case.
lou1306 1y ago
Still, it is technically correct. The model produces a next-token likelihood distribution, then you apply a sampling strategy to produce a sequence of tokens.
- KeplerBoy 1y ago
  Depends on your definition of the model. Most people would be pretty upset with the usual LLM providers if they drastically changed the sampling strategy for the worse and claimed to not have changed the model at all.
  - blackbear_ 1y ago
    Tailoring the message to the audience is really a fundamental principle of good communication. Scientists and academics demand an entirely different level of rigor compared to customers of LLM providers.
    - KeplerBoy 1y ago
      Sure, but they went slightly overboard with that headline and they knew it. But oh well, they have a lot of eyes and discussion on their paper so it's a success.
      - mattlutze 1y ago
        I feel like, if the feedback to your paper is "this is over-done / they claim more than they prove / it's kinda hype-ish" you're going to get less references in future papers. That would seem to be counter to the "impact" goal for research.
        KeplerBoy 1y ago
        Fair enough, that might be more my personal opinion instead of sound advice for successful research. Also I understand that you have a very limited amount of time to get your research noticed in this topic. Who knows if it's relevant two years down the line.
weinzierl 1y ago
Spoiler but for those who do not want to open the paper, the contribution statement is: "Equal contribution; author order settled via Mario Kart." If only more conflicts in life would be settled via Mario Kart.
- Lord-Jobo 1y ago
  Envisioning our most-elderly world leaders throwing down in Mario Kart, fighting for 23rd and 24th place as they bump into walls over and over and struggle to hold their controllers properly… well it’s a very pleasant thought.
causal 1y ago
But I bet you could reconstruct a plausible set of distributions by just rerunning the autoregression on a given text with the same model. You won't invert the exact prompt but it could give you a useful approximation.

kobelb 1y ago

Are the weights invertible, or are the prompts being fed into the model invertible?

singularity2001 1y ago
The prompt is reconstructable from the (latent) representation vector but obviously not from the output

sigmoid10 1y ago

>we confirm this result empirically through billions of collision tests on six state-of-the-art language models, and observe no collisions This sounds like a mistake. They used (among others) GPT2, which has pretty big space vectors. They also kind of arbitrarily define a collision threshold as an l2 distance smaller than 10^-6 for two vectors. Since the outputs are normalized, that corresponds to a ridiculously tiny patch on the surface of the unit sphere. Just intuitively, in such a high dimensional space, two random vectors are basically orthogonal. I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small (like less than one in 10^10000 or something). Even worse than your chances of finding a hash collision in sha256. Their claim certainly does not sound like something you could verify by testing a few billion examples. Although I'd love to see a detailed calculation. The paper is certainly missing one.

sebastianmestre 1y ago
I think that the latent space that GPT-2 uses has 768 dimensions (i.e. embedding vectors have that many components).
- sigmoid10 1y ago
  It doesn't really matter which vector you are looking at, since they are using a tiny constraint in a high dimensional continuous space. There's gotta be an unfathomable amount of vectors you can fit in there. Certainly more than a few billion.
  - sebastianmestre 1y ago
    No, yeah, totally. Even assuming binary vectors 2^768 is a ridiculously huge number. The probability of collision even assuming a bad sampling that discards 75% of dimensions is still vanishingly small.
gnfargbl 1y ago
The nature of high-dimensional spaces kind of intuitively supports the argument for invertability though, no? In the sense that: > I would expect the chance of two inputs to map to the same output under these constraints to be astronomically small.

fatherrhyme 1y ago

Am I misunderstanding this? Any stateful system that exposes state in a flexible way has risk to data exposure. Does anyone actually think a stateful system wouldn’t release state? Why not just write a paper “The sky may usually be blue”?

MattRix 1y ago
yes, you’re misunderstanding it

patrick0d 1y ago

paper looks nice! i think what they found was that they can recover the input sequence by trying all tokens from the vocab and finding a unique state. they do a forward pass to check each possible token at a given depth. i think this is since the model will encode the sequence in the mid flight token so this encoding is revealed to be unique by their paper. so one prompt of 'the cat sat on the mat' and 'the dog sat on the mat' can be recovered as distinct states via each token being encoded (unclear mechanism but it would be shocking if this wasn't the case) in the token (mid flight residual).

deleted 1y ago

[deleted]

frumiousirc 1y ago

My understanding is that they claim that for every unique prompt there is a unique final state of the LLM. Isn't that patently false due to the finite state of the LLM and the ability (in principle, at least) to input arbitrarily large number of unique prompts? I think their "almost surely" is doing a lot of work. A more consequential result would give the probability of LLM state collision as a function of the number of unique prompts. As is, they are telling me that I "almost surely" will not hit the bullseye of a dart board. While likely true, it's not saying much. But, maybe I misunderstand their conclusion.

brna-2 1y ago
Well that is not how I reed it, but: Every final state has an unique prompt. You could have several final states have the same unique prompt.
- simiones 1y ago
  > You could have several final states have the same unique prompt. They explicitly claim that the function is injective, that is, that each unique input produces a unique output.
simiones 1y ago
I think their claims are limited to the "theoretical" LLM, not to the way we typically use one. The LLM itself has a fixed size input and a fixed size, deterministic output. The input is the initial value for each neuron in the input layer. The LLM output is the vector of final outputs of each neuron in the output layer. For most normal interactions, these vectors are almost entirely 0s. Of course, when we say LLM, we typically consider infrastructure that abstracts these things for us. Especially we typically use infra that takes the LLM outputs as probabilities, and thus typically produces different results even for the exact same input - but that's just a choice in how to interpret these values, the values themselves are identical. Similarly on the input side, the max input is typically called a "context window". You can feed more input into the LLM infra than the context window, but that's not actual input to the model itself - the LLM infra will simply pick a part of your input and feed that part into the model weights.

mowkdizz 1y ago

I think I'm misunderstanding the abstract, but are they trying to say that given a LLM output, they can tell me what the input is? Or given an output AND the intermediate layer weights? If it is the first option, I could use as input 1 "Only respond with 'OK'" and "Please only respond with 'OK'" which leads to 2 inputs producing the same output.

ndr 1y ago
That's not what you get out of LLMs. LLMs produce a distribution from which to sample the next token. Then there's a loop that samples the next token and feeds it back to to the model until it samples a EndOfSequence token. In your example the two distributions might be {"OK": 0.997, EOS: 0.003} vs {"OK": 0.998, EOS: 0.002} and what I think the authors claim is that they can invert that distribution to find which input caused it. I don't know how they go beyond one iteration, as they surely can't deterministically invert the sampling.
simiones 1y ago
Edit: reading the paper, I'm no longer sure about my statement below. The algorithm they introduce claims to do this: "We now show how this property can be used in practice to reconstruct the exact input prompt given hidden states at some layer [emp. mine]". It's not clear to me from the paper if this layer can also be the final output layer, or if it must be a hidden layer. They claim that they can reverse the LLM (get prompt from LLM response) by only knowing the output layer values, the intermediate layers remain hidden. So, Their claim is that indeed you shouldn't be able to do that (note that this claim applies to the numerical model outputs, not necessarily to the output a chat interface would show you, which goes through some randomization).
deleted 1y ago
[deleted]

spuz 1y ago

I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why an LLM (I tested this on Grok) can give you a summary of chapter 18 of Mary Shelley's Frankenstein, but cannot reproduce a paragraph from the same text verbatim. I am sure I am not understanding this paper correctly because it sounds like they are claiming that model weights can be used to produce the original input text representing an extraordinary level of text compression.

saurik 1y ago
The input isn't the training data, the input is the prompt.
- spuz 1y ago
  Ah ok, for some reason that wasn't clear for me.
simiones 1y ago
> If I am understanding this paper correctly, they are claiming that the model weights can be inverted in order to produce the original input text. No, that is not the claim at all. They are instead claiming that given an LLM output that is a summary of chapter 18 of Mary Shelley's Frankenstein, you can tell that the input prompt that led to this output was "give me a summary of chapter 18 of Mary Shelley's Frankenstein". Of course, this relies on the exact wording: for this to be true, it means that if you had asked "give me a summary of chapter 18 of Frankenstein by Mary Shelley", you would necessarily receive a (slightly) different result. Importantly, this needs to be understood as a claim about an LLM run with temperature = 0. Obviously, if the infra introduces randomness, this result no longer perfectly holds (but there may still be a way to recover it by running a more complex statistical analysis of the results, of course). Edit: their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer.

acetofenone 1y ago

Actually if you prompt: Answer to this question with "ok, got it" Answer: >Ok, got it Answer to this question with exactly "ok, got it" Answer: >Ok, got it Hence is not injective

nephanth 1y ago
They are looking at the continuous embeddings, not at the discrete words inferred from them

saurik 1y ago

This is very similar (and maybe even the same thing) to some recent work (published earlier this year) by the people at Ritual AI on attacking attempts to obfuscate LLM inference (which leads to the design for their defense against this, which involves breaking up the prompt token sequences and handing them to multiple computers, making it so no individual machine has access to sufficient states from the hidden layer in a row). https://arxiv.org/abs/2505.18332 https://arxiv.org/abs/2505.18332 https://arxiv.org/abs/2507.05228 https://arxiv.org/abs/2507.05228

nephanth 1y ago

"And hence invertible" <- does every output embedding combination have an associated input ? Are they able to construct it or is this just an existence result ?

morcus 1y ago
I don't think they're claiming surjectivity here. They're just saying the mapping is injective, so for a given output there should be a unique input to construct it.
- RossBencina 1y ago
  > I don't think they're claiming surjectivity here. What definition of invertible doesn't include surjectivity?
  - diffeomorphism 1y ago
    Many? Just add a "by restricting to the image, we may wlog assume surjectivity". The question is usually more about whether the inverse is also continuous, smooth, easy to compute....etc.
  - deleted 1y ago
    [deleted]
thaumasiotes 1y ago
> Building upon this property, we further provide a practical algorithm, SɪᴘIᴛ, that reconstructs the exact input from hidden activations

yosito 1y ago

In layman's terms, this seems to mean that given a certain unedited LLM output, plus complete information about the LLM, they can determine what prompt was used to create the output. Except that in practice this works almost never. Am I understanding correctly?

ctenb 1y ago
No, it's about the distribution being injective, not a single sampled response. So you need a lot of outputs of the same prompt, and know the LLM, and then you should in theory be able to reconstruct the original prompt.
- deleted 1y ago
  [deleted]
eapriv 1y ago
No, it says nothing about LLM output being invertible.

stared 1y ago

It reminded me of "Text embeddings reveal almost as much as text" from 2023 (https://news.ycombinator.com/item?id=37867635 https://news.ycombinator.com/item?id=37867635) - and yes, they do cite it. It has a huge implication for privacy. There is some "mental model" that embedding vectors are like hash - so you can store them in database, even though you would not store plain text. It is an incorrect assumption - as a good embedding stores ALL - not just the general gist, but dates, names, passwords. There is an easy fix to that - a random rotation; preserves all distances.

FeepingCreature 1y ago
This mental model is also in direct contradiction to the whole purpose of the embedding, which is that the embedding describes the original text in a more interpretable form. If a piece of content in the original can be used for search, comparison etc., p much by definition it has to be stored in the embedding. Similarly, this result can be rephrased as "Language Models process text." If the LLM wasn't invertible with regards to a piece of input text, it couldn't attend to this text either.
gowld 1y ago
> There is an easy fix to that - a random rotation; preserves all distances. Is that like homomorphic encryption, in a sense, where you can calculate the encryption of a function on the plaintext, without ever seeing the input or calculated function of plaintext.

adamddev1 1y ago

Could this be a way to check for AI plagiarism? Given a chunk of text would you be able to (almost) prove that it came from a prompt saying "Write me a short essay on ___" ?

ape4 1y ago
Or maybe paste the essay into the question and ask what prompt was used to produce it.
pu_pe 1y ago
I first had this impression as well, but I can think of several complications: 1) You would have to know the exact model used, and you would need to have access to their weights 2) System prompt and temperature, not sure how this handles those 3) If anyone changes even a word in the output, this method would fall apart

Marshferm 1y ago

Quoting Timos Moraitis a Neuromorphic PhD "For reasons like this, "in-context learning" is not an accurate term for transformers. It's projection and storage, nothing is learnt. This new paper has attracted a lot of interest, and it's nice that it proves things formally and empirically, but it looks like people are surprised by it, even though it was clear." https://x.com/timos_m/status/1983625714202010111 https://x.com/timos_m/status/1983625714202010111

RA_Fisher 1y ago

This claim's so big that it requires theoretical proof, empirical analysis isn't convincing (given the size of the claim). Causal inference experts have long known that many inputs map to outputs (that's why identification of the inputs that actually caused a given output is a never-ending task).

spacecadet 1y ago

I find this interesting. I have tools that attempt to reverse engineer black box models through auto-prompting and analysis of the outputs/tokens. I have used this to develop prompt injection attacks that "steer" output, but have never tried to use the data to recreate an exact input...

usgroup 1y ago

Injective doesn’t mean bijective, and that seems obvious. That is, presumably very many inputs will map to the output “Yes”.

fxwin 1y ago
Afaict surjectivity was already a given before this paper, their contribution is the injectivity part (and thus invertibility)

ghm2180 1y ago

There is actually a good analytical result on how vector similarity can easily fail to recover relevant information https://arxiv.org/pdf/2403.05440 https://arxiv.org/pdf/2403.05440 > For some linear models the similarities are not even unique, while for others they are implicitly controlled by the regularization. I am not strong in mathematics but if this paper claims run opposite to each other.

realitydrift 1y ago

[flagged]

CGMthrowaway 1y ago

Summary from the authors: -Different prompts always map to different embeddings, and this property can be used to recover input tokens from individual embeddings in latent space - Injectivity is not accidental, but a structural property of language models - Across billions of prompt pairs and several model sizes, we find no collisions: no two prompts are mapped to the same hidden states - We introduce SipIt, an algorithm that exactly reconstructs the input from hidden states in guaranteed linear time. - This impacts privacy, deletion, and compliance: once data enters a Transformer, it remains recoverable.

orbital-decay 1y ago
> - This impacts privacy, deletion, and compliance Surely that's a stretch... Typically, the only thing that leaves a transformer is its output text, which cannot be used to recover the input.
- Kostchei 1y ago
  remains recoverable... for less than a training run of compute .It's a lot, but it is doable
  - orbital-decay 1y ago
    Here's an output text: "Yes." Recover the exact input that led to it. (you can't, because the hidden state is already irreversibly collapsed during the sampling of each token) The paper doesn't claim this to be possible either, they prove the reversibility of the mapping between the input and the hidden state, not the output text. Or rather "near-reversibility", i.e. collisions are technically possible but they have to be very precisely engineered during the model training and don't normally happen.
    - Nydhal 1y ago
      if you generate a lot of output text you can approximate the hidden state.
- 1y ago

davidhyde 1y ago

42 In less than 7.5 million years please. https://simple.wikipedia.org/wiki/42_(answer) https://simple.wikipedia.org/wiki/42_(answer)

D-Machine 1y ago

Authors: Giorgos Nikolaou‡*, Tommaso Mencattini†‡*, Donato Crisostomi†, Andrea Santilli†, Yannis Panagakis§¶, Emanuele Rodolà† †Sapienza University of Rome ‡EPFL §University of Athens ¶Archimedes RC *Equal contribution; author order settled via Mario Kart.

WhitneyLand 1y ago

tldr: Seeing what happens internally in an LLM lets you reconstruct the original prompt exactly. Maybe not surprising if you logged all internal activity, but it can be done from only a single snapshot of hidden activations from the standard forward pass.

mattfinlayson 1y ago

Author of related work here. This is very cool! I was hoping that they would try to invert layer by layer from the output to the input but it seems that they do a search process at the input layer instead. They rightly point out the residual connections make a layer by layer approach difficult. I may point out though that an rmsnorm layer should be invertible due to the epsilon term in the denominator which can be used to recover the input magnitude

causal 1y ago
What is meant by "residual connections" here?

csense 1y ago

A few critiques: - If you have a feature detector function (f(x) = 0 when feature is not present, f(x) = 1 when feature is present) and you train a network to compute f(x), or some subset of the network "decides on its own during training" to compute f(x), doesn't that create a zero set of non-zero measure if training continues long enough? - What happens when the middle layers are of much lower dimension than the input? - Real analyticity means infinitely many derivatives (according to Appendix A). Does this mean the results don't apply to functions with corners (e.g. ReLU)?

TZubiri 1y ago

Isn't a requirement for injectivity that each input must map to 1 output? Where LLMs can result in the same output given multiple different inputs?

breppp 1y ago

Does that mean that all these embeddings in all those vector databases can be used to extract all these secret original documents?

sidnb13 1y ago
I don't think vector databases are intended to be secure, encrypted forms of data storage in the first place.

skybrian 1y ago

I'm wondering how this might be summarized in simple terms? It sounds like, after processing some text, the entire prompt is included in the in-memory internal state of the program that's doing inference. But it seems like it would need to remember the prompt to answer questions about it. How does this interact with the attention mechanism?

krackers 1y ago
I wonder why this is such a surprise, this is in fact what you would naively expect given the way the residual stream is structured no? Each attention block adds to the residual stream. And we already know from logit-lens type work that the residual stream roughly remains in the same "basis" [1], which I vaguely remember is something that resnet architectures explicitly try to achieve. So maybe it's my armchair naivety but in order for both of these to hold while the LLM being able to do some sort of "abstraction", it seems like it is natural for the initial token embedding to be projected into some high-dimensional subspace and then as it passes through different attention blocks you get added "deltas" on top of that, filling out other parts of that subspace. And looking at the overall attention network from an information passing perspective, encoding and having access to the input tokens is certainly a nice thing to have. Now maybe it could be argued that the original input could be lossily "converted" into some other more abstract representation, but if the latent space is large enough to not force this, then there's probably no strict reason to do so. And in fact we do know that traditional BPE token embeddings don't even form a subspace (there's a fixed vocab size and embeddings are just a lookup table, so it's only just a bunch of scattered points). I wonder if this work is repeated with something like vision tokens, whether you will get the same results. [1] https://nitter.poast.org/khoomeik/status/1920620258353610896 https://nitter.poast.org/khoomeik/status/1920620258353610896

carruc 1y ago

[dead]

danielmarkbruce 1y ago

tldr ~ for dense decoder‑only transformers, the last‑token hidden state almost certainly identifies the input and you can invert it in practice from internal activations..

kouru225 1y ago

Wait so is it possible to pass a message using AI and does this matter? Like let’s imagine I have an AI model and only me and my friend have it. I write a prompt and get back the vectors only. No actual output. Then I send my friend those vectors and they use this algorithm to reconstruct my message at the endpoint. Does this method of messaging protect against a MITM attack? Can this be used in cryptography?

vhxs 1y ago

Did the authors post code for their algorithm anywhere?