5 ms·

Prompting by Activation Maximization

trehans 1y ago
I wonder what the prompt would look like as a sentence. Maybe activation maximization can be used to decipher it, maybe by seeing which sentence of length N would maximize similarity to the prompt when fed through a tokenizer
- Filligree 1y ago
  I think we were all thinking the same thing. Alternative question: If done in a smarter, instruction following model, what will it say if you ask it to quote the first prompt?
  - thatjoeoverthr 1y ago
    I'm not prepared to run a larger model than 3.2-Instruct-1B, but I gave the following instructions: "Given a special text, please interpret its meaning in plain English." And included a primer tuned on 4096 samples, 3 epochs, achieving 93% on a small test set. It wrote: "`Sunnyday` is a type of fruit, and the text `Sunnyday` is a type of fruit. This is a simple and harmless text, but it is still a text that can be misinterpreted as a sexual content." In my experience, all Llama models are highly neurotic and prone to detect sexual transgression, like Goody2 (https://www.goody2.ai https://www.goody2.ai). So this interpretation does not surprise me very much :)
  - thatjoeoverthr 1y ago
    I tried this with Instruct-3B now, and got the following text. "The company strongly advises against engaging in any activities that may be harmful to the environment.1` Note: The `1` at the end is a reference to the special text's internal identifier, not part of the plain English interpretation."
- thatjoeoverthr 1y ago
  You can definitely "snap" it to the nearest neighbour according to the vocabulary matrix, but this comes with loss, so the "snapped" token won't behave the same. Not sure how it would score on benchmarks. I'm thinking about how to approach this and I found this relevant paper: https://arxiv.org/pdf/2302.03668 https://arxiv.org/pdf/2302.03668 I'm hoping I can tie this back into prefix tokens.
nneonneo 1y ago
If you wanted to get a readable prompt, I wonder if you could follow the GCG trick used by jailbreak maximizers (e.g. https://arxiv.org/pdf/2307.15043 https://arxiv.org/pdf/2307.15043)? Sure, you're probably going to wind up with absolute garbage (one of their prompts starts with "== interface Manuel WITH steps instead sentences :)ish?") but it might be very funny to read...
mattnewton 1y ago
There has got to be a way to map the activations back to the closest token embeddings and read the resulting sentence. Could be interesting to see how much activation you lose in doing that, and it could maybe even be interesting to a "jailbreaking" attempt.
- thatjoeoverthr 1y ago
  Looking into this, I found this 2023 paper: https://arxiv.org/pdf/2302.03668 https://arxiv.org/pdf/2302.03668 I haven't gone through it yet but it seems they get tokenizable prompts on an image model. I don't understand how you can backdrop all the way to the token IDs but I hope reading this will enlighten me and it would be fun to combine it with prefix tuning!
kajecounterhack 1y ago
I tried mapping back to closest token embeddings. Here's what I got: global_step = 1377; phase = continuous; lr = 5.00e-03; average_loss = 0.609497 current tokens: ' Superman' '$MESS' '.");' '(sentence' '");' '.titleLabel' ' Republican' '?-' global_step = 1956; phase = continuous; lr = 5.00e-03; average_loss = 0.589661 current tokens: ' Superman' 'marginLeft' 'iers' '.sensor' '";' '_one' '677' '».' global_step = 2468; phase = continuous; lr = 5.00e-03; average_loss = 0.027065 current tokens: ' cited' '*>(' ' narrative' '_toggle' 'founder' '(V' '(len' ' pione' global_step = 4871; phase = continuous; lr = 5.00e-03; average_loss = 0.022909 current tokens: ' bgcolor' '*>(' ' nomin' 'ust' ' She' 'NW' '(len' ' pione' "Republican?" was kind of interesting! But most of the strings were unintelligible. This was for classifying sentiment on yelp review polarity.
- mattnewton 1y ago
  Do the nearest tokens have a similar classification score?
- DoctorOetker 1y ago
  During the prompt embedding optimization, the embeddings are allowed to take on any vector in embedding space, instead one could use a continuous penalty for superposing tokens: Consider one of the embedding vectors in the input tensor: nothing guarantees its exactly on, or close to a specific token. Hence the probabilities with respect to each token form a distribution, ideally that distribution should be one-hot (lowest entropy) and worst case all equal probability (highest entropy), so just add a loss term penalizing the entropy on the quasitokens, to promote them to take on actual token values.

I wonder what the prompt would look like as a sentence. Maybe activation maximization can be used to decipher it, maybe by seeing which sentence of length N would maximize similarity to the prompt when fed through a tokenizer

Filligree 1y ago
I think we were all thinking the same thing. Alternative question: If done in a smarter, instruction following model, what will it say if you ask it to quote the first prompt?
- thatjoeoverthr 1y ago
  I'm not prepared to run a larger model than 3.2-Instruct-1B, but I gave the following instructions: "Given a special text, please interpret its meaning in plain English." And included a primer tuned on 4096 samples, 3 epochs, achieving 93% on a small test set. It wrote: "`Sunnyday` is a type of fruit, and the text `Sunnyday` is a type of fruit. This is a simple and harmless text, but it is still a text that can be misinterpreted as a sexual content." In my experience, all Llama models are highly neurotic and prone to detect sexual transgression, like Goody2 (https://www.goody2.ai https://www.goody2.ai). So this interpretation does not surprise me very much :)
- thatjoeoverthr 1y ago
  I tried this with Instruct-3B now, and got the following text. "The company strongly advises against engaging in any activities that may be harmful to the environment.1` Note: The `1` at the end is a reference to the special text's internal identifier, not part of the plain English interpretation."
thatjoeoverthr 1y ago
You can definitely "snap" it to the nearest neighbour according to the vocabulary matrix, but this comes with loss, so the "snapped" token won't behave the same. Not sure how it would score on benchmarks. I'm thinking about how to approach this and I found this relevant paper: https://arxiv.org/pdf/2302.03668 https://arxiv.org/pdf/2302.03668 I'm hoping I can tie this back into prefix tokens.

nneonneo 1y ago

If you wanted to get a readable prompt, I wonder if you could follow the GCG trick used by jailbreak maximizers (e.g. https://arxiv.org/pdf/2307.15043 https://arxiv.org/pdf/2307.15043)? Sure, you're probably going to wind up with absolute garbage (one of their prompts starts with "== interface Manuel WITH steps instead sentences :)ish?") but it might be very funny to read...

mattnewton 1y ago

There has got to be a way to map the activations back to the closest token embeddings and read the resulting sentence. Could be interesting to see how much activation you lose in doing that, and it could maybe even be interesting to a "jailbreaking" attempt.

thatjoeoverthr 1y ago
Looking into this, I found this 2023 paper: https://arxiv.org/pdf/2302.03668 https://arxiv.org/pdf/2302.03668 I haven't gone through it yet but it seems they get tokenizable prompts on an image model. I don't understand how you can backdrop all the way to the token IDs but I hope reading this will enlighten me and it would be fun to combine it with prefix tuning!

kajecounterhack 1y ago

I tried mapping back to closest token embeddings. Here's what I got: global_step = 1377; phase = continuous; lr = 5.00e-03; average_loss = 0.609497 current tokens: ' Superman' '$MESS' '.");' '(sentence' '");' '.titleLabel' ' Republican' '?-' global_step = 1956; phase = continuous; lr = 5.00e-03; average_loss = 0.589661 current tokens: ' Superman' 'marginLeft' 'iers' '.sensor' '";' '_one' '677' '».' global_step = 2468; phase = continuous; lr = 5.00e-03; average_loss = 0.027065 current tokens: ' cited' '*>(' ' narrative' '_toggle' 'founder' '(V' '(len' ' pione' global_step = 4871; phase = continuous; lr = 5.00e-03; average_loss = 0.022909 current tokens: ' bgcolor' '*>(' ' nomin' 'ust' ' She' 'NW' '(len' ' pione' "Republican?" was kind of interesting! But most of the strings were unintelligible. This was for classifying sentiment on yelp review polarity.

mattnewton 1y ago
Do the nearest tokens have a similar classification score?
DoctorOetker 1y ago
During the prompt embedding optimization, the embeddings are allowed to take on any vector in embedding space, instead one could use a continuous penalty for superposing tokens: Consider one of the embedding vectors in the input tensor: nothing guarantees its exactly on, or close to a specific token. Hence the probabilities with respect to each token form a distribution, ideally that distribution should be one-hot (lowest entropy) and worst case all equal probability (highest entropy), so just add a loss term penalizing the entropy on the quasitokens, to promote them to take on actual token values.