4 ms·
There has got to be a way to map the activations back to the closest token embeddings and read the resulting sentence. Could be interesting to see how much acti
by mattnewton 1y ago
There has got to be a way to map the activations back to the closest token embeddings and read the resulting sentence. Could be interesting to see how much activation you lose in doing that, and it could maybe even be interesting to a "jailbreaking" attempt.
- thatjoeoverthr 1y agoLooking into this, I found this 2023 paper: https://arxiv.org/pdf/2302.03668 https://arxiv.org/pdf/2302.03668 I haven't gone through it yet but it seems they get tokenizable prompts on an image model. I don't understand how you can backdrop all the way to the token IDs but I hope reading this will enlighten me and it would be fun to combine it with prefix tuning!