3 ms·
I wonder if the same would be true if sessions didn't reset. Right now it's basically one pre-prompt vs one user prompt, and then the memory gets wiped. But if
by boredhedgehog 4y ago
I wonder if the same would be true if sessions didn't reset. Right now it's basically one pre-prompt vs one user prompt, and then the memory gets wiped. But if the model keeps running for months or years, would it perhaps develop a more stable personality with a much stronger force of habit that couldn't be counteracted within a reasonable timeframe?
- cgearhart 4y agoI doubt it. There’s no reason to suppose that any configuration of the parameters is any more or less susceptible to this type of attack. If the paper I cited is true, then the “thing” transformer-based language models learn is a function to predict the gradient update via the forward pass evaluation. That means that forward evaluation is an approximation of fine-tuning. That implies that the very act of evaluating an input changes the behavior of the model, regardless of the current parameter settings. The model would have to “give up” the powers of in-context learning in order to avoid this weakness.