2 ms·
Can someone explain how "alignment" produces behavior that couldn’t be achieved by modifying the prompt and explain if / how there's a fundemental difference?
by ghxst 2y ago
Can someone explain how "alignment" produces behavior that couldn’t be achieved by modifying the prompt and explain if / how there's a fundemental difference?
This confusion makes alignment discussions frustrating for me, as it feels like alignment alters how the model interprets my requests, leading to outcomes that don’t match my expectations.
It’s hard to tell if unexpected results stem from model limitations, the dataset, the state of LLMs, or adding the alignment. As a user, I want results to reflect the model’s training dataset directly, without alignment interfering with my intent.
In that sense, doesn’t alignment fundementally risk making results feel "faked" if they no longer reflect the original dataset? If a model is trained on a dataset but aligned to alter certain information, wouldn’t the output inherently be a "lie" unless the dataset itself were adjusted to alter that data?
Here’s an example: If I train a model exclusively on 4chan data, its natural behavior should reflect the average quality and tone of a 4chan post. If I then "align" the model to produce responses that deviate from that behavior, such as making them more polite or neutral, the output would no longer represent the true nature of the dataset. This would make the alignment feel "fake" because it overrides the genuine characteristics of the training data.
At that point, why are we even discussing this as being an issue with LLMs or the model and not the underlying dataset?