That is an architectural problem. If you want the post to be rephrased: it is paradoxical to have changes made near the output level, "changing words before it

That is an architectural problem. If you want the post to be rephrased: it is paradoxical to have changes made near the output level, "changing words before it says them", given that the expected is to work with ideas. (And even then, selection would not be at the output level - it would be during the definition of the structure.) So, articles like this submission - while interesting from many points of view - make the elephant in the room more evident. > You cannot define an idea as a training loss objective What tells you so? If you see a technical limit, note e.g. that sentences and paragraphs can have their own position in an embedding space.