3 ms·
WORDS IN CAPS are different tokens than lowercase, so maybe the lowercase tokens tie into more trained parts of the manifold.
by tape_measure 1y ago
WORDS IN CAPS are different tokens than lowercase, so maybe the lowercase tokens tie into more trained parts of the manifold.
- maxbond 1y agoThat's a super interesting hypothesis. From an information theory perspective, rarer tokens are more informative. Maybe this results in the caps lock tokens being weighted higher by the attention mechanism.