4 ms·
LLMs ought to be good at generating simple substitutions too, but they're not: https://claude.ai/share/9495246c-b77a-496e-8643-dbc5ec58dbcb https://claude.ai/sh
by xnorswap 2y ago
LLMs ought to be good at generating simple substitutions too, but they're not: https://claude.ai/share/9495246c-b77a-496e-8643-dbc5ec58dbcb https://claude.ai/share/9495246c-b77a-496e-8643-dbc5ec58dbcb
It tries but it fails to both generate and apply a simple substitution. It starts substituting based on applying the substitution twice to some letters.
It was a very frustrating time I spent trying to solve it, especially after asking to confirm it had created a 1:1 mapping.
Since it has no reasoning, it just lies about having done so.
- GardenLetter27 2y agoWhy should they be? The token is at the word level more or less, so the model can't see the letters to substitute. When we get character-level models this should be more viable.