3 ms·
I got 3 tokens: st, raw, and berry. My point still stands: processing "berry" as a single token does not allow the model to learn its spelling directly, the way
by kadushka 1y ago
I got 3 tokens: st, raw, and berry. My point still stands: processing "berry" as a single token does not allow the model to learn its spelling directly, the way human readers do. It still has to rely on an explicit mapping of the word "berry" to b e r r y explained in some text in the training dataset. If that explanation is not present in the training data, it cannot learn the spelling - in principle.
- brookst 1y agoExactly. If “st” is 123, “raw” is 456, “berry” is 789, and “r” is 17… it makes little sense to ask the models to count the [17]’s in [123,466,789]: it demands an awareness of the abstraction that does not exist. To the extent the knowledge is there it’s from data in the input corpus, not direct examination of the text or tokens in the prompt.
- asadotzler 1y agoSo much for generalized intelligence, I guess.
- kadushka 1y agoIs a human who never learned how to read not generally intelligent?