3 ms·
Try this: 'this is a day that this sentence with clarify that day. Is this not a good day?' [5661, 318, 257, 1110, 326, 428, 6827, 351, 18282, 326, 1110, 13,
by eternalban 4y ago
Try this:
'this is a day that this sentence with clarify that day. Is this not a good day?'
[5661, 318, 257, 1110, 326, 428, 6827, 351, 18282, 326, 1110, 13, 1148, 428, 407, 257, 922, 1110, 30]
Note 'day' is solidly 1110 here. Now start a sentence with day.
"day began with laughter"
[12393, 2540, 351, 20263, 13]
So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well.
"Day after day after this day"
[12393, 706, 1110, 706, 428, 1110]
"Day day Home home home"
[12393, 1110, 5995, 1363, 1363]
"day day day home home home"
[820, 1110, 1110, 1363, 1363, 1363]
[corrected/edited: so case-sensitive and position sensitive as well.]
btw doesn't the output array contain the prompt as well (because of the transformer architecture? not entirey sure ~iirc)
- fenomas 4y ago> So the logical word -> token(s, p) -> id(s) function definitely has 1 position parameter as well. You're missing that it groups in spaces. The position isn't relevant, but "day" is a different token than " day".
- eternalban 4y agoAh, you're right. [try "datedate". interesting how it partitions that as 'dated' + 'ate'. Compare with "matemate" -> 'mat', 'emate'.] p.s. "ifthiswasagermanword if this was a german word." It's not even spaces. That second sequence ' german' is chopped up as 'ag' 'erman'.
- hn_throwaway_99 4y agoIt's a lot simpler than that. You can see in the tokenizer that the boundary for words includes the preceding space. So, since the first word doesn't have a preceding space, it has a different token.
- sebzim4500 4y agoThere's no position parameter, it's just that " day" and "day" are different tokens.