3 ms·
So in training, chatgpt turned words into embedding , and given context window N , looked at N embeddings an created a probabilities list for the following next
by petra 4y ago
So in training, chatgpt turned words into embedding , and given context window N , looked at N embeddings an created a probabilities list for the following next embedding ?
And if I tell it something that was excatly in it's trained context windows, I get the most likely next word and the one after itm
But what happens if I ask it something slighty different than it's training context ? Or something largely different?
- MacsHeadroom 4y agoBy "embedding" in this context what you're actually referring to is called a "token" which are sub-word strings of usually 1-4 characters. It's not possible for you to ask it things even slightly different from it training data, unless you ask exclusively in emojis that didn't exist yet when it was trained (in which case it sees nothing, just like when someone sends you an emoji your phone doesn't support). Any novel sentence and even novel words like "Blobdarfnk" ARE in its training data. "Blobdarfnk" is encoded as the five tokens Bl, ob, dar, fn, and k.