3 ms·
I understand the basic idea of predicting the words in a sequence, but what totally eludes me is how this relates to the prompt. After all, you don't give it a
by ribit 4y ago
I understand the basic idea of predicting the words in a sequence, but what totally eludes me is how this relates to the prompt. After all, you don't give it a sequence to continue, you give it a direct request. Is there some special processing going on here or do they really just take the prompt as is and encode it?
- Sharlin 4y agoA direct request is just a specific sort of a sequence to complete. There are extra training steps involved in ChatGPT that are designed to make it particularly good at predicting sequences that look like semantically valid, coherent dialogue, but fundamentally there's nothing special about dialogue. It's just tokens after tokens. For example, even a fairly simple model trained on only play and movie scripts would be naturally good at completing pieces of dialogue because that would be all that it knows!