9 ms·
It's a hard problem that token space only partially solve, because with filling the token space to the brim with documents will dilute the instructions in the p
by avereveard 4y ago
It's a hard problem that token space only partially solve, because with filling the token space to the brim with documents will dilute the instructions in the prompt.
One may argue that we'd need a separate channel for the instructions, but it would make harder to do anything but zero shot question and answer.
I think it's likely that systems with metadata for tagging sources, tasks separately from the human and ai interaction will eventually emerge. The gptchat api already works somewhat that way, but it isn't following the structure that we'll, probably because it was a later tune and not something built in the model from the beginning.
- pclmulqdq 4y agoIt sounds like that's a good use for encoder-decoder architecture. GPT-3 and LLaMA are decoder-only architectures. Training that encoder sounds hard, though.
- drakmo 4y agoThe LLM should categorize the prompts and answers and consider this for evaluation of further responses. It could give instructions a "more local weight" for the context but e.g. knowledge be it generated or provided a "more global weight". The impact of instructions or other local context parts would get lesser and lesser. This could be a function of time or length of conversation. Probably methods like this one are already tested with "reflection" behaviours of LLMs.