3 ms·
I wonder if we could implement the equivalent of a JIT compilation, whereby context sequences which get repeatedly reused would be used for an online fine-tunin
by falcor84 2y ago
I wonder if we could implement the equivalent of a JIT compilation, whereby context sequences which get repeatedly reused would be used for an online fine-tuning.
- HumanOstrich 2y ago[flagged]
- nostrebored 2y agoThey are asking if you can take the context being passed per interaction and train it into a session in real time (via an online algorithm). Essentially bake the context passed in to the attention layer so that you can pass only the relevant chat context. Your post wasn’t a particularly charitable interpretation.
- sp332 2y agoNo, but you can just cache the state after processing the prompt. https://github.com/ggerganov/llama.cpp/tree/master/examples/main#prompt-caching https://github.com/ggerganov/llama.cpp/tree/master/examples/...