3 ms·
Even if the user input at every step is just normal conversation sentences? By that I mean very short, not MB of text.
by kleene_op 3y ago
Even if the user input at every step is just normal conversation sentences? By that I mean very short, not MB of text.
- sp332 3y agoYes, it would take several times as much memory (e.g. the AdamW optimizer uses 8 bytes per paramter). Plus, your latest sentences would just get mixed in to the model, and it wouldn't have a concept of the current conversation. There are very different approaches that might behave more like what you want, maybe RWKV.
- kleene_op 3y agoThanks for your suggestion about RWKV. I'll read up on that.
- westurner 3y agohttps://github.com/BlinkDL/RWKV-LM#rwkv-discord-httpsdiscordggbdsbumefpc-7k-members https://github.com/BlinkDL/RWKV-LM#rwkv-discord-httpsdiscord... lists a number of implementations of various versions of RWKV. https://github.com/BlinkDL/RWKV-LM#rwkv-parallelizable-rnn-with-transformer-level-llm-performance-pronounced-as-rwakuv-from-4-major-params-r-w-k-v https://github.com/BlinkDL/RWKV-LM#rwkv-parallelizable-rnn-w... : > RWKV: Parallelizable RNN with Transformer-level LLM Performance (pronounced as "RwaKuv", from 4 major params: R W K V) > RWKV is an RNN with Transformer-level LLM performance, which can also be directly trained like a GPT transformer (parallelizable). And it's 100% attention-free. You only need the hidden state at position t to compute the state at position t+1. You can use the "GPT" mode to quickly compute the hidden state for the "RNN" mode. > So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding (using the final hidden state). > Our latest version is RWKV-6,