4 ms·
Oh interesting, so you're not using Llama 2, you're using the original. Have you begun to evaluate Llama 2 to determine the differences in performance? How are
by agg23 3y ago
Oh interesting, so you're not using Llama 2, you're using the original. Have you begun to evaluate Llama 2 to determine the differences in performance?
How are you determining what notes (or snippets of notes?) to be injected as context? Especially given the small 2048 context limit with Llama 1.
- sabaimran 3y agoQuick clarification, we are using LlamaV2 7B. We didn't experiment with Llama 1 because we weren't sure of the licensing limitations. We determine note relevance by using cosine similarity between the query and the knowledge base (your note embeddings). We limit the context window for Llama2 to 3 notes (while OpenAI might comfortably take up to 9). The notes are ranked based on most to least similar and truncated based on the context window limit. For the model we're using, we're still limited to 2048 tokens for Llama v2.
- bugglebeetle 3y agoHave you looked at using the long context (32K) version of the Llama v2 7B released by Together AI? https://together.ai/blog/llama-2-7b-32k https://together.ai/blog/llama-2-7b-32k
- 110 3y agoOh neat, thanks for sharing that! Having a 32K offline model is pretty promising. Let me test out how it performs
- OkGoDoIt 3y agoI thought llama V2 has a context window of 4096?