4 ms·

can you select a context length that fits in your GPU though? I suppose even a 128k model would be more than enough for almost everyone running these models on

can you select a context length that fits in your GPU though? I suppose even a 128k model would be more than enough for almost everyone running these models on their own hardware.

wkat4242 2y ago
No you can't right now. Hopefully they will add this to ollama.
- ml2 2y ago
  256k (actually 262k) is also up on HF: https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k https://huggingface.co/gradientai/Llama-3-8B-Instruct-262k