3 ms·
I've found that for the most part the articles that I want summarized are those which only fit the largest context models such as Claude. Because otherwise I ca
by RicoElectrico 2y ago
I've found that for the most part the articles that I want summarized are those which only fit the largest context models such as Claude. Because otherwise I can skim-read the article possibly in reader mode for legibility.
Is llama 2 a good fit considering its small context window?
- tcsenpai 2y agoPersonally I use llama3.1:8b or mistral-nemo:latest which have a decent contex window (even if it is less than the commercial ones usually). I am working on a token calculator / division of the content method too but is very early
- garyfirestorm 2y agowhy not llama3.2:3B? it has fairly large context window too
- reissbaker 2y agoI assume because the 8B model is smarter than the 3B model; it outperforms it on almost every benchmark: https://huggingface.co/meta-llama/Llama-3.2-3B https://huggingface.co/meta-llama/Llama-3.2-3B If you have the compute, might as well use the better model :) The 3.2 series wasn't the kind of leap that 3.0 -> 3.1 was in terms of intelligence; it was just: 1. Meta releasing multimodal vision models for the first time (11B and 90B), and 2. Meta releasing much smaller models than the 3.1 series (1B and 3B).
- reissbaker 2y agoI don't think this is intended for Llama 2? The Llama 3.1 and 3.2 series have very long context windows (128k tokens).
- tempodox 2y agoWhat about using a Modelfile for ollama that tweaks the context window size? I seem to remember parameters for that in the ollama GitHub docs.
- tcsenpai 2y agoI applied (for now) a pre-filled table with a 4096 default limit. Users can also specify an upper or lower limit from the UI directly now. Added chunk and recursive summarization too.
- htrp 2y agodo multi stage summarization?
- tcsenpai 2y agoHi! This was a good suggestion! I implemented it in v 1.1 which is already out :)