9 ms·
> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens
by yjftsjthsd-h 2y ago
> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B.
> We have released pre-trained base and instruction-tuned checkpoints checkpoints under the Apache 2.0 license to promote adoption for researchers and enterprises. Mistral NeMo was trained with quantisation awareness, enabling FP8 inference without any performance loss.
So that's... uniformly an improvement at just about everything, right? Large context, permissive license, should have good perf. The one thing I can't tell is how big 12B is going to be (read: how much VRAM/RAM is this thing going to need). Annoyingly and rather confusingly for a model under Apache 2.0, https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407 https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407 refuses to show me files unless I login and "You need to agree to share your contact information to access this model"... though if it's actually as good as it looks, I give it hours before it's reposted without that restriction, which Apache 2.0 allows.
- xena 2y agoEasy head math: parameter count times parameter size plus 20-40% for inference slop space. Anywhere from 8-40GB of vram required depending on quantization levels being used.
- imtringued 2y agoThey did quantization aware training for fp8 so you won't get any benefits from using more than 12GB of RAM for the parameters. What you might be using more RAM is the much bigger context window.
- exe34 2y agotensors look about 20gb. not sure what that's like in vram.
- kelsey98765431 2y agosame size
- renewiltord 2y agoAccording to nvidia https://blogs.nvidia.com/blog/mistral-nvidia-ai-model/ https://blogs.nvidia.com/blog/mistral-nvidia-ai-model/ it was made to fit on a 4090 so it should work with 24 GB.
- bernaferrari 2y agoif you want to be lazy, 7b = 7gb of vRAM, 12b = 12gb of vRAM, but quantizing you might be able to do with with ~6-8. So any 16gb Macbook could run it (but not much else).
- hislaziness 2y agoisn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?
- ancientworldnow 2y agoThis was trained to be run at FP8 with no quality loss.
- hislaziness 2y agoThe model description on huggingface says - Model size - 12.2B params, Tensor type - BF16. Is the Tensor type different from the training param size?
- fzzzy 2y agoit's very common to run local models in 8 bit int.
- qwertox 2y agoYes, but it's not common for the original model to be 8 bit int. The community can downgrade any model to 8 bit int, but it's always linked to quality loss.
- BaculumMeumEst 2y agoWhat about for fine-tuning? Are the memory requirements comparable to inference? If not, is there a rule of thumb for the difference? Would it be realistic to do it on a macbook with 96G of unified memory?
- peterleiser 2y ago
- Bumblonono 2y agoIt fits a 4090. Nvidia lists the models and therefore i assume 24gig is min
- michaelt 2y agoA 4090 will just narrowly fit a 34B parameter model at 4-bit quantisation. A 12B model will run on a 4090 with plenty room to spare, even with 8-bit quantisation.
- wongarsu 2y agoYou could consider the improvement in model performance a bit of a cheat - they beat other models "in the same size category" that have 30% fewer parameters. I still welcome this approach. 7B seems like a dead end in terms of reasoning and generalization. They are annoyingly close to statistical parrots, a world away from the moderate reasoning you get in 70B models. Any use case where that's useful can increasingly be filled by even smaller models, so chasing slightly larger models to get a bit more "intelligence" might be the right move
- mistercheph 2y agoI strongly disagree, have you used fp16 or q8 llama3 8b?
- amrrs 2y ago>reasoning and generalization any example use-cases or prompts? how do you define those?
- K0balt 2y agoI usually tell the model that I will be testing its reasoning capabilities by describing a scenario and then asking questions about the evolving scenario. I typically give it a description of a limited environment with objects in it, and say that “we “ are in this environment. I then describe actions that I take within the environment and ask questions about the updated world-state that must be inferred from the actions. This tests a lot of “common sense” reasoning skills, which I find to be more important for real world tasks than logic puzzle type reasoning.
- yjftsjthsd-h 2y agoI actually meant execution speed from quantisation awareness - agreed that comparing against smaller models is a bit cheating.
- qwertox 2y agoAren't small models useful for providing a language-based interface - spoken or in writing - to any app? Tuned specifically for that app or more likely enriched via RAG and possibly also by using function calling? It doesn't have to be intelligent like we expect it from the top-tier, huge models, just capable of understanding some words in sentences, mostly commands, and how to react to them.