3 ms·
My favourite thing about llamafile is that it means we can now download a single 4GB file and get both a language model and the software needed to execute again
by simonw 3y ago
My favourite thing about llamafile is that it means we can now download a single 4GB file and get both a language model and the software needed to execute against that model in a single unit, which we can use on multiple different platforms.
Me, a few weeks ago: "Stick that file on a USB stick and stash it in a drawer as insurance against a future apocalypse. You’ll never be without a language model ever again."
There are llamafiles for Mixtral-8x7B-Instruct now too, which is currently the most capable openly licensed model: https://huggingface.co/jartine/Mixtral-8x7B-v0.1.llamafile/tree/main https://huggingface.co/jartine/Mixtral-8x7B-v0.1.llamafile/t... - those ones are 32GB.
I wrote some more notes on llamafile here: https://simonwillison.net/2023/Nov/29/llamafile/ https://simonwillison.net/2023/Nov/29/llamafile/
To save us some time...
1. A llamafile is just a zip, so you don't have to download the GGUF of a model separately if you already have the llamafile, just unzip it
2. You can use llamafile against other models without having to bundle them together - the llamafile binary without a bundled model is about 5MB. Notes on that here: https://simonwillison.net/2023/Nov/29/llamafile/#llamafile-trying-other-models https://simonwillison.net/2023/Nov/29/llamafile/#llamafile-t...
3. If you already have a preferred way of running models on your own computer that isn't llamafile, great! This software isn't for you.
- tmikaeld 3y agoI thought Mixtral required 34GB of RAM to run at that quantization? Yet RAM requirements aren't mentioned anywhere (in your links).
- biscuitech 3y ago[dead]