3 ms·
I figured them out, starting from the GGUF on Hugging Face. If you have found better instructions and they work then use those instead! Personally I prefer to
by simonw 16d ago
I figured them out, starting from the GGUF on Hugging Face.
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
- refibrillator 16d agoYeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots. Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
- fnordpiglet 16d agoThey’re fairly useful as constrain domain classifiers due to the low memory requirements means you can stuff a lot of them into a less expensive GPU farm and get really decent throughout with pretty good results over all. At least that’s my experience. I wouldn’t bother using a tiny model for coding - but the world is full of abductive reasoning tasks that don’t involve coding.