3 ms·
I run it at home at q8 on my dual Epyc server. I find it to be quite good, especially when you host it locally and are able to tweak all the settings to get the
by DrPhish 2y ago
I run it at home at q8 on my dual Epyc server. I find it to be quite good, especially when you host it locally and are able to tweak all the settings to get the kind of results you need for a particular task.
- rightbyte 2y agoI've used it too locally. It is great for some kind of querries or writing bash, which I refuse to learn properly. I really don't want my querries to leave my computer, ever. It is quite surreal how this 'open weights' model get so little hype.
- selfhoster11 2y agoIt helps to be able to run the model locally, and currently this is slow or expensive. The challenges of running a local model beyond say 32B are real.
- rightbyte 2y agoYe the compressed version is not nearly as good. I would be fine though with like 10 times the wait time. But I guess consumer hardware need some serius 'ram pipeline' upgrade for big models to be run at crawl speeds.