3 ms·
It helps to be able to run the model locally, and currently this is slow or expensive. The challenges of running a local model beyond say 32B are real.
by selfhoster11 2y ago
It helps to be able to run the model locally, and currently this is slow or expensive. The challenges of running a local model beyond say 32B are real.
- rightbyte 2y agoYe the compressed version is not nearly as good. I would be fine though with like 10 times the wait time. But I guess consumer hardware need some serius 'ram pipeline' upgrade for big models to be run at crawl speeds.