4 ms·
Some of the models in TheBloke's repo are optimised for things like ggml, which runs well on Mac chips. Not sure if that's what you mean? Have been experimenti
by hereonout2 3y ago
Some of the models in TheBloke's repo are optimised for things like ggml, which runs well on Mac chips. Not sure if that's what you mean?
Have been experimenting with deploying to Mac's for inference as they are cheaper and use less power. But also still deploy to a substantial amount of gpus too.
Either way I try to take connections to huggingface hub out of the equation at deployment time.
- l5870uoo9y 3y agoThanks for responding. I was curious if there are any alternative to OpenAI in terms of pricing. Deployment of a capable model starts at around $80/mo and I am not sure how many requests it can handle then. Either way it is a lot higher than GPT-3.5.