4 ms·
Model is too large to load onto the free Inference API. I don't understand which models one can use for free on HuggingFace and which not. For example this 18
by TekMol 3y ago
Model is too large to load onto the free Inference API.
I don't understand which models one can use for free on HuggingFace and which not.
For example this 180B model works:
https://huggingface.co/spaces/tiiuae/falcon-180b-demo https://huggingface.co/spaces/tiiuae/falcon-180b-demo
Why? Is someone paying for it so I can use it for free?
- nulld3v 3y agoThat link is a HuggingFace "space". The owner of the space selects what GPU is available to the space and pays for all GPU usage.
- nodja 3y agoYes, an A100 costs $4.13/hour https://huggingface.co/pricing#spaces https://huggingface.co/pricing#spaces
- TekMol 3y agoSo who is paying for the freely available models? For example there is StableDiffusion XL here: https://huggingface.co/spaces/latent-consistency/lcm-lora-for-sdxl https://huggingface.co/spaces/latent-consistency/lcm-lora-fo...
- nodja 3y agoThe owner of the space.
- dragonwriter 3y agoSDXL is only a 6.6B parameter model, its not a big model compared to LLMs.
- milansuk 3y agoYeah, they probably don't want their 'demo' links on the HN home page.
- Der_Einzige 3y agoYes they do want those links on HN home page. Clout/likes/stars are worth more than GPU credits to most.
- BoorishBears 3y agoThat 180B is also too large for the free inference API: https://huggingface.co/tiiuae/falcon-180B-chat https://huggingface.co/tiiuae/falcon-180B-chat HF has a spare capacity based free inference tier
- antirez 3y agoJust use llama.ccp and run the quantized model like yi-34b-chat.Q4_K_M.gguf (you will need a 32GB Macbook, likely).
- numpad0 3y agoNote also that 2x32GB DDR4 UDIMM kits are ~$120. It's still an overkill for most purposes, but a cheap one.
- Der_Einzige 3y agoHighly quantized models are crippled versions of the original model. Don't evaluate a model for a production use case based on it's quantized versions.
- deleted 3y ago[deleted]
- antirez 3y agoThis depends on the quantization level. Sure, they lose part of the performance, but are still quite useful and cheap to run.
- eigenvalue 3y agoI find the larger Q5 versions from theBloke to be pretty good and allow you to run a larger model than you’d be able to otherwise. But there is definitely an impact, although it’s hard to quantify. It’s sort of like how you get less effective at coding when you’ve only slept 5 hours.
- nacs 3y ago4_k_m and 5_k_m are very good quant levels that don't sacrifice much.