4 ms·
Is anyone using this models self-hosted in production? What cloud hosting provider/plan do you use and how is it performance wise?
by ensocode 3y ago
Is anyone using this models self-hosted in production? What cloud hosting provider/plan do you use and how is it performance wise?
- rybosome 3y agoVery interested to hear a similar answer, but if anyone has tried running it locally on high-end consumer grade hardware, eg an Nvidia 4090?
- mdcurrent 3y agostumbled on this thread looking for tips- so far i haven't had success. i can load the model into memory by lowering the precision, I was OOMing on my 128GB RAM, now im OOMing on loading the model into the 4090 on the 17th shard at 4 bit qtization so i would imagine theres another knob or two im missing to get this to run
- rybosome 3y agoThanks for your insight. Sounds like it’s quite a challenge, 128GB of RAM and a 4090 are pretty beefy for a consumer PC.
- reqo 3y agoConsidering that this model has basically zero censorship I would say it is going to take a while before companies have fine-tuned this enough before putting it in production.
- dannyw 3y agoThis seems like grounds for an interesting and sustainable business model. * Offer uncensored, pure-instruction-following open source models. * Offer censored, 'corporate-values' models commercially.
- anonyfox 3y agoTo the contrary, I want need things locally/cheap/fast for generic internal business automation, so it will never generate content any outsider would ever see. Basically glue code between services with classification of data built-in.
- seydor 3y agoThe community will do that for them. Btw it is unfortunate that 'censored' became the default expectation. Mistral gives the raw model because that makes it useful for all kinds of purposes, e.g. moderation. Censorship is an addon module (or a lora or sth).
- cthalupa 3y agoBusinesses have a huge incentive to censor these models - it's appreciated that these are released without it, but a business has a lot of concerns about what is said around anything they are offering as a service. The point was just that production implies a business use and that implies the need to make sure there are guardrails in place to make sure the model sticks to the business purpose instead of teaching people how to make pipe bombs. Not that anyone thinks that prevents people from learning how to make pipe bombs - they just don't want to be the ones doing the teaching.
- drexlspivey 3y agoThis model dropped on Friday, how would anyone be using it “in production” by Monday morning?
- computerex 3y agoNot to mention the fact that only model weights dropped without the inference code so people had to hack that together.