4 ms·
> Are they practical and useful, or still facing significant limitations? They are. Working on a product using a fine-tuned Mistral-7B-Instruct-v0.2 model and
by dvt 3y ago
> Are they practical and useful, or still facing significant limitations?
They are. Working on a product using a fine-tuned Mistral-7B-Instruct-v0.2 model and it's pretty mind-blowing. Works flawlessly on my RTX3090 and serviceable on my M1 MBP as well. I'm building in Rust (using the candle crate), but for personal usage Python is probably the better choice since it's easier to get up and running.
- hasty_pudding 3y ago[dead]
- malux85 3y agoMistral-7B-Instruct-v0.2 - I'm using this exact model too, and it is mind blowing, but to get the most out of it, make sure you use llama.cpp and turn on self-extend (I'm not sure if support for self-extend has been merged into main yet, I manually merged a dev branch)
- FezzikTheGiant 3y agoI asked this another place in this thread, but curious to know how you plan to run and deploy this model. What's the cheapest or I guess most cost efficient way to do this without burning money. I'm a college student for reference so I don't have a lot of money lying around to experiment.
- malux85 3y agoYou’re right it’s expensive, I have infra in my house, not gonna lie it cost a lot, I have a rack with 30k of equipment in it (including 5 GPUs) But this would probably run on an AWS P2 instance with is 0.90 USD an hour, or there’s lambda labs which is also pretty cheap (no affiliation, just satisfied customer)
- FezzikTheGiant 3y agoThanks for the info! Appreciate it
- bilsbie 3y agoWhat’s the cheapest machine I could Buy to do this?
- sosuke 3y agoShooting from the hip here I’d say $800 would be a good start for all new parts. You might get down to $500 with used parts. The most expensive piece in either build is a video card. You want to be able to load the LLM file into your video card RAM. I just got a 16gb new card for this. I can load up to 34b models on it but poorly. Anything 13b or less runs perfectly. A 12gb card would be able to run 7b models and with the right training I think 7b models can be awesome. New 4060ti Used 3090
- talldayo 3y agoMistral-7B will run on a Raspberry Pi at q04 and a little bit of patience. For good acceleration though, you'll want an Nvidia machine with enough VRAM to comfortable store the model.
- FezzikTheGiant 3y agoWhat's the cheapest way to run and deploy a fine-tuned model today? If I understand correctly I'll need to run cloud GPUs right
- ThrowAIAway 3y ago+1 interested to know this as well.
- xyc 3y agoMistral-7B-Instruct-v0.2 is amazing. I've set it as the default model of RecurseChat (a local AI chat app). It works great, until recently WizardLM 2 came out and I tried it with local RAG. The difference is quite significant: https://twitter.com/chxy/status/1780101542311579942 https://twitter.com/chxy/status/1780101542311579942 Considering switching the default model after they put it back on HF.