4 ms·
Stable LM 3B: Bringing Sustainable, High-Performance LMs to Smart Devices
- r3trohack3r 3y ago> Developers should be mindful that Stable LM 3B is a base model. That means it needs to be adjusted for safe performance in specific applications, such as a chat interface. Depending on their use case, developers must evaluate and fine-tune the model before deployment. I don't know how to fine tune an LLM. Does anyone have good resources on how to do this?
- nickthegreek 3y ago/r/LocalLLaMA/ is a good resource.
- 2bitencryption 3y ago/r/LocalLLaMA/ is such an interesting mix of academia/researchers (it was called out in a recent paper, regarding context length, IIRC) and odd anarcho-futurist weirdos. And I kind of love it for that.
- antupis 3y agoYup kinda best subreddit at the moment weird mix of llm training tips and tricks and people who wanna make sex bots.
- politelemon 3y agoHa Stable Diffusion is very similar I notice. People understanding and thriving in new techniques and advanced workflows, mixed in with people who want to generate boobies in new ways.
- lawlessone 3y agoI found they can be very unappreciative of their data sources for training. Any mention that scraping art from art sites like DA without their consent was unethical is met with arguments about how its exactly the same as a human looking at art and replicating the style. They refuse to even acknowledge why those people might be annoyed their work was used without their consent or compensation to put them out of work.
- yieldcrv 3y agoyou know what you’re walking into there, lawlessone
- omneity 3y agoHuggingface tutorials are a decent starting point. The HF toolchain is pretty mature and most llm finetuning projects are a wrapper around HF models, HF Trainer and some config templates. An LLM by default would be trained like in the example below, but it would take a lot of VRAM and time. https://huggingface.co/docs/transformers/training https://huggingface.co/docs/transformers/training That’s where things like PEFT LoRa, gptq and accelerate contribute to make your training faster/require less VRAM so you could do it on a consumer GPU with 16-24GB. For example: https://huggingface.co/docs/peft/quicktour https://huggingface.co/docs/peft/quicktour Then for the tips and tricks, either reddit as suggested by a sibling comment, or Discord communities. Huggingface, EleutherAI and LAION discord servers are all great and have super helpful, friendly and knowledgeable people.
- Nevin1901 3y agoI’m developing a service just for that. It’s called https://useftn.com https://useftn.com and it’ll fine tune models off your json formatted dataset.
- redox99 3y agoI'm surprised by your pricing. $1.20/hr for an A100? That's significantly cheaper than something like runpod. Are you offering this service at a loss?
- omneity 3y agoWhich dataset formats are you planning to support? (OpenAI, ShareGPT, Alpaca, Context-based ...)
- Nevin1901 3y agoSame as OpenAI, where it's either a json or jsonl file with {"input": "text", "output": "text"}
- lhl 3y agoFor getting started, I'd recommend https://huggingface.co/autotrain https://huggingface.co/autotrain - you should be able to find docs on the CLI which can let you do your first training run w/ a single command (which includes automatically pulling the model and training dataset from Huggingface). As long as you have an Nvidia GPU w/ enough VRAM should take only a few hours to tune a 3B model on a small dataset like Alpaca.
- jnwatson 3y ago3 billion parameters is a bit large for most smart devices, is it not? Do they use some smaller quantization?
- esafak 3y agoThey say they are targeting edge or home PCs.
- redox99 3y ago> 3 billion parameters is a bit large for most smart devices, is it not? Not really. A 3B model quantized to 4 bits should run in any reasonable smartphone (using around 2GB of memory).
- coder543 3y agoDepends on your definition of edge or smart device. My phone can run a 7B parameter model at 12 tokens per second, which is probably faster than most humans are comfortable reading, and definitely faster than a virtual assistant would speak. Out of curiosity, I tested a 3B parameter model, and it runs at about 21 tokens per second on my phone.
- omneity 3y agoGenerating text fast enough for a human to read it imo is only the bare minimum. New classes of use cases would be possible if you could generate 100s of tokens. For example, on-device classification of emails/texts, LLM-powered recommendation system based on your local data (to go beyond simply parsing dates in text for example), context-aware text or email auto-responder (I'm sorry, can't reply as I'm driving/in a meeting, I'm not home next week, can you deliver to this address instead etc.) ... Many of these use cases are possible today with either specialized models, or are old school and rule-based. Being able to have an LLM apply soft judgment on a device that generates so much contextual information, and completely locally/privately, is bound to make smartphones an entirely new kind of device.
- coder543 3y ago
- brrrrrm 3y agoCurious how this compares to Mistral's release? 3b->7b doesn't seem like a huge leap and Mistral has shown phenomenal results for its size
- omneity 3y agoDo you have a link/reference about Mistral's performance? I'm interested to read more about it.
- brrrrrm 3y agohttps://mistral.ai/news/announcing-mistral-7b/ https://mistral.ai/news/announcing-mistral-7b/ looking at the 3b results (here https://github.com/Stability-AI/StableLM#stablelm-alpha-v2 https://github.com/Stability-AI/StableLM#stablelm-alpha-v2 ?), it looks like Mistral (which outperforms Llama-2 13b) is far more powerful
- refulgentis 3y agon.b. absolutely no tuning on 3B currently, no RLHF, no instruct, no chat
- stavros 3y agoDoes this mean that this is straight-up text completion? That's still pretty useful, you just have to know how to write the prompt, no?
- emadm 3y agoIt's a model 40% of the size of Mistral's designed to be transparent (full training details, datasets & evals here: https://stability.wandb.io/stability-llm/stable-lm/reports/StableLM-3B-4E1T--VmlldzoyMjU4?accessToken=u3zujipenkx5g7rtcj9qojjgxpconyjktjkli2po09nffrffdhhchq045vp0wyfo https://stability.wandb.io/stability-llm/stable-lm/reports/S...) and work on edge devices. There are improved versions coming but this is the best 3b model and Mistral is the best 7b model.
- naillo 3y agoStability is so awesome, I love them
- sp332 3y agoIt's CC BY-SA-4.0. Anyone got a torrent?
- kristianp 3y ago> You need to agree to share your contact information to access this model This repository is publicly accessible, but you have to accept the conditions to access its files and content. > Login or Sign up to review the conditions and access this model content. How does that work with just calling `tokenizer = AutoTokenizer.from_pretrained("stabilityai/stablelm-3b-4e1t")`
- nacs 3y agoThe CLI will also prompt you to sign into huggingface in a browser and accept the model terms before it downloads.