3 ms·
Fine Tuning Language Models
I want to fine tune a language model to summarize the action items from meeting transcriptions. Due to privacy concerns, I need to run on the model on my local machine. I also need to be able to fine tune the model for our specific need.
How do I pick a model to start with? There are so many options, and I feel lost.
- brucethemoose2 3y agoMost AI runners just ignore licensing and (illegally?) run LLaMA finetunes. There are tons of good 33B/65B instruct models you can run on any decent PC. But if you want to avoid the non commercial LLaMA license, you have 3 good options for a base model. - OpenLlama 13B - MPT 30B - Falcon 40B Of these, Falcon 40B is very difficult to run (slow in 4 bit, basically requires a professional GPU, no good cpu offloading yet). OpenLLaMA 13B only supports a context size of 2048 as of today... But that could change soon. So you probably want MPT instruct 30B, specifically this one: https://huggingface.co/TheBloke/mpt-30B-instruct-GGML https://huggingface.co/TheBloke/mpt-30B-instruct-GGML As the page says, you can try it out on a decent PC of your own with the OpenCL build of KoboldCPP. Change it to "instruct" mode, use the template on the page, offload as many layers as you can to your PC's dGPU, and run it in instruct mode. It may already work for your summarization needs. If not, you can finetune it with MPT's code and summarization data. https://github.com/mosaicml/llm-foundry https://github.com/mosaicml/llm-foundry Or train OpenLLaMA 13B with SuperHOT + summarization data using QLORA.
- brucethemoose2 3y ago*But if your input prompts are short, an OpenLLaMA finetune like this would work: https://huggingface.co/TheBloke/open-llama-13b-open-instruct-GGML/tree/main https://huggingface.co/TheBloke/open-llama-13b-open-instruct...
- matrix2596 3y agoIs there a good summary of the trade offs of the various available models (open source) and their comparison to other models.