6 ms·
Mistral 7B Fine-Tune Optimized
- nickthegreek 3y agoAnytime I see a claim that our 7b models are better than gpt-4 I basically stop reading. If you are going to make that claim, give me several easily digestible examples of this taking place.
- achille 3y agoThey can absolutely outperform gpt4 for specific use cases.
- nickthegreek 3y agoI am very open to believing that. I'd love to see some examples.
- GaggiX 3y agoWell it's pretty easy to find examples online, this one using Llama 2, not even Mistral or fancy techniques: https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...
- turnsout 3y agoI agree, I think they need an example or two on that blog post to back up the claim. I'm ready to believe it, but I need something more than "diverse customer tasks" to understand what we're talking about.
- shiftpgdn 3y agoThey're quite close in arena format: https://chat.lmsys.org/?arena https://chat.lmsys.org/?arena
- bugglebeetle 3y agoYou can fine-tune a small model yourself and see. GPT-4 is an amazing general model, but won’t perform the best at every task you throw at it, out of the box. I have a fine-tuned Mistral 7B model that outperforms GPT 4 on a specific type of structured data extraction. Maybe if I fine-tuned GPT-4 it could beat it, but that costs a lot of money for what I can now do locally for the cost of electricity.
- TOMDM 3y agoYeah, a 7B foundation model is of course going to be worse when expected to perform on every task. But finetuning on just a few tasks? Depending on the task, it's totally reasonable to expect that a 7B model might eke out a win against stock GPT4. Especially if there's domain knowledge in the finetune, and the given task is light on demand for logical skills.
- deleted 3y ago[deleted]
- holoduke 3y agoNot for translations. Did a lot of experimenting different local models. None come even a bit close to the capabilities of chatgpt. Most local models just outputting plain wrong intormation. I am still hoping one day it will be possible. For our business a huge opportunity.
- ijk 3y agoFor translation, you're probably better off with a model that's specifically designed for translation, like MADLAD-400 or DeepL's services.
- gmuslera 3y ago“… with my definition of better” should be the default interpretation whenever you see the word better anywhere.
- filterfiber 3y agoIn their second sentence they have the most honest response I've seen so far at least: " averaged across 4 diverse customer tasks, fine-tunes based on our new model are _slightly_ stronger than GPT-4, as measured by GPT-4 itself."
- hospitalJail 3y agoSome things to note about gpt4: >Sometimes it will spit out terrible horrid answers. I believe this might be due to time of the day/too many users. They limit tokens. >Sometimes it will lie because it has alignment >Sometimes I feel like it tests things on me So, yes you are right, gpt4 is overall better, but I find myself using local models because I stopped trusting gpt4.
- moffkalast 3y agoHow are local models better in terms of trust? GPT 4 is the only model I've seen actually tuned to say no when it doesn't have the information being asked for. Though I do agree it used to run better earlier this year. The best open source has to offer is Mixtral that will confidently make up a biography of a person it's never heard of before or write a script with nonexistant libraries.
- mattkevan 3y agoI once asked Llama whether it’d heard of me. It came back with such a startlingly detailed and convincing biography of someone almost but not quite entirely unlike me that I began to wonder if there was some kind of Sliding Doors alternate reality thing going on. Some of the things it said I’d done were genuinely good ideas, and I might actually go and do them at some point. ChatGPT just said no.
- hospitalJail 3y agoTo be clear, the comparison was originally with GPT3 and ChatGPT3. ChatGPT3 would lie about anti-vaxx books never existing. GPT3 would answer facts.
- crooked-v 3y agoDon't forget that ChatGPT 4 also has seasonal depression [1]. [1]: https://twitter.com/RobLynch99/status/1734278713762549970 https://twitter.com/RobLynch99/status/1734278713762549970 (Though with that said, the seasonal issue might be common to any LLM with training data annotated by time of year.)
- tomrod 3y agoLooks like they utilized the Bradley-Terry model, but that's not one I'm super familiar with. https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model https://en.wikipedia.org/wiki/Bradley%E2%80%93Terry_model
- huac 3y agothe BTL model is just a way to infer 'true' skill levels given some list of head to head comparisons. the head to head comparisons / rankings are the most important!!!! and in this case, the rankings come from GPT-4 itself. so take any subsequent score with all the grains of salt you can muster. their methodology also appears to be 'try 12 different models and hope 1 of them wins out.' multiple hypothesis adjustments come to mind here :)
- brucethemoose2 3y agoIDK about GPT4 specifically, but I have recently witnessed a case where small finetuned 7Bs greatly outperformed larger models (Mixtral Instruct, Llama 70B finetunes) in a few very specific tasks. There is nothing unreasonable about this. However I do dislike it when that information is presented in a fishy way, implying that it "outperforms GPT4" without any qualification.
- jug 3y agoWhat I think they’re claiming is that it’s a base model aimed for further fine tuning, that when further tuned might perform better than GPT-4 on certain tasks. It’s an argument they make at least as much to market fine tuning as their own model. This is not a generic model that outperforms another generic model (GPT-4). That can of course have useful applications because the resource/cost is then comparatively minuscule for certain business use cases.
- deleted 3y ago[deleted]
- thorum 3y agoAnecdotally, I finetuned Mistral 7B for a specific (and slightly unusual) natural language processing task just a few days ago. GPT-4 can do the task, but it needs a long complex prompt and only gets it right about 80-90% of the time - the finetuned model performs significantly better with fewer tokens. (In fact it does so well that I suspect I could get good results with an even smaller model.)
- oceanplexian 3y agoI have a fine tuned version of Mistral doing a really simple task and spitting out some JSON. I'm getting equivalent performance to GPT-4 on that specialized task. It's lower latency, it's outputting more tokens/sec., more reliable, private, and completely free. I don't think we will have an Open Source GPT4 for a long time so this is sorta clickbait, but for the small, specialized tasks, tuned on high quality data, we are already in the "Linux" era of OSS models. They can do real, practical work.
- YetAnotherNick 3y ago> completely free Not according to my calculation. For low request rate it is likely more expensive than GPT4.
- tomrod 3y agoBeen my thought for awhile now. Can you recommend where I can learn more about hardware requirements for running Mistral/Mixtral?
- cced 3y agoHow are you guys fine tuning?
- skeletonjelly 3y agoCan you please point me in the direction of the guide you used for fine tuning? Did you use QLoRA?
- mistercheph 3y agohttps://chat.lmsys.org/?arena https://chat.lmsys.org/?arena Try a few blinds, mixtral 8x7b-instruct and gpt-4 are 50-50 for me, and it outperforms 3.5 almost every time, and you can run inference on it with a modern cpu and 64 GB of RAM on a personal device lmfao. and the instruct finetuning has had nowhere near the $$$ and rlhf that openai has. It's not a done deal, but people will be able to run models better than today's SOTA on <$1000 hardware in <3 months, I hope for their own sake that OpenAI is moving fast.
- kcorbitt 3y ago(Post author here). Totally fair concern. I'll find some representative examples on a sample task we've done some fine-tuning on and add them to the post. EDIT: Ok so the prompt and outputs are long enough that adding them to the post directly would be kind of onerous. But I didn't want to leave you waiting, so I copied an example into a Notion doc you can see here: https://opipe.notion.site/PII-Redaction-Example-ebfd29939d254e1db1e674ec7347b2b6?pvs=25 https://opipe.notion.site/PII-Redaction-Example-ebfd29939d25...
- xrd 3y agoWe've tried to sell variants of the open source models to our existing enterprise customers. I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. Saying something performs slightly better than the industry standard offerings (OpenAI) means that OpenAI is going to laugh all the way to the bank. Everyone will just use their APIs over anything else. I'm excited about the LLM space and I can barely keep up with the model names, much less all the techniques for fine tuning. A customer is going to have an even worse time. No one will ever get fired for buying OpenAI (now that IBM is dead, and probably sad Watson never made a dent). I do use Mistral for all my personal projects but I'm not sure that is going to have the same effect on the industry as open source software did in the past.
- turnsout 3y agoThere's a lot of truth to this, but I have seen clients get really interested in local models—mostly due to cost and/or confidentiality. For example, some healthcare clients will never upload medical records to OpenAI, regardless of the enterprise agreement.
- esafak 3y agoOpenAI is nothing like IBM in its heyday. I bet a very healthy proportion of companies will not share their data with OpenAI. I saw some numbers on this a while back I don't have the link handy. Trust has to be earned.
- bugglebeetle 3y agoThe problem here is that the platform offering here is overly complicated to get started with and quite limited. 2000 dataset entries for $50 a month when I can do 10x as many as that on Colab for free with axolotl or unsloth? Yeah, no thanks.
- wavemode 3y ago> I think the adage about "a solution needs to be 10x other solutions to make someone switch" applies here. Cheaper and faster is also better. The cheapest version of GPT-4 costs $0.01/$0.03 per 1K input/output tokens [1]. Mistral AI is charging 0.14€/0.42€ per ONE MILLION input/output tokens for their 7B model [2]. It's night and day. If people can start fine-tuning a 7B model to do the same work they were doing with GPT-4, they will 100% switch. [1]: https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp... [2]: https://docs.mistral.ai/platform/pricing/ https://docs.mistral.ai/platform/pricing/
- gpjanik 3y agoI want an interactive prompt box with some example prompts and answers from the model and a comparison with GPT-4. My random guess is that this finetuned Mistral-7B is better at nothing or almost nothing than GPT-4 and that's why instead of the above, we got a table with a bunch of irrelevant metrics.
- boredumb 3y agoOf course mistral7b is worse than GPT-4, but I can run mistral-7b at home.
- gpjanik 3y agoThe point is that the article states "averaged across 4 diverse customer tasks, fine-tunes based on our new model are slightly stronger than GPT-4, as measured by GPT-4 itself" and then proves it with nothing tangible, just the 4 selected metrics where it performs the best. I mean obviously a finetuned 7B LLM could perform, let's say, text summarization well. The question is what happens if that text contains code, domain-specific knowledge where some facts are less relevant than the other, etc., and that isn't going to be answered by any metric alone. Fundamentally, with enough diverse metrics, each based on a different dataset, the one with the biggest overlap of the dataset for finetuning will perform really well, and the rest, well, not so well. Bsically, the statistic means that there's a set of data for which that particular (finetuned) network performs slightly better than GPT-4, and everywhere else, pretty bad. It's just not generalizable to everything while GPT-4 is. It's just as good as saying "calculators outperform GPT-4 at counting". Like, yes, they probably do, but I would like to see - is it applicable and practical, or did you just train a LLM to write all the names in Polish alphabetically really well? And that's why qualitative approach for evaluation LLMs is just better.
- avereveard 3y agonot a bad model, becomes incoherent at above 8k token, and it's not helped by the fact that's very verbose, but seems very coherent and stay on topic closely until then: https://chat.openai.com/share/089d1b8c-3467-4c01-af9f-6568c0f95262 https://chat.openai.com/share/089d1b8c-3467-4c01-af9f-6568c0... fails at math of course, even if the problem is very easy, like all mistrals. good for genration, probably not the best for RAG, there's mistral tunes that stay coherent to 16k tokens, and that cuts down chunking significanty
- jahsome 3y ago~Is the 'k' in your token sizes a typo?~ Edit: mistook tokens for parameters for a moment there. Keeping up with AI jargon is exhausting for an idiot like me.
- avereveard 3y agoNo it's the sequence length like how much long is the string in the prompt so to say, 8192 token and it starts losing coherence and by 10000 tokens it was emitting gibberish, like empty lines and half words, I didn't put the worst part into the link. What do you mean by ELII?
- jahsome 3y agoExplain like I'm an idiot :D
- avereveard 3y agoAh hope it was clear enough in the answer :D What your see in the link is the copy paste of a discussion between me and the model in question, that I pasted into gpt4 with the instructions to evaluate it.the answer with the votes in 10/10 is gpt evaluating the chart between me and the smaller model. The smaller model is producing the text after ASSISTANT, the question that I do as USER is part of a fixes script that I run with every new model so that I have a sort of a validation set before doing some more rigorous testing.
- YetAnotherNick 3y agoOne thing that most people don't realize is that (full parameter)finetuned models are costly unless you run it in batched mode. Which means unless the request rate is very high and consistent, it is better to use prompts with GPT-3.5. e.g. batch of 1, mistral is more expensive than GPT-4[1]. [1]: https://docs.mystic.ai/docs/mistral-ai-7b-vllm-fast-inference-guide https://docs.mystic.ai/docs/mistral-ai-7b-vllm-fast-inferenc...
- ComputerGuru 3y agoBut this doesn’t apply to self-hosted, o?
- lyjackal 3y agoIt does. LLMs are most efficient when running large batches, so the gpu cost is super high if you’re underutilizing it. It will cost more than a cloud provider like open ai who has the volume to keep their GPUs saturated
- jay-barronville 3y agoYup. It’s also important to mention that OpenAI enjoys the luxury of having large clusters of H100s (the last time I checked).
- MacsHeadroom 3y agoI cloud host Mistral 7B for 20x cheaper than GPT-4-Turbo. And Mistral 7B API is $0.00/1M tokens, i.e. free : https://openrouter.ai/models/mistralai/mistral-7b-instruct https://openrouter.ai/models/mistralai/mistral-7b-instruct
- laweijfmvo 3y agois it possible to fine tune something like Mistral 7B on a large PDF (i’m thinking like multi-hundred page spec/standard docs) and ask it questions on the topic?
- reqo 3y agoI believe RAG is more appropriate for this! While you can certainly fine tune on a pdf, you are essentially fine turning with batch size == 1, so you should not expect good results! Also you need a label (for example summary) in order to fine-tune!
- m3kw9 3y agoDoesn’t really follow instructions too well, if you ask it to list 10 songs on or 5 things it’s give you way more. I’m not sure why some models do it well like Mistral instruct v1, ChatGPT 3.5/4 but here it extremely verbose and it outputs like a short circuited robot
- coder543 3y agoThey released a base model. It is not instruction-tuned, so it won't really follow instructions unless you fine-tune it to do that. "There are lots of Mistral fine-tunes. Why another one? A very healthy ecosystem of Mistral fine-tunes already exists, but they’re typically optimized for direct use. We wanted something different — a model optimized to be the strongest base model for further fine-tunes to be built on."
- m3kw9 3y agoThen how come the base model can somewhat follow instructions but not very well, or why is it that the base model won’t follow instructions well?
- coder543 3y agoBase models are just trying to autocomplete the input text. The most logical completion for an instruction is something approximately like what you asked, but base models are raw. They have not been taught to follow instructions, so they generally do a poor job. They're especially bad at knowing when to stop, and they will often generate their own questions to answer, which they will then answer, followed by more questions and more answers. When chat models are trained, they are first pre-trained (the "PT" in "GPT"), which creates a base model, then they are "fine tuned" (RLHF, aligned, whatever you want to call it). A base model can be fine tuned with an instruction dataset (like OpenOrca[0]) to learn how to follow instructions or how to chat. It can also be fine-tuned with a collection of any inputs and the expected outputs, and learn how to do that specific task. OpenPipe appears to specialize in fine-tuning base models for specific applications. They wanted a better base model. If you want it instruction-tuned, I'm sure they would be happy to help with that, or you can wait for someone in the community to make one of those from their base model... but I believe the whole point of the article is that a small, specialized model can outperform a large, general model. Their goal does not seem to be to build a tiny, general, chat-tuned model that outperforms GPT-4 in everything. They want you to train the base model on a very specific task, with the expectation that it will outperform GPT-4 and be tremendously cheaper to run at the same time. Many LLM tasks are centered around summarization, extraction, or classification, which have nothing to do with chatting. [0]: https://huggingface.co/datasets/Open-Orca/OpenOrca https://huggingface.co/datasets/Open-Orca/OpenOrca
- m3kw9 3y agoI’m really struggling to find a use case for these local models when even ChatGPT 3.5 can do it as good as any of them so far.
- coder543 3y agoThe article shows (fine tuned) Mistral 7B outperforming GPT-4, never mind GPT-3.5.
- m3kw9 3y agoThis model is not close to even 3.5 from when I used it. It first of all does not follow instructions properly and it just runs on and on
- deleted 3y ago[deleted]
- coder543 3y agoWhat you're describing is the behavior you get from any base model that has not been instruction-tuned. The article is clear that this model is not for "direct use". It needs tuning for a specific application.
- m3kw9 3y agohow does one fine tune it to follow instructions? I would have thought they have open source training set for these instruction-follow fine tunes?
- wongarsu 3y agoNot everyone wants to send all their data to OpenAI or Microsoft. Sometimes it isn't legally possible even if you want to. And not every use-case is blessed with a permanent internet connection. And for some use-cases, the "alignment" work on GPT 3.5 and 4 gets more in the way than it helps (even OpenAI admits that alignment makes the model perform worse, even on generic benchmarks).
- Havoc 3y agoNot sure what the point is? It’s well know that small fine tunes outperform big models for specific tasks. But unless my task happens to be something similar to what was tested and fine tuned here it doesn’t really help?
- emporas 3y ago>Model merging is, to me, one of the most counterintuitive empirical results in modern deep learning. It turns out that you can actually more-or-less naively merge the weights of two different models and produce a new one that captures some or all of the abilities of its parents! I would hope the article give some more details on model merging. Is it merging two different fine-tuned models, one fine-tuned on dogs, another fine-tuned on cats, and the merging of the two different models is good on cats and dogs as if by magic? Like fine-tune one model just on Python and test it thoroughly, fine-tune one on Java and test it thoroughly, and then if the need arises for a project that uses both Java and Python, merge the two together and use that. If there is no need for Java, use the one fine-tuned just on Python. Pretty magical indeed! Let alone the fact, that a separate smaller model of half a billion parameters could figure out how to merge the two together. If the cost of LMs could be reduced by a factor of 100, why not reduce it by a factor of 1000?
- airgapstopgap 3y agoThis is not so surprising if you consider the fact that finetuning is extremely sparse and barely imparts any new knowledge to the model. The paper "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch"[1] made this clear: > We initially demonstrate that SFT LM (either encoder- or decoder-based) always tends to acquire excessively redundant delta parameters. To be specific, we present DARE, which randomly resets some delta parameters to zeros based on a drop rate p and subsequently scales the remaining parameters by a factor of 1/(1 − p). Despite its simplicity, with the assistance of DARE, when the LM model parameters reach 70 billion, we can eliminate up to 99% delta parameters with minimal impact on model performance (see Figure 1(a)). The more parameters the LM has, the larger p it can tolerate. This discovery suggests that SFT LM indeed learns a multitude of low-rank structures akin to LoRA [25] Insofar as those adaptations are mostly distinct, you can just preserve both sets and that's what explains successes of merging, I guess. 1. https://arxiv.org/abs/2311.03099 https://arxiv.org/abs/2311.03099
- refulgentis 3y agoFunnily enough, and not so concidentally, this has been well known in practice by...drumroll please...America's greatest innovators, the Adult Entertainment Hobbyists. It doesn't have order-of-magnitude, or I'd even wager 50%, benefits in enabling smaller models. But you nailed it exactly. Fine tune on dogs, fine tune on cats, then...just...average the weights. And you have something better than the original with minimal loss from finetuning. LoRA's end up being more popular for that use case because they're easier to combine and mix, match, and scale. Model merging is still a key technique for a successful base model.
- danielhanchen 3y agoIf anyone wants to finetune their own Mistral 7b model 2.2x faster and use 62% less memory - give our open source package Unsloth a try! https://github.com/unslothai/unsloth https://github.com/unslothai/unsloth thanks! :)
- timthelion 3y agoI wonder if model merging allows for peer to peer collaborative learning ala folding at home. I also wonder if it can have protective effects against catasrophic forgetting similar to the way DNA is merged in sexual reproduction.