12 ms·
Ask HN: People who switched from GPT to their own models. How was it?
How was your experience in switching from GPT to a custom model for a production use case ?
- ParetoOptimal 3y agoFor personal use I tried this. I no longer use llms.
- geoduck14 3y agoI fine-tuned an LLM to do technical stuff. It works pretty darn good. What I actually discovered is that when evaluating LLMs, it is surprisingly difficult to evaluate them. And, also, that GPT 4 isn't that great, in general.
- maxlamb 3y agoWhich model did you start with?
- geoduck14 3y agoLlama2 Code
- avinassh 3y agoWhich base model did you use and what methodologies you used?
- m3kw9 3y agoLikely mistral
- alwinaugustin 3y agoCould you provide more details on this matter? Specifically, I'm interested in knowing which base model you've utilized and the approach you've taken to fine-tune it. Your insights would be greatly appreciated and highly beneficial.
- m3kw9 3y agoFor narrow stuff you can do better job than base gpt4/mistral/etc model. You fine tune it with your very custom data, stuff that got didn’t seem to be trained on, it will generalize it well.
- simonw 3y agoHave you done this? How did you do it? I've been looking forward to someone providing a detailed guide on how to "fine tune it with your custom data" for ages!
- arbitrandomuser 3y agothis is imo the secret sauce that gives people an edge and not a lot of people will want to reveal
- washadjeffmad 3y agoYou're not wrong. There's been a lot of drama over licensing and releasing datasets, and a lot of the LLM scene are just pitchmen and promoters with no better grasp over what they're doing than "trust me, it's better". Like with "prompt engineering", a lot of people are just hiding how much of the heavy lifting is from base models and a fluke of the merge. The past few "secret" set leaks were low/no delta diffs to common releases. I said it a year ago, but if we want to wowed, make this a job for MLIS holders and references librarians. Without thorough, thoughtful curation, these things are just toys in the wrong hands.
- ipaddr 3y agohttps://www.datacamp.com/tutorial/fine-tuning-llama-2 https://www.datacamp.com/tutorial/fine-tuning-llama-2
- anon373839 3y agoThis is a very nice resource: https://github.com/mlabonne/llm-course https://github.com/mlabonne/llm-course
- deleted 3y ago[deleted]
- visarga 3y ago> GPT 4 isn't that great, in general same here, it doesn't adhere to explicit instructions, maybe one or two simple instructions are ok but not more complex ones
- theshrike79 3y agoYMMV, it depends very heavily on what you're asking it to do.
- exploringBytes 3y agoMore complex instructions should be split into multiple prompts, solving the challenge step by step
- MaxMatti 3y agoWhat model are you using for more complex prompts?
- ninetyninenine 3y agoMaybe the key to a good universal LLM is having multiple fine tuned models for various domains. The user thinks he's querying a single model but really there's some mechanism that selecting the best model for his query out of say like 300 different possibilities. This also helps distributes traffic as a side effect. I guess the problem is how the conversation would flow. If the user changes topics from say art to quantum physics then asks a question about quantum physics and art then I'm not sure what the algorithm should do.
- BillyTheMage 3y agoThe user could talk to an "expert opinion aggregator" model which in turn makes a bunch of queries to specialized models.
- 0x008 3y agoThat is actually the same idea as the (now) popular "Mixture of Experts" approach.
- apetresc 3y ago> This also helps distributes traffic as a side effect. I'm not sure it's "distributing" traffic so much as amplifying it.
- ninetyninenine 3y agoTwo users. One user is talking about physics, the other about art. Two different models are utilized. Load is divided across 2 models. Load balancing is a feature for free and division is across subjects. Of course this is assuming each model owns it's own set of gpus.
- apetresc 3y agoAh okay, I misunderstood. I thought you meant inferencing on all the different models and letting them judge to what extent they're the best fit. What you're suggesting is just simply intent classification and using a specific model per intent. That's what everyone did _before_ LLMs.
- iAkashPaul 3y agoRunning Mistral-Instruct-0.1 for call/email summarization, Mixtral for contract mining & OpenChat to augment agentic chatbot equipped with RAG tools(Instruct again). Experience has been great, INT8 tradeoffs are acceptable until hardware FP8(FP4 anyone?) becomes more widely & cheaply available. On-prem costs have been absorbed already for few boxes of A100s & legacy V100s running millions of such interactions.
- namanyayg 3y agoHey Akash would you mind elaborating about what you mean by contract mining?
- iAkashPaul 3y agoBasically augmenting users parsing PDFs & looking to prefill values into Excel instead of typing it all out. Ex. Liabilities, time-period/frequencies mentioned, owners of clauses etc.
- Klaus23 3y agoWhat are the trade-offs with INT8? I thought even the INT4 loss of accuracy was small and the INT8 loss almost unmeasurable.
- alecco 3y ago> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For some reason it's not used for models (AFAIK). Some people report throttling issues in some cases, though.
- iAkashPaul 3y agoI think PyTorch recently added FP8 & even Intel's Neural-Compressor has experimental support. But yeah most HF LLM models are currently just INT8 or FP4/NF4 loaded thanks to BitsandBytes by Tim Dettmers.
- gokulkrishh09 3y ago[dead]
- jianfgo 3y agoAnyone has a tutorial how to achieve it to own a self-hosted model?
- alwinaugustin 3y agoyou can use Ollama and download many models. Performance depends on your laptop's capacity.
- tuanm 3y agoYou can try going get some pre-trained (sometimes, fine-tuned) models on HuggingFace, following their instructions. Good luck!
- fbdab103 3y agoBit "Draw the rest of the owl" there.
- serf 3y agoit's all pretty well put together nowadays honestly. here's a dead simple way : (1) download LM Studio, install it[0] (2) download a model from within the client when prompted (3) have a ball. the program is fairly intuitive, it takes care of finding the relevant files, and it can even accept addendum prompts and various ways to flavor or specialize answers. Learn the basics there, take what you learn to a more 'industrial' playground later on. [0]: https://lmstudio.ai/ https://lmstudio.ai/
- jassyr 3y agoReddit community r/LocalLlama has great info
- cosmosgenius 3y agoI use lmstudio and continue.dev. Deepseek model usually but i try out other models every now and then.
- fbdab103 3y ago
- MaxLeiter 3y agoMy 2024 prediction is we will see far more people moving off of openai once they encounter its cost and latency compared to (less proven/scaled) competitors. It’s often a speed versus quality tradeoff, and I’ve seen multiple providers 3x faster than OpenAI with far more than 1/3 the quality
- osigurdson 3y agoI greatly prefer to use ChatGPT-4 instead of 3.5 despite the slowness. Really a good feature for them to have would be to easily re-run a prompt on 4. However, the glitchiness of the service is kind of annoying.
- notso411 3y ago[dead]
- arthur_sav 3y agoOpenAI currently leads the front on (commercial) AI, so I doubt people will switch. In fact most offerings become outdated pretty fast and everyone else tries to play catch up. Imagine using a GPT-2 type model when everyone else is using GPT-4. Until the dust settles there's no point in investing in alt models imo, unless you're leading the research.
- zingelshuher 3y agoWhen using it for coding I'd rather have quality than garbage 3x faster and cheaper. So, I'm not moving. For narrow tasks it makes sense. Also interesting is the idea of having several models working together locally. With, may be, occasional ping to GPT-4.
- codybontecou 3y agoI've been using [continue](https://continue.dev/ https://continue.dev/) alongside Ollama. My go-to llm has been [deepseek-coder 7b](https://ollama.com/library/deepseek-coder https://ollama.com/library/deepseek-coder). The setup feels as good as ChatGPT 4, local first, and overall, I enjoy it.
- manca 3y agoI've tried code-llama with Ollama, along with Continue.dev and found it to be pretty good. The only downside is that I couldn't "productively" run the 70B version, even on my MBP with M3 Max with 36GB of RAM (which interestingly should be enough to hold quantized model weights). It was simply painfully slow. 34B one works good enough for most of my use-cases, so I am happy.
- 0x008 3y agoI tried to use codellama 34B and I think it is pretty bad. For Example I asked it to convert a comment into a docstring and it would hallucinate a whole function around it.
- 3abiton 3y agoSame, CodeLlama 70B is known to suck. Deepseek is the best for coding so far in my experience, Mixtral 8x7B is another great contender (to be frank, for most tasks). Miqu is making a buzz, but so far I haven't tested it personally yet.
- gpjt 3y agoWhat quantization were you using? I've been getting some weird results with 34b quantized to 4 bits -- glitching, dropped tokens, generating Java rather than Python as requested. But 7b, even at 4 bits, works OK. Posted about it earlier on this evening: https://www.gilesthomas.com/2024/02/llm-quantisation-weirdness https://www.gilesthomas.com/2024/02/llm-quantisation-weirdne...
- jwells89 3y agodeepseek-coder 6.7b is seriously impressive for how quickly it runs on an M1 Max. There’s a few spots where it still doesn’t fare quite as well as ChatGPT but it’s a small tradeoff considering that it’s fully local and doesn’t even spin up my laptop’s fans.
- theolivenbaum 3y agoWe support both in our app and enterprise product. The APIs (OpenAI) vs libraries (i.e. llama.cpp for on-device) are so similar that the switch is basically transparent to the user. We're adding support for other platforms APIs soon, and everything we've looked so far is as easy to integrate as OpenAI - except Google that for some reason complicates everything on Google Cloud.
- itake 3y agoWe have a first pass with our own model and then escalate to gpt if we aren't sure of our own model's results.
- kgeist 3y agoThe main hurdle is lack of multilingual support in open weights models.
- armini 3y agoWould be great if people could share their app demo, host & model used/trained for better context.
- sgc 3y agoTo add to this question, are there LLMs that I can run on my own data, that also can provide citations similar to the way phind.com does for their results? Even better if they are multilingual.
- lee101101 3y ago[dead]
- wesleyyue 3y agoI tested a bunch of models while building https://double.bot https://double.bot but ended up back on gpt4. Other models are fun to play with but it gets frustrating even if they miss 1/100 questions that gpt4 gets. I find that right now I get more value implementing features around the model that fixes all the GitHub copilot papercuts (autocomplete that closes brackets properly, auto import upon accepting suggestions, disable suggestions when writing comments to be less distracting, midline completions, etc etc) Hopefully os models can catch-up to gpt4 in the next six months when we fixed all the low hanging fruit outside of the model itself
- rukuu001 3y agoIt’s been ok. I’m running llama2 7b and it’s … fine. The results I get from gpt4 aren’t much better. This for general tasks. Mostly I think I need to use LLMs more effectively
- kcorbitt 3y agoObviously talking my own book here, but we've helped dozens of customers make the transition from prompted GPT-4 or GPT-3.5 to their own fine-tuned models at OpenPipe. The most common reaction I get is "wow, I didn't expect that to work so well with so little effort". For most tasks, a fine-tuned Mistral 7B will consistently outperform GPT-3.5 at a fraction of the cost, and for some use cases will even match or outperform GPT-4 (particularly for narrower tasks like classification, information extraction, summarization -- but a lot of folks have that kind of task). Some aggregate stats are in our blog: https://openpipe.ai/blog/mistral-7b-fine-tune-optimized https://openpipe.ai/blog/mistral-7b-fine-tune-optimized
- jonathanleane 3y agoHave you started migrating anyone over to Mistral Large yet?
- kcorbitt 3y agoI don't think they've released a fine-tuning API, but we'll definitely support it once they do!
- ilaksh 3y agoAnd those dozens are all doing narrow tasks like summarization right?
- notso411 3y ago[dead]
- ashu1461 3y agoWere their any tasks like intent / entity detection etc ? - I guess classification / information extraction covers that, but still anything more specific ?
- abdullin 3y agoI prefer to use local models when running data extraction or processing over 10k or more records. Hosted services would be slow and brittle at this point. Mistral 7B fine-tunes (OpenChat is my favorite) just chug through the data and get the job done. Details: using vLLM to run the models. Using ChatGPT-4 to condense information for complex prompts (that the local models will execute). I think, the situation will just keep on getting better with each month.
- weinzierl 3y agoA couple of months ago I attended a presentation of an on-prem LLM. An audience member asked, if it was using OpenAI in any way. The presenter, somewhat overeagerly, "Why not ask our new AI?" and went on to type: "Are you an independent model or do you use OpenAI?" To chat bot answered in flourish language that sure it was using ChatGPT as a backend. Which it was not and which was kind of the whole point of the presentation.
- mdekkers 3y agoNever demo anything unrehearsed, or where you cannot predict the outcome with a very high level of confidence
- WhrRTheBaboons 3y ago> where you cannot predict the outcome with a very high level of confidence so never demo LLMs. got it.
- waynesonfire 3y agoCorrect.. like how google does it https://www.tomshardware.com/news/google-gemini-ai-video-staged https://www.tomshardware.com/news/google-gemini-ai-video-sta...
- hnfong 3y agoIf you tune down the temperature you can get a high level of confidence that the output will be the same.
- deleted 3y ago[deleted]
- seeknotfind 3y agoWhy not? This seems like a great thing to happen. If the author's heart is really in the project, then this is a huge opportunity moment to learn something new.
- 0x008 3y agoMixed results. I think llama2 in general is pretty bad, especially at anything else than english. I've had very good results with Mixtral for Chat. Of course all of them feel like a Frankenstein compared to actual ChatGPT. They feel similar and work just as well until, sometimes, they put out complete and utter garbage or artifacts and you wonder if they skimped on fine-tuning.
- samus 3y agoIn a certain sense, all open models skimp on fine-tuning. Since ChatGPT gets user feedback, OpenAI theoretically sits on a growing pile of data that can be used for continuous fine-tuning and alignment. For local models, you'd have to look for new checkpoints every few months or track good and bad responses and do your own alignment.
- rspoerri 3y agoIm using mixtral 8x7b (q5) for my use cases, such as scripting, searching for ideas and or definitions that i allways need to factcheck. Currently i use lmstudio on my m2 with 96gb ram. But i‘m looking into switchin to ollama or another oss solution.
- adawg4 3y agoMistral 7B was great for flights without wifi! Answers pretty good for information you need to find but its step by step instructions are hit or miss when it tries to do it for you.
- adawg4 3y agoMistral 7B was great for flights without wifi! Answers pretty good for information you need to find but its step by step instructions are hit or miss when it tries to do it for you.