9 ms·
Generating from a finetuned GPT 3.5 Turbo is 8x the cost of generating from the base model, so you really have to be in the “reduce prompt size by 90%” bucket t
by minimaxir 3y ago
Generating from a finetuned GPT 3.5 Turbo is 8x the cost of generating from the base model, so you really have to be in the “reduce prompt size by 90%” bucket they mention to get cost effectiveness out of it.
- lamroger 3y agoi didn't catch that but getting consistency is probably worth it in most cases. depends on your budget
- ChrisCinelli 3y agoI wonder why is much more expensive.
- GaggiX 3y agoThey would need to store and load the model, even if I imagine that they are using something similar to LoRA to finetune their models.
- swyx 3y agoi would guess that the ideal price is also to raise the charge to make finetuning a last resort rather than a first resort; its probably much better cost- and research-wise if everybody just prompts the same model than silo off in their own minimodels.
- sebzim4500 3y agoI don't think I'd consider it a 'last resort', since a lot of people will be choosing between finetuned GPT-3.5 and non-finetuned GPT-4, in which case finetuning is the cheap option.
- danielbln 3y agoIf you do a lot of retrieval augmentation, then the 8x cost might still be cheaper than burning a lot of tokens on injected context.
- BoorishBears 3y agoFine tuning is not a replacement for retrieval augmentation. If you wanted to do that you'd be better off fine-tuning BERT to retrieve data more effectively (identify key points, smarter summarization, cross-encoder relevancy, etc)
- danielbln 3y agoCould you elaborate? It may not be a replacement, but I can think of a few use cases where RAG could be avoided entirely by having the model be fine tuned on the right dataset.
- minimaxir 3y agoThere is a risk of (greater) hallucination if there aren't explicit sample inputs provided by RAG.
- huac 3y agoFinetuning teaches skills. RAG provides 'facts.'
- lukev 3y agoHallucination, but also the ability to cite a source. A model using RAG can tell you why it answered a certain way, and cite chapter and verse from the underlying data. That's impossible just from model weights.
- stavros 3y agoSpeaking of RAG, does anyone know of a Python library that (hopefully) doesn't need a server, that can do embeddings, along with an example? I want to do RAG in Python, and I can wrangle the prompt to where I want it, but I have no idea how to basically search for relevant documents with embeddings.
- alooPotato 3y agoI think you have to compare it to your alternative which is prob gpt4 in which case its much cheaper?
- minimaxir 3y agoCorrect, GPT-4 is 20-30x the cost compared to GPT-3.5-turbo, but that's more in another class entirely. IMO a finetuned-GPT-3.5-turbo won't beat base GPT-4 for the use cases where you really really need GPT-4, but that remains to be seen. The real alternative to a finetuned GPT-3.5-turbo is still the base model with a very very complicated system prompt.
- josephg 3y agoFrom the article: > Early tests have shown a fine-tuned version of GPT-3.5 Turbo can match, or even outperform, base GPT-4-level capabilities on certain narrow tasks. It sounds like it really depends on what you're doing.
- bookmark1231 3y agoGiven that GPT4 is essentially 16 different ensembled GPT3.5s[1] that’s not too surprising [1] https://www.semianalysis.com/p/gpt-4-architecture-infrastructure https://www.semianalysis.com/p/gpt-4-architecture-infrastruc...
- Xenoamorphous 3y agoFrom https://twitter.com/OfficialLoganK/status/1694062949177139610 https://twitter.com/OfficialLoganK/status/169406294917713961... Early testers have reduced prompt size by up to 90% by fine-tuning instructions into the model itself, speeding up each API call and cutting costs. I wonder if that 90% is precisely due to the calculation you made.
- MPSimmons 3y agoI'm out of this particular loop. What does 'reduce prompt size' mean in this case? What is the advantage?
- napworth 3y ago$$$
- sbierwagen 3y agoIf you've got a high context task, such that it requires lots of explanation in the question, you can either have a very long prompt, or customize the model so the context is included and the prompt can be shorter.
- simonw 3y agoOpenAI models charge by the token (a token is usually about 3/4 of a word). A short prompt - "Say hi" - costs less money than a long prompt: "Five cute names for a pet weasel". You're charged for both the input and the output tokens: https://openai.com/pricing https://openai.com/pricing So for a fine-tuned model that costs 8x more to be a purely financial win, it would need to let you reduce the size of your input and output prompts by 8x or more. (I couldn't resist trying that pet weasel quote against Llama 2 13B running on my laptop with https://llm.datasette.io/ https://llm.datasette.io/ and I had to tiny-jailbreak-it because it didn't think weasels were appropriate pets: https://gist.github.com/simonw/8b935c2034fb69a9ecb52205ae342047 https://gist.github.com/simonw/8b935c2034fb69a9ecb52205ae342... )
- notatoad 3y agothat is actually hilarious. i've been enjoying ChatGPT instead of stack overflow precisely because it avoids that sort of judgemental answer - the AI just answers my question no matter how dumb it is, whereas stackoverflow will tell me the question is dumb whether it is or not. it's funny to see that llama has replicated stack overflow's judginess.
- e-clinton 3y ago
- MuffinFlavored 3y agowhat is a real world use case example of how to finetune, what to feed it during fine tuning, and then how to interact with the model post-fine tuning differently than if you hadn’t fine tuned it?
- rsiqueira 3y agoThis means that the cost would be more than US$ 1 for each 16 user interactions, using this fine-tuning pricing: 16 interactions * ($0.012*4 input + $0.016 output) = US$ 1.02 It's 8x more expensive, indeed. I'm comparing with my use case, the standard gpt-3.5 API, where my users consume 4k input tokens (due to context plus chat history) and almost 1k output tokens.
- msp26 3y ago8x is a big gap. I feel that it might be better to just use few-shot prompting and supply some examples (not applicable to every use case ofc) every prompt to steer the output. But it's worth experimenting with at least. Edit: no function calling (until later this fall) too. That's most of my usage right now so I'll pass. Curious on what the verdict on the OAI finetunes are. I remember reading this thread which indicated it wasn't really necessary: https://news.ycombinator.com/item?id=37174850 https://news.ycombinator.com/item?id=37174850
- jondwillis 3y agoThat’ll add latency.
- explaininjs 3y agoand reduce maximum output length
- sheepscreek 3y agoThere’s a place for this. Like fine tuning the model on advanced particle physics so that it can decipher cutting edge research papers more accurately, without teaching the model particle physics every time in the prompt.
- drcode 3y agoUsing prompts is error-prone, especially if the prompts are complicated. You will likely be able to get better accuracy using fine tuning, because it's a "show, don't tell" process. I think gpt4 with fine tuning, used in a specialized domain with good examples, will be extremely powerful, much more powerful than just gpt4+prompts
- robertnishihara 3y agoI think of fine-tuning as an avenue to significantly reduce LLM inference costs, so I think this is an exciting development. You're right if you compare GPT-3.5-turbo to fine-tuned GPT-3.5-turbo, but if it's anything like fine-tuning the Llama-2 models, you'll be able to achieve GPT-4 level performance for a wide range of practical use cases (SQL query generation is an example), but probably not for math or coding (at least not without fine-tuning on a significant amount of data). In fact, we've seen GPT-4 level performance from even the 7B Llama-2 model after fine-tuning. [1] [1] https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...