3 ms·
> However, from a code perspective, it is a function of the key and a relatively small API. You're thinking of traditional apps and APIs. In an AI application
by lolinder 3y ago
> However, from a code perspective, it is a function of the key and a relatively small API.
You're thinking of traditional apps and APIs.
In an AI application, most of the work is in prompt engineering, not wiring up the API to your app. Prompts that work well for one model will fail horribly for another. People spend months refining their prompts before they're safe to share with users, and switching platforms will require doing most of that refinement over again.
- edgyquant 3y agoSwitching from one API to another generally requires refactoring. I’ve not had much problems moving between LLMs (openai to Anthropic)
- wokwokwok 3y agoThen you’re either not testing your prompts or doing something trivial. Remember: a good model with a good prompt will generate bad outputs sometimes. A bad model with a bad prompt will generate a good output sometimes. That is simply a fact with these non deterministic models. You have to do many iterations for each prompt to verify they are working correctly. > I’ve not had much problems moving between LLMs… If you want to move your prompts to a different model, you’re effectively replacing one: f(prompt + seed) => output With different black box implementation. Unless you’re measuring the output over multiple iterations of (seed) and verifying your prompt still does the right thing, it’s actually very likely that what you’ve done if take an application with a known output space and converted it to an application with an unknown output space… …that partially overlaps the original output space! So it looks like it’s the same. …but it isn’t, and the “isn’t” is in weird edge cases. Unless you’re measuring that, you simply now have an app that does “eh, who knows?” So yes. Porting is trivial if you don’t care if you have the same functionality. …but reliably porting is much harder (or longer).
- esafak 3y agoHow many people are even writing tests for these things?
- dbmikus 3y agoVery few. Many deployed apps don't have a good quantitative grasp of the quality of their LLMs. Some are doing testing or evaluation, through things like unit tests, A/B testing different prompts, collecting user feedback. I think we're exiting the phase where people can launch an AI app and have people use it just because of the initial "wow factor" and moving into the phase where users will start churning and businesses will need to make sure that their AI agent is performing and they they understand how well it's performing.
- wizzard0 3y ago> many iterations of each prompt BTW its much faster and cheaper to artive at a good prompt if you sample the model in deterministic mode (ie temperature=0) By default you have to guess if the difference is due to the prompt change or due to the dice roll, as you’ve noticed, but you don’t need to!
- wokwokwok 3y agoYou should have a read of https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate (the section on sampling, with regard to setting temperature to zero). This is degenerate (greedy) behaviour, and not representative of the what the prompt will behave like at a higher temperature. (At least, that’s my understanding; it’s a complex topic but broadly speaking there no specific reason, as far as I’m aware, to expect that a particular combination of params/prompt is representative of any other combination of params/prompt for the same model; it may be, but it may not. Certainly on models like GPT4 it is not, for reasons that are not clear to anyone. So… take care with your prompt testing. setting temperature to 0 is basically meaningless unless you expect to use a temperature of 0 in production. The results you get from your prompts at temp 0 are not generally reflective of the results you will get at temp > 0).
- Aeolun 3y agoI’d be more worried about this if OpenAI had a track record of increasing prices, but the opposite happens. I get more for the same price basically every 6 months.
- dom96 3y agoSure, they are decreasing the prices right now. But once it comes time for them to become profitable they can easily reverse course.
- jairuhme 3y agoI'd imagine that will start to switch back the other way at some point. Decrease prices to gain market share and get you locked in, then increase prices to earn more money to keep VC's happy
- Rastonbury 3y agoStill moving between models is less arduous than switching cloud providers, depending on use case and price difference of course. Most models hold GPT4 as the benchmark they aspire to and should converge to its capabilities.
- lolinder 3y agoIt's not a question of converging on its capabilities, it's a question of responding equivalently to the nuances of the prompt you've crafted. Two models can be equally capable, but one might interpret a phrase in your prompt slightly differently.