3 ms·
> weren't flexible enough for common scenarios Does openAI support that stuff, or is that a more general MLOps (or whatever you'd call it) list of functionali
by andy99 3y ago
> weren't flexible enough for common scenarios
Does openAI support that stuff, or is that a more general MLOps (or whatever you'd call it) list of functionality you need? The API call aspect seems like a pretty easy part that's not a barrier to switching models. It's whatever customizations for a given application that I'd expect to be more work. But re your text I quoted, I'm always wary of "one line of code" solutions that are harder to customize than it would be to just write the the thing.
- lmeyerov 3y agoopenai and langchain natively support all of that or expose the hooks to do it all that stuff is what we needed to do to support multiple providers vs hard-coding a single one. none of it is specific to our codebase. if we were going to take on a third-party dependency, we'd need it to be serious about this kind of thing, else we have an external dependency to work around on our critical path Edit: As an example, if the switching layer doesn't implement model negotiation and doesn't expose key model details (which vary by model provider service, and often require REST calls to introspect), we can't add model negotiation / retries / etc on top, and would have to edit the innards to enable that, which opens up all sorts of questions
- numlocked 3y agoWhat is “model negotiation” in this context?
- lmeyerov 3y agoEx: An agent pipeline is asked to generate some code, triggers a code agent, and azure openai is hooked up with both primary and fallback service regions + a few deployments of 3.5 turbo, 4, etc. Which model gets used first, second, ...? Why? And for the model abstraction layer, what does it need to expose vs hide?