3 ms·

The main problem is actually in making your prompts perform similarly enough with a new model. For sure, switching models is trivial, but it'll disrupt the qual

The main problem is actually in making your prompts perform similarly enough with a new model. For sure, switching models is trivial, but it'll disrupt the quality of your service without significant effort in migrating your prompts. I'm not saying this is an intrinsic moat, just that there's a barrier to change.

mejutoco 2y ago
That same problem exists everytime your model served from a third-party is optimized/lobotomized. Only and offline model guarantees this wont happen.
deepsquirrelnet 2y ago
This demonstrates a need for more robust engineering practices. Typically you would solve this by testing a validation set against a new model. Some tools like DSPy or Agenta are helping to encourage this. But creating good evaluators for generative responses isn’t easy, and in my experience people tend to punt.
- rmbyrro 2y ago
  Prompting is different than programming. Auto-testing your prompts is good practice and helps during a migration. But it doesn't change the fact that the prompts won't work the same in a different model. It's like saying having Python tests eliminates bugs when rewriting a codebase into JavaScript.
drusepth 2y ago
To be fair though, you typically end up doing that process even when you remain within the OpenAI ecosystem, because they're putting out new (cheaper) models. For example, a couple weeks ago I migrated a few game mechanics from GPT-4-Turbo to GPT-4o and it caused significant enough issues that I needed to revert them all until I could go back and retune all the prompts, which ended up taking 2-3 days. That's probably about as much time as it would have taken to tune prompts for any other model, especially since I'm using an API that standardizes format/structure across whatever model I select (as simple as a dropdown) and lets me just focus on prompt messages independent of which LLM they're going to.

mejutoco 2y ago

That same problem exists everytime your model served from a third-party is optimized/lobotomized. Only and offline model guarantees this wont happen.

deepsquirrelnet 2y ago

This demonstrates a need for more robust engineering practices. Typically you would solve this by testing a validation set against a new model. Some tools like DSPy or Agenta are helping to encourage this. But creating good evaluators for generative responses isn’t easy, and in my experience people tend to punt.

rmbyrro 2y ago
Prompting is different than programming. Auto-testing your prompts is good practice and helps during a migration. But it doesn't change the fact that the prompts won't work the same in a different model. It's like saying having Python tests eliminates bugs when rewriting a codebase into JavaScript.

drusepth 2y ago

To be fair though, you typically end up doing that process even when you remain within the OpenAI ecosystem, because they're putting out new (cheaper) models. For example, a couple weeks ago I migrated a few game mechanics from GPT-4-Turbo to GPT-4o and it caused significant enough issues that I needed to revert them all until I could go back and retune all the prompts, which ended up taking 2-3 days. That's probably about as much time as it would have taken to tune prompts for any other model, especially since I'm using an API that standardizes format/structure across whatever model I select (as simple as a dropdown) and lets me just focus on prompt messages independent of which LLM they're going to.