4 ms·

> I am having trouble understanding what the complaint is here. The appropriate level of due diligence for each LLM model transition is to run your various pro

> I am having trouble understanding what the complaint is here. The appropriate level of due diligence for each LLM model transition is to run your various prompts into the new model and make sure they still produce the correct output; and, if they don't produce good output, to update the prompts so that they continue to produce good output. Just yesterday, I was experimenting with 4o and assumed I could do a flat migration for some work. 4o actually provided worse results - results I explicitly asked to *not* have in my 4 output (and that I didn't have in my 4 output). It's tedious to have to change models after you've already done a proper validation suite against one model. That would be (at least my) complaint. I've even version-stamped the models I use on purpose to avoid surprises.

hit8run 2y ago
At that point why not use your own hosted open source model that is more reproducible for you?
- t-writescode 2y ago
  Cost of maintaining architecture, cost of complexity of internal infrastructure, knowledge level required for self-hosting, complexity of local one-boxing, and a slew of other reasons.
  - xena 2y ago
    Everything's a tradeoff, but it seems that part of the tradeoffs include access to tools critical for your product to function correctly being taken away without a way to get it back. Maybe that can be an acceptable tradeoff, but I'd personally not like living with that.
    - t-writescode 2y ago
      At present, my startup isn't making money - in fact, it's not even released. As a result, I'm trying to prioritize getting it out of the door while still being affordable for myself enough to bootstrap it. To do this, I've made, and continue to make tradeoffs. Among many of the tradeoffs I'm currently making that I intend to resolve ASAP is having OpenAI as a single source of failure. I intend to have some of the other hosted solutions as other options for LLM processing. One of the many, many options that will be considered at that time is self-hosting it, as well, as another option. I've already spent more time than I should perfecting various smaller pieces, increasing reliability, etc. Each time I choose perfection, I lose more time, more runway, more potential market share; and, something I've recently had to learn: Each time I lock myself in a previous step to get that step perfect, I miss the lessons I'm about to have to learn in the next stage of the process, including new issues I'll run into that increase the next step's complexity above my initial estimates. Everything is a tradeoff. Choosing to use a commercially available solution with known and relatively set costs while accepting it may slowly change underfoot (while also knowing I have alternatives I can swap to if an emergency comes up that should only take a little bit to transfer to) is one I've made.
- ComplexSystems 2y ago
  Because open source models aren't as good as GPT-4.
psanford 2y ago
Not directly addressing your point but asking an LLM to not include something in its output often doesn't work well. Its a bit like saying to someone "whatever you do, don't think about elephants."
jdsully 2y ago
4o is pretty bad in my experience. They shouldn't have used the "4.0" naming for it or said its Lite or something. Instead they are trying to market it as roughly equivalent which it definitely is not.

hit8run 2y ago

At that point why not use your own hosted open source model that is more reproducible for you?

t-writescode 2y ago
Cost of maintaining architecture, cost of complexity of internal infrastructure, knowledge level required for self-hosting, complexity of local one-boxing, and a slew of other reasons.
- xena 2y ago
  Everything's a tradeoff, but it seems that part of the tradeoffs include access to tools critical for your product to function correctly being taken away without a way to get it back. Maybe that can be an acceptable tradeoff, but I'd personally not like living with that.
  - t-writescode 2y ago
    At present, my startup isn't making money - in fact, it's not even released. As a result, I'm trying to prioritize getting it out of the door while still being affordable for myself enough to bootstrap it. To do this, I've made, and continue to make tradeoffs. Among many of the tradeoffs I'm currently making that I intend to resolve ASAP is having OpenAI as a single source of failure. I intend to have some of the other hosted solutions as other options for LLM processing. One of the many, many options that will be considered at that time is self-hosting it, as well, as another option. I've already spent more time than I should perfecting various smaller pieces, increasing reliability, etc. Each time I choose perfection, I lose more time, more runway, more potential market share; and, something I've recently had to learn: Each time I lock myself in a previous step to get that step perfect, I miss the lessons I'm about to have to learn in the next stage of the process, including new issues I'll run into that increase the next step's complexity above my initial estimates. Everything is a tradeoff. Choosing to use a commercially available solution with known and relatively set costs while accepting it may slowly change underfoot (while also knowing I have alternatives I can swap to if an emergency comes up that should only take a little bit to transfer to) is one I've made.
ComplexSystems 2y ago
Because open source models aren't as good as GPT-4.

psanford 2y ago

Not directly addressing your point but asking an LLM to not include something in its output often doesn't work well. Its a bit like saying to someone "whatever you do, don't think about elephants."

jdsully 2y ago

4o is pretty bad in my experience. They shouldn't have used the "4.0" naming for it or said its Lite or something. Instead they are trying to market it as roughly equivalent which it definitely is not.