10 ms·

The problem seems to be right now that LLMs are 98% predictable. OpenAI functions for example invoke properly 90%+ of the time, but every now again, even with n

The problem seems to be right now that LLMs are 98% predictable. OpenAI functions for example invoke properly 90%+ of the time, but every now again, even with nearly identical input, it will not invoke. These problems become multiplicative, if you have 3-4 chatgpt calls that are chained together your failure rate is closer to 10%. Unfortunately in the health tech space, even 1% failure rate is unacceptable. I have some theories and PoC work I am doing to improve the rate. As with all "AI/ML" projects its not as simple as take problem and apply AI to solve it.

fnordpiglet 3y ago
Can you not simply reattempt? Then 10% failure appears more like volatility in the latency 10% of the time. Of course 1% of the time you have to reattempt twice, .1 three times etc.
- bearjaws 3y ago
  A failure isn't just a 503, it's that instead of producing one of your function calls you specify using OAI functions, it instead produces a chat message. In some contexts it may make sense to have calls that can result in a function call, or just a chat message. One approach I am trying is to reduce the number of arguments, another is to reduce the number of function calls available. Either way, all of this introduces complication to building with OpenAI.
  - fnordpiglet 3y ago
    Ah I see - you don’t know a priori whether a text response is appropriate or a function call. Maybe I’m just being cheeky but what if you had an echo() function and asked that it always put chat responses through the echo function. That way if it breaks protocol you know?
cesarvarela 3y ago
This has been my experience with function calling, and it kind of pushed me into the valley of disillusionment. I'm talking about gpt 3.5 btw, do you have numbers for gpt 4 ?
RA_Fisher 3y ago
Exactly! With even just 3 chained calls, 98% individual chance of failure ^ 3 calls ~ 94.1% joint probability of success or 5.9% failure global rate. That's a failure rate of 1 out of 17. Problems like those and my experience as a reliability statistician are why I built ModelGym: https://modelgymai.com/ https://modelgymai.com/ I want to help LLM app developers deliver on their visions and accelerate the realization of AI's potential. It's free to try. If you have any feedback, I'd love to learn of it.

fnordpiglet 3y ago

Can you not simply reattempt? Then 10% failure appears more like volatility in the latency 10% of the time. Of course 1% of the time you have to reattempt twice, .1 three times etc.

bearjaws 3y ago
A failure isn't just a 503, it's that instead of producing one of your function calls you specify using OAI functions, it instead produces a chat message. In some contexts it may make sense to have calls that can result in a function call, or just a chat message. One approach I am trying is to reduce the number of arguments, another is to reduce the number of function calls available. Either way, all of this introduces complication to building with OpenAI.
- fnordpiglet 3y ago
  Ah I see - you don’t know a priori whether a text response is appropriate or a function call. Maybe I’m just being cheeky but what if you had an echo() function and asked that it always put chat responses through the echo function. That way if it breaks protocol you know?

cesarvarela 3y ago

This has been my experience with function calling, and it kind of pushed me into the valley of disillusionment. I'm talking about gpt 3.5 btw, do you have numbers for gpt 4 ?

RA_Fisher 3y ago

Exactly! With even just 3 chained calls, 98% individual chance of failure ^ 3 calls ~ 94.1% joint probability of success or 5.9% failure global rate. That's a failure rate of 1 out of 17. Problems like those and my experience as a reliability statistician are why I built ModelGym: https://modelgymai.com/ https://modelgymai.com/ I want to help LLM app developers deliver on their visions and accelerate the realization of AI's potential. It's free to try. If you have any feedback, I'd love to learn of it.