3 ms·
latency = f(fit_quality, fit_algorithm) An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve tha
by fwungy 3y ago
latency = f(fit_quality, fit_algorithm)
An LLM can be thought of as a curve fitting function where the query is a set of points and the output is the curve that runs through those points with the minimum of error.
You can increase a fit by increasing the number of variables in the fit function. In statistics this is can lead to overfitting, which will lead to an imprecise model, but in an LLM overfitting is a good thing, because the modeled domain in specific to the set of prompts.
But that comes with a cost, compute power and latency. I suspect what's happening is they're turning down the fit quality to save money as demand increases. Most people aren't paying for AI access. As a side effect it's also lowering the hype factor that was attracting unwanted regulatory attention.
- n4te 3y agoExactly, this is what it feels like to me. Sometimes it feels like I get more of it's attention (processing power) and the answers are good and correct. Other times it makes many mistakes easily. I would pay more to get more processing power for my questions.
- afro88 3y agoHow do you turn down the fit quality of a static model?
- fwungy 3y agoReduce the parameter count. Increase the acceptable error. The more parameters on a curve fit the better the fit will be, but the compute power increases too.
- afro88 3y agoIs that possible? I thought parameter counts were fixed in the model.
- fwungy 3y agoThere are always ways to trade precision for speed in computer statistics models.
- afro88 3y agoSure, generally speaking. Is that true for static, fixed parameter count LLMs like GPT4? I think you're hand waving a lot just to claim that OpenAI are (somehow) reducing accuracy of their models during high load. And I'm not sure why.