4 ms·
One thing I love to see here, mainly because I was going to make it myself, is the chart on latency. GPT4 is practically unusable unless you spend $10k+ a mont
by bearjaws 3y ago
One thing I love to see here, mainly because I was going to make it myself, is the chart on latency.
GPT4 is practically unusable unless you spend $10k+ a month and have an enterprise account.
No real end user wants to wait 20-40 seconds for a response, only to find it was 80% of what they wanted.
- swyx 3y ago> ...our experiment showed that fine-tuned GPT-3.5 models were significantly faster than [base GPT-3.5] by 3.6 to 3.76 times. does this pass the sniff test? under what situations could a finetuned single tenant model be faster than a batchable multitenant base model?
- gnahzby 3y agoThis result really surprised us too, but it seems like we were not alone in observing this: https://medium.com/@gabrielgrinberg/openai-api-fine-tuned-gpt-3-5-vs-base-gpt-3-5-a-case-study-f3619b4f8cd8 https://medium.com/@gabrielgrinberg/openai-api-fine-tuned-gp... That being said, both accounts that we experimented on were personal accounts ( with a very limited budget) so maybe there are some special behaviors (e.g. some rate limit that only exists for base models)
- eqian99 3y agoYeah we were definitely surprised! This was a small scale experiment; we uploaded our notebook for latency tests here if anyone is interested taking a look at the code / trying it their own finetuned models: https://github.com/eqian99/finetuning-experiment https://github.com/eqian99/finetuning-experiment
- dnnssl2 3y agoUnder the same conditions where enterprise versions of the API have significantly less latency and better reliability than personal. OpenAI can change anything about the underlying infrastructure.