5 ms·
Does OpenAI on azure still have that insane latency for content filtering? Last time I checked it added a huge # to time to first token, making azure hosting fo
by com2kid 1y ago
Does OpenAI on azure still have that insane latency for content filtering? Last time I checked it added a huge # to time to first token, making azure hosting for real time scenarios impractical.
- shakna 1y agoYes. Unless you convince MS to let you at the "Provisioned Throughput" model. Which also requires being big enough for sales to listen to you.