4 ms·
Yes, but the batch API takes up to 24 hours to respond. We use it, but not for user-facing search queries.
by philip1209 2y ago
Yes, but the batch API takes up to 24 hours to respond. We use it, but not for user-facing search queries.
- rahimnathwani 2y agoI don't think janalalcm is talking about the batch API, but just putting multiple profiles in a single API call. If your prompt is short, it won't make much difference to the cost. But if your prompt is, say, about the same length as a single profile, you could save almost half your inference costs by analysing ten at a time.
- philip1209 2y agoAh I’ll look into that. Thanks.
- alach11 2y agoI don't believe OpenAI supports batch calls for inference... only for embeddings. If you're interested in cost optimization, however, you're likely better off using Claude 3.5 Sonnet (as a stand-in for gpt-4o) or Claude 3 Haiku (as a stand-in for gpt-3.5-turbo).