3 ms·
I don't think janalalcm is talking about the batch API, but just putting multiple profiles in a single API call. If your prompt is short, it won't make much di
by rahimnathwani 2y ago
I don't think janalalcm is talking about the batch API, but just putting multiple profiles in a single API call.
If your prompt is short, it won't make much difference to the cost. But if your prompt is, say, about the same length as a single profile, you could save almost half your inference costs by analysing ten at a time.
- philip1209 2y agoAh I’ll look into that. Thanks.
- alach11 2y agoI don't believe OpenAI supports batch calls for inference... only for embeddings. If you're interested in cost optimization, however, you're likely better off using Claude 3.5 Sonnet (as a stand-in for gpt-4o) or Claude 3 Haiku (as a stand-in for gpt-3.5-turbo).