4 ms·
Typically with an LLM you would tokenize strings in a batch and use attention masks to run inference in parallel. OpenAI must have some similar capability. Loo
by janalsncm 2y ago
Typically with an LLM you would tokenize strings in a batch and use attention masks to run inference in parallel.
OpenAI must have some similar capability. Looks like they have a batch API.
- philip1209 2y agoYes, but the batch API takes up to 24 hours to respond. We use it, but not for user-facing search queries.
- rahimnathwani 2y agoI don't think janalalcm is talking about the batch API, but just putting multiple profiles in a single API call. If your prompt is short, it won't make much difference to the cost. But if your prompt is, say, about the same length as a single profile, you could save almost half your inference costs by analysing ten at a time.
- philip1209 2y agoAh I’ll look into that. Thanks.
- alach11 2y agoI don't believe OpenAI supports batch calls for inference... only for embeddings. If you're interested in cost optimization, however, you're likely better off using Claude 3.5 Sonnet (as a stand-in for gpt-4o) or Claude 3 Haiku (as a stand-in for gpt-3.5-turbo).