3 ms·
We're using PGVector. The way it works is that we use some heuristics to find candidates for your search. That's the 1.4k number you see - and that does take m
by philip1209 2y ago
We're using PGVector.
The way it works is that we use some heuristics to find candidates for your search. That's the 1.4k number you see - and that does take milliseconds. Then, we go through and analyze each candidate individually with an LLM. So, that's 2 minutes that it ran ~1.4k calls to OpenAI.
- janalsncm 2y agoTypically with an LLM you would tokenize strings in a batch and use attention masks to run inference in parallel. OpenAI must have some similar capability. Looks like they have a batch API.
- philip1209 2y agoYes, but the batch API takes up to 24 hours to respond. We use it, but not for user-facing search queries.
- rahimnathwani 2y agoI don't think janalalcm is talking about the batch API, but just putting multiple profiles in a single API call. If your prompt is short, it won't make much difference to the cost. But if your prompt is, say, about the same length as a single profile, you could save almost half your inference costs by analysing ten at a time.
- philip1209 2y agoAh I’ll look into that. Thanks.
- alach11 2y agoI don't believe OpenAI supports batch calls for inference... only for embeddings. If you're interested in cost optimization, however, you're likely better off using Claude 3.5 Sonnet (as a stand-in for gpt-4o) or Claude 3 Haiku (as a stand-in for gpt-3.5-turbo).