3 ms·
What vectordb are you using? guessing you're just slammed with traffic right now, but it says it searched 1.4k records and it took over 2 minutes. Should be ab
by mritchie712 2y ago
What vectordb are you using?
guessing you're just slammed with traffic right now, but it says it searched 1.4k records and it took over 2 minutes. Should be able to run it subsecond.
- philip1209 2y agoWe're using PGVector. The way it works is that we use some heuristics to find candidates for your search. That's the 1.4k number you see - and that does take milliseconds. Then, we go through and analyze each candidate individually with an LLM. So, that's 2 minutes that it ran ~1.4k calls to OpenAI.
- janalsncm 2y agoTypically with an LLM you would tokenize strings in a batch and use attention masks to run inference in parallel. OpenAI must have some similar capability. Looks like they have a batch API.
- philip1209 2y agoYes, but the batch API takes up to 24 hours to respond. We use it, but not for user-facing search queries.
- rahimnathwani 2y agoI don't think janalalcm is talking about the batch API, but just putting multiple profiles in a single API call. If your prompt is short, it won't make much difference to the cost. But if your prompt is, say, about the same length as a single profile, you could save almost half your inference costs by analysing ten at a time.
- philip1209 2y agoAh I’ll look into that. Thanks.
- alach11 2y agoI don't believe OpenAI supports batch calls for inference... only for embeddings. If you're interested in cost optimization, however, you're likely better off using Claude 3.5 Sonnet (as a stand-in for gpt-4o) or Claude 3 Haiku (as a stand-in for gpt-3.5-turbo).