3 ms·
It seems you can make it hang if you input something that triggers the LLM's guardrails. For example, I entered 'where they sell drugs,' and the API endpoint hu
by kgeist 2y ago
It seems you can make it hang if you input something that triggers the LLM's guardrails. For example, I entered 'where they sell drugs,' and the API endpoint hung every time I submitted it. However, on a few occasions, it returned a response immediately (with an error in the API response). I suspect there are too many retries if the structured JSON output is not formatted correctly when the guardrails are activated