4 ms·

I presume this doesn't support streaming...

• cmdalsanto 2y ago
  Yep, we support both evaluations and autocorrections for streaming as well.
  • internetter 2y ago
    Aren't streams write-only? Can you delete parts of the stream if it is later deemed incorrect somehow?
    • cmdalsanto 2y ago
      There's some secret sauce here, but since we intercept each chunk as the LLM pushes them out, we can perform evaluations on them and decide what gets sent back to the client if we detect a fault.