2 ms·
I think the nature/feel of the quick response is likely due to the fact the completion call is streaming "stream: true" [1]. Then, they are parsing it back to t
by awayto 3y ago
I think the nature/feel of the quick response is likely due to the fact the completion call is streaming "stream: true" [1]. Then, they are parsing it back to text as it streams the response back.
[1] https://github.com/steven-tey/novel/blob/62d32be43dae8aa40fab04c0c40f305646028dda/app/api/generate/route.ts#L67 https://github.com/steven-tey/novel/blob/62d32be43dae8aa40fa...
- LoganDark 3y agoI'd have been more surprised if there wasn't streaming. All the LLMs I have ever used, stream tokens in real-time. I'm talking about the time spent per token.