4 ms·
How streaming LLM APIs work
- bionhoward 2y agoLooks like a ton of wasted data on extraneous fields
- CharlieDigital 2y agoYou can exploit this and build your own stream of streams for interesting use cases: https://chrlschn.dev/blog/2024/05/need-for-speed-llms-beyond-openai-w-dotnet-sse-channels-llama3-fireworks-ai/ https://chrlschn.dev/blog/2024/05/need-for-speed-llms-beyond... (screen in action: https://chrlschn.dev/img/need-for-speed/generation-example.gif https://chrlschn.dev/img/need-for-speed/generation-example.g...) Most interesting is combining this with web components and having GPT directly output streams with small web components.
- jackmpcollins 2y agoThat gif is really cool! I built a Python package magentic [0] which similarly parses the LLM streamed output and allows it to be used before it is finished being generated. There are plenty of use cases / prompts that can be refactored into a "generate list, then generate for each item" pattern to take advantage of this speedup from concurrent generation. [0] https://magentic.dev/streaming/#object-streaming https://magentic.dev/streaming/#object-streaming
- Yanael 2y agoI have been working with streaming LLMs and Server Sent Events. It provides a very simple interface to work with, but you can feel SSE was never designed for this use case. As mentioned in the blog post: > Annoyingly these can't be directly consumed using the browser EventSource API because that only works for GET requests, and these APIs all use POST. It is not designed to send data to open a connection. You will then struggle to work with this streaming approach using frameworks and libraries that are based on the EventSource API.
- ekojs 2y agoEventSource is really really limited. However, you can instead use Fetch via something like https://github.com/Azure/fetch-event-source https://github.com/Azure/fetch-event-source to consume SSEs.
- Yanael 2y agoThis looks very good. The Fetch API is a nice one, so leveraging it sounds perfect. Thanks for the link.
- deleted 2y ago[deleted]
- vintagedave 2y agoHere’s a bit more info on generating streams like this: https://parnassus.co/building-a-copilot-1-server-fundamentals/#wsgi-asgi-what https://parnassus.co/building-a-copilot-1-server-fundamental... I’m slowly building a copilot stack, and end up wrapping multiple layers of streaming: SSE as in this article, parsed on the fly as it streams from JSON (ie parsi g incomplete invalid JSON), parsed on the fly as it streams to extract Markdown, parsed on the fly as it streams to format that Markdown and render it. You can read about this here: https://parnassus.co/building-a-copilot-2-parsing-converting/ https://parnassus.co/building-a-copilot-2-parsing-converting...
- threecheese 2y agoInteresting subject, but came here to comment that you are “doing the lord’s work” by writing an LLM tool for Delphi developers. All six of them! (i kid) Best of luck with Owl.
- vintagedave 2y agoThankyou! I plan to expand it to other under-served languages. Delphi is a fun starting one :)
- Yanael 2y agoWhen you ask to return JSON data using streaming, you will notice that the response is incomplete and unparseable by JSON libraries, resulting in malformed errors. You will have to wait for the entire stream to complete. To solve this problem I tried to define a spec and built a lib for it: - [lib] https://github.com/st3w4r/openai-partial-stream/tree/main https://github.com/st3w4r/openai-partial-stream/tree/main - [spec] https://github.com/st3w4r/openai-partial-stream/blob/main/spec/THE_SPEC.md#introduction https://github.com/st3w4r/openai-partial-stream/blob/main/sp...
- vintagedave 2y agoVery interesting. I tried to solve this problem too, and my code parses incomplete JSON allowing partial values and fully complete values to be accessed. Why do you wait for the entire stream to be complete? Some objects in the JSON structure can be shown to be complete before the stream ends.
- Yanael 2y agoYeah, it's an interesting problem to solve. The library is designed to parse incomplete json without waiting for the stream to finish.
- simonw 2y agoI’ve been using the ijson Python library for that - I have notes on that here: https://til.simonwillison.net/json/ijson-stream https://til.simonwillison.net/json/ijson-stream
- Yanael 2y agoNice, it looks like a good library to build on top of. I like the available events: start_map, end_map, etc. I did try a library in JS that had similar ones, but it lacked the granularity to cover all use cases for individual fields instead of an entire item. I'll keep a note of this one if I do Python JSON streaming.
- jackmpcollins 2y ago
- ekojs 2y agoI do actually wonder if it's more efficient to use something like MessagePack instead of using JSON. It's a lot of strings so it may not matter too much I guess.
- brrrrrm 2y agoshould really be titled streaming output, as full duplex streaming isn't mentioned at all. that'd be necessary for things low latency things like speech etc.
- simonw 2y agoDo you know of any public APIs from LLM vendors that do that? As far as I know the ChatGPT voice chat API isn’t public.
- deleted 2y ago[deleted]
- mmoustafa 2y agoOpenAI streaming has many peculiarities at production scale. e.g. you will get “half-chunks” occasionally which are not parseable on their own and must be concatenated with the previous or subsequent chunk for parsing.