4 ms·
OpenAI's own documentation makes note of how difficult it is to work with SSE and to just use their library instead. My team wrote our own parser for these stre
by RevEng 2y ago
OpenAI's own documentation makes note of how difficult it is to work with SSE and to just use their library instead. My team wrote our own parser for these streaming events from an OpenAI compatible LLM server. The streaming format is awful. The double newline block separator also shows up in a bunch of our text, making parsing a nightmare. The "data:" signifier is slightly better, but when working with scientific software, it still occurs too often. Instead we've had to rely on the totally-not-reliable fact that the server returns each as a separate packet and the receiving end can be set up to return each packet in the stream.
The suggestions I've found online for how to deal with the newline issue are to fold together consecutive newlines, but this loses formatting of some documents and otherwise means there is no way to transmit data verbatim. That might be fine for HTML or other text formats where newlines are pretty much optional, but it sucks for other data types.
I'm happy to have something like SSE but the protocol needs more time to cook.
- skue 2y ago> I'm happy to have something like SSE but the protocol needs more time to cook. Just how well done do you like your protocols? SSE has been part of the WHATWG standard for almost 20 years. Every protocol requires some sort of data encoding. For SSE you need to either restrict yourself to payloads that can never conflict with the message structure (e.g. an enumeration of short strings to indicate different sorts of events), or you need to encode the data. It sounds like you are trying to send raw, unencoded data and are surprised that it sometimes conflicts with the message structure. Well of course it does! You can’t blame the protocol for that.
- RevEng 2y agoEvery other protocol I've used has a standard way to encode arbitrary data, but especially text data, usually using some kind of escape sequence. SSE does not. Just because it has been around for a long time does not mean it is well thought out or complete.