4 ms·

Curiously, the express model ends up being a more clean style when creating long lived writer streaming endpoints, which is becoming much more popular with LLM’

Curiously, the express model ends up being a more clean style when creating long lived writer streaming endpoints, which is becoming much more popular with LLM’s. All that was old will be new again.

bhouston 2y ago
Are they using SSE? Or web sockets?
- aphantastic 2y ago
  The standard set by openai and shared by everyone else is to use SSE when “stream: true” is set. Personally I like to strip all the excess data as soon as possible and expose the raw response as a plain “transfer-encoding: chunked” stream. That way the client can decide if they want to accept each chunk as they come or simply await the whole response with limited code change. Either way, being able to easily pass the response writer around and set headers, write, close, as needed without being tied to a single function’s control flow is nice. It can be done in the more modern frameworks ofc, but ends up being a bit more machinery required at each endpoint.
  - deleted 2y ago
    [deleted]
nailer 2y ago
A generator function that yields responses would fit that scenario perfectly.
- aphantastic 2y ago
  Not that perfectly. You still have the question of how to represent the one-time header data.
  - nailer 2y ago
    I'm imagining: yield the headers (Content-Type: text/event-stream) the first time, yield the newline-formatted event streams on subsequent yields. I haven't done it though, I used to really like SSE (it's a little more REST-y) but the world seems to have decided on websockets.
    - johnny22 2y ago
      I think that's how it works with nestjs's observer setup with rxjs. I just return the SSE stream and it works. I use it with fastify, not express.
    - aphantastic 2y ago
      That could work. Probably it’d be a very easy bit of middleware/express extension to write, it’s basically just wrapping the existing writeHeader, write, end, in the hot new syntax. Probably you’d get some benefits in composability, and lose some performance in the extreme.

bhouston 2y ago

Are they using SSE? Or web sockets?

aphantastic 2y ago
The standard set by openai and shared by everyone else is to use SSE when “stream: true” is set. Personally I like to strip all the excess data as soon as possible and expose the raw response as a plain “transfer-encoding: chunked” stream. That way the client can decide if they want to accept each chunk as they come or simply await the whole response with limited code change. Either way, being able to easily pass the response writer around and set headers, write, close, as needed without being tied to a single function’s control flow is nice. It can be done in the more modern frameworks ofc, but ends up being a bit more machinery required at each endpoint.
- deleted 2y ago
  [deleted]

nailer 2y ago

A generator function that yields responses would fit that scenario perfectly.

aphantastic 2y ago
Not that perfectly. You still have the question of how to represent the one-time header data.
- nailer 2y ago
  I'm imagining: yield the headers (Content-Type: text/event-stream) the first time, yield the newline-formatted event streams on subsequent yields. I haven't done it though, I used to really like SSE (it's a little more REST-y) but the world seems to have decided on websockets.
  - johnny22 2y ago
    I think that's how it works with nestjs's observer setup with rxjs. I just return the SSE stream and it works. I use it with fastify, not express.
  - aphantastic 2y ago
    That could work. Probably it’d be a very easy bit of middleware/express extension to write, it’s basically just wrapping the existing writeHeader, write, end, in the hot new syntax. Probably you’d get some benefits in composability, and lose some performance in the extreme.