6 ms·

This feels ill advised and I don't believe that HTTP streaming was designed with this pattern in mind Perhaps I'm wrong, but I believe HTTP streaming is for ch

This feels ill advised and I don't believe that HTTP streaming was designed with this pattern in mind Perhaps I'm wrong, but I believe HTTP streaming is for chunking large blobs. I worry that if you use this pattern and treat streaming like a pub/sub mechanism, you'll regret it. HTTP intermediaries don't expect this traffic pattern (e.g., NGINX, CloudFlare, etc.). And I suspect every time your WiFi connection drops while the stream is open, the fetch API will raise an error as if the request failed. However, I agree you probably don't need WebSockets for many of the ways they're used—server-sent events are a simpler solution for many situations where people reach for WebSockets... It's a shame SSEs never received the same fanfare.

hobofan 1y ago
With the current AI/LLM wave SSE have received a lot of attention again, and most LLM chat frontends use them. At least from my perception as a result of this, support for SSEs in major HTTP server frameworks has improved a lot in the last few years. It is a bit of a shame though, that in order to do most useful things with SSEs you have to resort to doing non-spec-compliant things (e.g. send initial payload with POST).
- blensor 1y ago
  Also MCP uses it
- ljm 1y ago
  Same with graphql subscriptions. Arguably it’s also because of serverless architecture where SSE can be used more easily than WS or streaming. If you want any of that on Lambda and API Gateway, for example, and didn’t anticipate it right off the bat, you’re in for quite a bit of pain.
- nkozyra 1y ago
  SSE limitations in the browser are still a drag for this, too.
skrebbel 1y ago
> I don't believe that HTTP streaming was designed with this pattern in mind > server-sent events are a simpler solution Fwiw Server-Sent Events are a protocol on top of HTTP Streaming. In fact I'm somewhat surprised that the article doesn't mention it, instead rolling their own SSE alternative that looks (to my non-expert eyes) like a lower level version of the same thing. It seems a bit weird to me to use chunks as a package boundary, I'd worry that that has weird edge cases (eg won't large responses be split into multiple chunks?)
- notjoemama 1y ago
  Because of TCP, large chunks are always split into smaller chunks. It’s just that at the HTTP level we don’t know and don’t see it. UDP forces people into designing their own protocols if the data is a defined package of bytes. Having done some socket coding my impression is web sockets would be good for a high end browser based game, browser based simulations, or maybe a high end trading system. At that point the browser is just a shell/window. As others have pointed out, there are already plenty of alternatives for web applications.
  - jeroenhd 1y ago
    The problem for things like video games and trading is that websockets only support TCP by default. Technologies like WebRTC allow for much faster updates. I think websockets certainly have their uses. Mostly in systems where SSE isn't available quickly and easily, or when sending a bunch of quick communications one after another as there's no way to know if the browser will pipeline the requests automatically or if it'll set up a whole bunch of requests.
- benwilber0 1y ago
  I pretty much always prefer SSE over websockets just because of the simplicity end-to-end. It's "just HTTP", so all the HTTP-based tech and tools apply out-of-the-box without any really special configuration that is required for WS. Curl (or even netcat) "just works", no special client. I don't have to do any special CDN configuration to proxy connections or terminate SSL aside from just turning off buffering. Websockets requires almost a completely new L7 stack and tons of special configuration to handle Upgrade, text or data frames, etc. And once you're out of "HTTP mode" you now have to implement the primitive mechanics of basically everything yourself, like auth, redirects, sessions, etc. It's why I originally made Tiny SSE which is a purpose-built SSE server written in Rust and programmable with Lua. https://tinysse.com https://tinysse.com https://github.com/benwilber/tinysse https://github.com/benwilber/tinysse
osigurdson 1y ago
The issue I have with SSE and what is being proposed in this article (which is very similar), is the very long lived connection. OpenAI uses SSE for callbacks. That works fine for chat and other "medium" duration interactions but when it comes to fine tuning (which can take a very long time), SSE always breaks and requires client side retries to get it to work. So, why not instead use something like long polling + http streaming (a slight tweak on SSE). Here is the idea: 1) Make a standard GET call /api/v1/events (using standard auth, etc) 2) If anything is in the buffer / queue return it immediately 3) Stream any new events for up to 60s. Each event has a sequence id (similar to the article). Include keep alive messages at 10s intervals if there are no messages. 4) After 60s close the connection - gracefully ending the interaction on the client 5) Client makes another GET request using the last received sequence What I like about this is it is very simple to understand (like SSE - it basically is SSE), has low latency, is just a standard GET with standard auth and works regardless of how load balancers, etc., are configured. Of course, there will be errors from time to time, but dealing with timeouts / errors will not be the norm.
- thedufer 1y ago
  I don't understand the advantages of recreating SSE yourself like this vs just using SSE. > SSE always breaks and requires client side retries to get it to work Yeah, but these are automatic (the browser handles it). SSE is really easy to get started with.
  - osigurdson 1y ago
    My issue with eventsource is it doesn't use standard auth. Including the jwt in a query string is an odd step out requiring alternate middleware and feels like there is a high chance of leaking the token in logs, etc. I'm curious though, what is your solution to this? Secondly, not every client is a browser (my OpenAI / fine tune example is non-browser based). Finally, I just don't like the idea of things failing all time with something working behind the scenes to resolve issues. I'd like errors / warnings in logs to mean something, personally. >> I don't understand the advantages of recreating SSE yourself like this vs just using SSE This is more of a strawman and don't plan to implement it. It is based on experiences consuming SSE endpoints as well as creating them.
runeks 1y ago
> Perhaps I'm wrong, but I believe HTTP streaming is for chunking large blobs. You are wrong in the case of Chrome and Firefox. I have tried it and streaming e.g. unordered list elements are displayed instantly. But for Safari, "text/html" streaming happens in 512 byte chunks[1]. [1] https://bugs.webkit.org/show_bug.cgi?id=265386 https://bugs.webkit.org/show_bug.cgi?id=265386
- lxgr 1y ago
  GP is talking about intermediary proxies, CDNs etc. that might be unhappy about long-running connections with responses trickling in bit by bit, not doubting that it works on the client side. That said, I'd be surprised if proxy software or services like Cloudflare didn't have logic to automatically opt out of "CDN mode" and switch to something more transparent when they see "text/event-stream". It's not that uncommon, all things considered.

hobofan 1y ago

With the current AI/LLM wave SSE have received a lot of attention again, and most LLM chat frontends use them. At least from my perception as a result of this, support for SSEs in major HTTP server frameworks has improved a lot in the last few years. It is a bit of a shame though, that in order to do most useful things with SSEs you have to resort to doing non-spec-compliant things (e.g. send initial payload with POST).

blensor 1y ago
Also MCP uses it
ljm 1y ago
Same with graphql subscriptions. Arguably it’s also because of serverless architecture where SSE can be used more easily than WS or streaming. If you want any of that on Lambda and API Gateway, for example, and didn’t anticipate it right off the bat, you’re in for quite a bit of pain.
nkozyra 1y ago
SSE limitations in the browser are still a drag for this, too.

skrebbel 1y ago

> I don't believe that HTTP streaming was designed with this pattern in mind > server-sent events are a simpler solution Fwiw Server-Sent Events are a protocol on top of HTTP Streaming. In fact I'm somewhat surprised that the article doesn't mention it, instead rolling their own SSE alternative that looks (to my non-expert eyes) like a lower level version of the same thing. It seems a bit weird to me to use chunks as a package boundary, I'd worry that that has weird edge cases (eg won't large responses be split into multiple chunks?)

notjoemama 1y ago
Because of TCP, large chunks are always split into smaller chunks. It’s just that at the HTTP level we don’t know and don’t see it. UDP forces people into designing their own protocols if the data is a defined package of bytes. Having done some socket coding my impression is web sockets would be good for a high end browser based game, browser based simulations, or maybe a high end trading system. At that point the browser is just a shell/window. As others have pointed out, there are already plenty of alternatives for web applications.
- jeroenhd 1y ago
  The problem for things like video games and trading is that websockets only support TCP by default. Technologies like WebRTC allow for much faster updates. I think websockets certainly have their uses. Mostly in systems where SSE isn't available quickly and easily, or when sending a bunch of quick communications one after another as there's no way to know if the browser will pipeline the requests automatically or if it'll set up a whole bunch of requests.
benwilber0 1y ago
I pretty much always prefer SSE over websockets just because of the simplicity end-to-end. It's "just HTTP", so all the HTTP-based tech and tools apply out-of-the-box without any really special configuration that is required for WS. Curl (or even netcat) "just works", no special client. I don't have to do any special CDN configuration to proxy connections or terminate SSL aside from just turning off buffering. Websockets requires almost a completely new L7 stack and tons of special configuration to handle Upgrade, text or data frames, etc. And once you're out of "HTTP mode" you now have to implement the primitive mechanics of basically everything yourself, like auth, redirects, sessions, etc. It's why I originally made Tiny SSE which is a purpose-built SSE server written in Rust and programmable with Lua. https://tinysse.com https://tinysse.com https://github.com/benwilber/tinysse https://github.com/benwilber/tinysse

osigurdson 1y ago

The issue I have with SSE and what is being proposed in this article (which is very similar), is the very long lived connection. OpenAI uses SSE for callbacks. That works fine for chat and other "medium" duration interactions but when it comes to fine tuning (which can take a very long time), SSE always breaks and requires client side retries to get it to work. So, why not instead use something like long polling + http streaming (a slight tweak on SSE). Here is the idea: 1) Make a standard GET call /api/v1/events (using standard auth, etc) 2) If anything is in the buffer / queue return it immediately 3) Stream any new events for up to 60s. Each event has a sequence id (similar to the article). Include keep alive messages at 10s intervals if there are no messages. 4) After 60s close the connection - gracefully ending the interaction on the client 5) Client makes another GET request using the last received sequence What I like about this is it is very simple to understand (like SSE - it basically is SSE), has low latency, is just a standard GET with standard auth and works regardless of how load balancers, etc., are configured. Of course, there will be errors from time to time, but dealing with timeouts / errors will not be the norm.

thedufer 1y ago
I don't understand the advantages of recreating SSE yourself like this vs just using SSE. > SSE always breaks and requires client side retries to get it to work Yeah, but these are automatic (the browser handles it). SSE is really easy to get started with.
- osigurdson 1y ago
  My issue with eventsource is it doesn't use standard auth. Including the jwt in a query string is an odd step out requiring alternate middleware and feels like there is a high chance of leaking the token in logs, etc. I'm curious though, what is your solution to this? Secondly, not every client is a browser (my OpenAI / fine tune example is non-browser based). Finally, I just don't like the idea of things failing all time with something working behind the scenes to resolve issues. I'd like errors / warnings in logs to mean something, personally. >> I don't understand the advantages of recreating SSE yourself like this vs just using SSE This is more of a strawman and don't plan to implement it. It is based on experiences consuming SSE endpoints as well as creating them.

runeks 1y ago

> Perhaps I'm wrong, but I believe HTTP streaming is for chunking large blobs. You are wrong in the case of Chrome and Firefox. I have tried it and streaming e.g. unordered list elements are displayed instantly. But for Safari, "text/html" streaming happens in 512 byte chunks[1]. [1] https://bugs.webkit.org/show_bug.cgi?id=265386 https://bugs.webkit.org/show_bug.cgi?id=265386

lxgr 1y ago
GP is talking about intermediary proxies, CDNs etc. that might be unhappy about long-running connections with responses trickling in bit by bit, not doubting that it works on the client side. That said, I'd be surprised if proxy software or services like Cloudflare didn't have logic to automatically opt out of "CDN mode" and switch to something more transparent when they see "text/event-stream". It's not that uncommon, all things considered.