4 ms·
There's absolutely nothing wrong with SSE. The problem is that the author is, apparently, stapling SSE directly onto streaming LLM responses, using the LLM infe
by cheald 10mo ago
There's absolutely nothing wrong with SSE. The problem is that the author is, apparently, stapling SSE directly onto streaming LLM responses, using the LLM inference provider as the "message queue", and then complaining that it doesn't provide seekability and resumption. Things that aren't MQs make pretty bad MQs. The problem isn't the message bus, it's trying to treat an LLM as a message queue.
Use an MQ if you want MQ things. Worse, the conclusion is wrong - pubsub isn't an MQ, though it often uses them. Pubsub is a pattern for message delivery, which by itself, makes no inherent guarantees about message durability or ordering; Rails' ActionCable, for example, utilizes the PubSub pattern (and in fact is built on Redis' pubsub feature!), but due to its implementation, it makes no delivery order guarantees, which is completely inappropriate for LLM inference.