4 ms·

The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t sup

by paulgb 4y ago

The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) The other operation that I haven’t seen elsewhere, but is vital to enabling stream compaction without a leader, is the idea of a stream rollup up to a specific stream number. NATS Jetstream, for example, has the ability to roll up an entire stream, but if another message hits the stream between when the rollup is computed and when it arrives at the server, that message too will be replaced (IIRC). So I thought about using NATS (which already has a WebSocket protocol), but ruled it out.

alexisread 4y ago
MQTT-over-websocket does exist (https://github.com/mqttjs/MQTT.js https://github.com/mqttjs/MQTT.js), and most MQTT brokers support it (Mosquito, AmazonMQ etc.). You're right about the compaction - MQTT doesn't have anything in it's protocol about compaction, and I don't know of any brokers that implement it. Having said that, you could use an MQTT-kafka bridge. Something like Mosquito + https://github.com/nodefluent/mqtt-to-kafka-bridge https://github.com/nodefluent/mqtt-to-kafka-bridge + Redpanda in a docker image would work, though obv. this might be a bit overkill for most. Having said that, it does open many new avenues for interaction at scale. You pays your money...
- fud101 4y ago
  What is compaction?
  - paulgb 4y ago
    Compaction is where you take a chunk of messages and replace them with a single message. For example, one of the DriftDB demos is a counter (https://demos.driftdb.com/counter https://demos.driftdb.com/counter). State is synchronized by putting increment/decrement events into a stream. When a new user connects, their client get all the messages in the stream, plays them back, and arrives at the same state as everyone else. If that’s all we did, over time, the stream would grow unruly. It would take ages to load the page because we’d have to load every state change. But we only really care about a single numeric value. Compaction takes a chunk of messages that look like this: {"apply":"increment"} {"apply":"increment"} {"apply":"decrement"} {"apply":"increment"} And replaces them with a message that looks like this: {"reset":2} DriftDB doesn’t know how to compute the compaction, it relies on clients to do that. When a client does something that increases the length of the stream, the server sends back the new length of the stream, so that the client can decide whether to compact it (i.e. if it passes some threshold). The important part that I haven’t seen elsewhere is that when a client compacts the stream, it includes a sequence number of the last message that’s part of the compaction. The server will preserve messages greater than that sequence number, since they are not part of the compaction.
chrisdalke 4y ago
Most MQTT implementations do support MQTT-over-Websocket. I use it extensively at work and it's been fairly reliable!
elithrar 4y ago
> The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) We support MQTT over WS (or JSON over WS, or just HTTP) in Cloudflare Pub/Sub, FWIW - https://developers.cloudflare.com/pub-sub/learning/websockets-browsers/ https://developers.cloudflare.com/pub-sub/learning/websocket... I also agree with the comments re: MQTT being well suited to a lot of these "broadcast" use-case, but that the IoT roots seem to hold it back. MQTT 5.0 is just a great protocol — clear spec, explicit about errors, flexible payloads — that make it well suited to these broadcast/fan-in/real-time workloads. The traditional cloud providers do MQTT (3.1.1) in their respective IoT platforms but never grew it beyond that.
- jconley 4y ago
  Can I get in on that beta? Submitted the form yesterday. Currently building something that could use it. ;)
Kinrany 4y ago
Isn't it fairly easy to implement rollups as a stream of rollups with matching stream numbers?

by paulgb 4y ago

alexisread 4y ago

MQTT-over-websocket does exist (https://github.com/mqttjs/MQTT.js https://github.com/mqttjs/MQTT.js), and most MQTT brokers support it (Mosquito, AmazonMQ etc.). You're right about the compaction - MQTT doesn't have anything in it's protocol about compaction, and I don't know of any brokers that implement it. Having said that, you could use an MQTT-kafka bridge. Something like Mosquito + https://github.com/nodefluent/mqtt-to-kafka-bridge https://github.com/nodefluent/mqtt-to-kafka-bridge + Redpanda in a docker image would work, though obv. this might be a bit overkill for most. Having said that, it does open many new avenues for interaction at scale. You pays your money...

fud101 4y ago
What is compaction?
- paulgb 4y ago
  Compaction is where you take a chunk of messages and replace them with a single message. For example, one of the DriftDB demos is a counter (https://demos.driftdb.com/counter https://demos.driftdb.com/counter). State is synchronized by putting increment/decrement events into a stream. When a new user connects, their client get all the messages in the stream, plays them back, and arrives at the same state as everyone else. If that’s all we did, over time, the stream would grow unruly. It would take ages to load the page because we’d have to load every state change. But we only really care about a single numeric value. Compaction takes a chunk of messages that look like this: {"apply":"increment"} {"apply":"increment"} {"apply":"decrement"} {"apply":"increment"} And replaces them with a message that looks like this: {"reset":2} DriftDB doesn’t know how to compute the compaction, it relies on clients to do that. When a client does something that increases the length of the stream, the server sends back the new length of the stream, so that the client can decide whether to compact it (i.e. if it passes some threshold). The important part that I haven’t seen elsewhere is that when a client compacts the stream, it includes a sequence number of the last message that’s part of the compaction. The server will preserve messages greater than that sequence number, since they are not part of the compaction.

chrisdalke 4y ago

Most MQTT implementations do support MQTT-over-Websocket. I use it extensively at work and it's been fairly reliable!

elithrar 4y ago

> The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) We support MQTT over WS (or JSON over WS, or just HTTP) in Cloudflare Pub/Sub, FWIW - https://developers.cloudflare.com/pub-sub/learning/websockets-browsers/ https://developers.cloudflare.com/pub-sub/learning/websocket... I also agree with the comments re: MQTT being well suited to a lot of these "broadcast" use-case, but that the IoT roots seem to hold it back. MQTT 5.0 is just a great protocol — clear spec, explicit about errors, flexible payloads — that make it well suited to these broadcast/fan-in/real-time workloads. The traditional cloud providers do MQTT (3.1.1) in their respective IoT platforms but never grew it beyond that.

jconley 4y ago
Can I get in on that beta? Submitted the form yesterday. Currently building something that could use it. ;)

Kinrany 4y ago

Isn't it fairly easy to implement rollups as a stream of rollups with matching stream numbers?