7 ms·

I've not looked at DriftDB in depth (cloudflare worker running this is neat!), but can't MQTT handle this sort of workload? Obv. there's not a cloudflare worke

I've not looked at DriftDB in depth (cloudflare worker running this is neat!), but can't MQTT handle this sort of workload? Obv. there's not a cloudflare worker running say an MQTT server over websockets, but you can scope topics with wildcards (https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices/ https://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topi...), replay missed messages on reconnection, last-will-and-testament, ACLs, dynamic topic creation, binary messages etc. I'm asking as many of these websocket projects seem to use custom protocols rather than anything standard aka interoperable.

paulgb 4y ago
The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) The other operation that I haven’t seen elsewhere, but is vital to enabling stream compaction without a leader, is the idea of a stream rollup up to a specific stream number. NATS Jetstream, for example, has the ability to roll up an entire stream, but if another message hits the stream between when the rollup is computed and when it arrives at the server, that message too will be replaced (IIRC). So I thought about using NATS (which already has a WebSocket protocol), but ruled it out.
- alexisread 4y ago
  MQTT-over-websocket does exist (https://github.com/mqttjs/MQTT.js https://github.com/mqttjs/MQTT.js), and most MQTT brokers support it (Mosquito, AmazonMQ etc.). You're right about the compaction - MQTT doesn't have anything in it's protocol about compaction, and I don't know of any brokers that implement it. Having said that, you could use an MQTT-kafka bridge. Something like Mosquito + https://github.com/nodefluent/mqtt-to-kafka-bridge https://github.com/nodefluent/mqtt-to-kafka-bridge + Redpanda in a docker image would work, though obv. this might be a bit overkill for most. Having said that, it does open many new avenues for interaction at scale. You pays your money...
  - fud101 4y ago
    What is compaction?
    - paulgb 4y ago
      Compaction is where you take a chunk of messages and replace them with a single message. For example, one of the DriftDB demos is a counter (https://demos.driftdb.com/counter https://demos.driftdb.com/counter). State is synchronized by putting increment/decrement events into a stream. When a new user connects, their client get all the messages in the stream, plays them back, and arrives at the same state as everyone else. If that’s all we did, over time, the stream would grow unruly. It would take ages to load the page because we’d have to load every state change. But we only really care about a single numeric value. Compaction takes a chunk of messages that look like this: {"apply":"increment"} {"apply":"increment"} {"apply":"decrement"} {"apply":"increment"} And replaces them with a message that looks like this: {"reset":2} DriftDB doesn’t know how to compute the compaction, it relies on clients to do that. When a client does something that increases the length of the stream, the server sends back the new length of the stream, so that the client can decide whether to compact it (i.e. if it passes some threshold). The important part that I haven’t seen elsewhere is that when a client compacts the stream, it includes a sequence number of the last message that’s part of the compaction. The server will preserve messages greater than that sequence number, since they are not part of the compaction.
- chrisdalke 4y ago
  Most MQTT implementations do support MQTT-over-Websocket. I use it extensively at work and it's been fairly reliable!
- elithrar 4y ago
  > The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) We support MQTT over WS (or JSON over WS, or just HTTP) in Cloudflare Pub/Sub, FWIW - https://developers.cloudflare.com/pub-sub/learning/websockets-browsers/ https://developers.cloudflare.com/pub-sub/learning/websocket... I also agree with the comments re: MQTT being well suited to a lot of these "broadcast" use-case, but that the IoT roots seem to hold it back. MQTT 5.0 is just a great protocol — clear spec, explicit about errors, flexible payloads — that make it well suited to these broadcast/fan-in/real-time workloads. The traditional cloud providers do MQTT (3.1.1) in their respective IoT platforms but never grew it beyond that.
  - jconley 4y ago
    Can I get in on that beta? Submitted the form yesterday. Currently building something that could use it. ;)
- Kinrany 4y ago
  Isn't it fairly easy to implement rollups as a stream of rollups with matching stream numbers?
nine_k 4y ago
Maybe it's the richness of MQTT that makes it a worse choice for a startup. Offering a conformant MQTT broker is a lot of work, and the semantics come from elsewhere, not geared towards emphasizing your unique advantages. Building a much simpler, custom-tailored protocol allows to ship faster, and improve gradually. If the point is to deploy on Cloudflare in a massively-parallel fashion (which is likely harder for a regular MQTT broker), the custom protocol allows to concentrate on that special advantage, and not on standards conformance or interoperability with a bevy of existing libraries.
manv1 4y ago
Funny, the IoT space has bought into MQTT but the general internet space has not. MQTT scales and works. And it's easy, fast, and small. I've been trying to get our guys to do MQTT-based pub/sub, and they're rather do their own thing with web sockets because MQTT is scary. <shrug>. That's the problem when front-end guys make decisions about tech sometimes, they choose stuff that seems easy to integrate without caring about things like deployment, scalability, capabilities, etc.
- manv1 4y ago
  I mean, it'd be trivial to write stream replay for MQTT. It's literally just stashing messages and sending them back on connect. Not sure what the issue is there.
- Scottopherson 4y ago
  Jeez that's a big paint brush you're slinging around. That's the problem when non-front-end guys make decisions about tech sometimes, they choose stuff that seems easy to integrate without caring about things like accessibility, design scalability, client device capabilities, etc.
  - nine_k 4y ago
    How does a wire protocol relate to UX concerns like accessibility or design scalability? Client device capabilities are there, MQTT is neither rocket science nor a resource hog, since it was designed for underpowered IoT devices.

paulgb 4y ago

The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) The other operation that I haven’t seen elsewhere, but is vital to enabling stream compaction without a leader, is the idea of a stream rollup up to a specific stream number. NATS Jetstream, for example, has the ability to roll up an entire stream, but if another message hits the stream between when the rollup is computed and when it arrives at the server, that message too will be replaced (IIRC). So I thought about using NATS (which already has a WebSocket protocol), but ruled it out.

alexisread 4y ago
MQTT-over-websocket does exist (https://github.com/mqttjs/MQTT.js https://github.com/mqttjs/MQTT.js), and most MQTT brokers support it (Mosquito, AmazonMQ etc.). You're right about the compaction - MQTT doesn't have anything in it's protocol about compaction, and I don't know of any brokers that implement it. Having said that, you could use an MQTT-kafka bridge. Something like Mosquito + https://github.com/nodefluent/mqtt-to-kafka-bridge https://github.com/nodefluent/mqtt-to-kafka-bridge + Redpanda in a docker image would work, though obv. this might be a bit overkill for most. Having said that, it does open many new avenues for interaction at scale. You pays your money...
- fud101 4y ago
  What is compaction?
  - paulgb 4y ago
    Compaction is where you take a chunk of messages and replace them with a single message. For example, one of the DriftDB demos is a counter (https://demos.driftdb.com/counter https://demos.driftdb.com/counter). State is synchronized by putting increment/decrement events into a stream. When a new user connects, their client get all the messages in the stream, plays them back, and arrives at the same state as everyone else. If that’s all we did, over time, the stream would grow unruly. It would take ages to load the page because we’d have to load every state change. But we only really care about a single numeric value. Compaction takes a chunk of messages that look like this: {"apply":"increment"} {"apply":"increment"} {"apply":"decrement"} {"apply":"increment"} And replaces them with a message that looks like this: {"reset":2} DriftDB doesn’t know how to compute the compaction, it relies on clients to do that. When a client does something that increases the length of the stream, the server sends back the new length of the stream, so that the client can decide whether to compact it (i.e. if it passes some threshold). The important part that I haven’t seen elsewhere is that when a client compacts the stream, it includes a sequence number of the last message that’s part of the compaction. The server will preserve messages greater than that sequence number, since they are not part of the compaction.
chrisdalke 4y ago
Most MQTT implementations do support MQTT-over-Websocket. I use it extensively at work and it's been fairly reliable!
elithrar 4y ago
> The problem with MQTT is that most of the use cases I’m interested in involve a web browser as at least one party of the connection, and the browser doesn’t support MQTT. I could wrap MQTT in a WebSocket, but then I’d lose the advantages of MQTT’s compactness and interoperability (unless MQTT-over-WebSocket is a thing?) We support MQTT over WS (or JSON over WS, or just HTTP) in Cloudflare Pub/Sub, FWIW - https://developers.cloudflare.com/pub-sub/learning/websockets-browsers/ https://developers.cloudflare.com/pub-sub/learning/websocket... I also agree with the comments re: MQTT being well suited to a lot of these "broadcast" use-case, but that the IoT roots seem to hold it back. MQTT 5.0 is just a great protocol — clear spec, explicit about errors, flexible payloads — that make it well suited to these broadcast/fan-in/real-time workloads. The traditional cloud providers do MQTT (3.1.1) in their respective IoT platforms but never grew it beyond that.
- jconley 4y ago
  Can I get in on that beta? Submitted the form yesterday. Currently building something that could use it. ;)
Kinrany 4y ago
Isn't it fairly easy to implement rollups as a stream of rollups with matching stream numbers?

nine_k 4y ago

Maybe it's the richness of MQTT that makes it a worse choice for a startup. Offering a conformant MQTT broker is a lot of work, and the semantics come from elsewhere, not geared towards emphasizing your unique advantages. Building a much simpler, custom-tailored protocol allows to ship faster, and improve gradually. If the point is to deploy on Cloudflare in a massively-parallel fashion (which is likely harder for a regular MQTT broker), the custom protocol allows to concentrate on that special advantage, and not on standards conformance or interoperability with a bevy of existing libraries.

manv1 4y ago

Funny, the IoT space has bought into MQTT but the general internet space has not. MQTT scales and works. And it's easy, fast, and small. I've been trying to get our guys to do MQTT-based pub/sub, and they're rather do their own thing with web sockets because MQTT is scary. <shrug>. That's the problem when front-end guys make decisions about tech sometimes, they choose stuff that seems easy to integrate without caring about things like deployment, scalability, capabilities, etc.

manv1 4y ago
I mean, it'd be trivial to write stream replay for MQTT. It's literally just stashing messages and sending them back on connect. Not sure what the issue is there.
Scottopherson 4y ago
Jeez that's a big paint brush you're slinging around. That's the problem when non-front-end guys make decisions about tech sometimes, they choose stuff that seems easy to integrate without caring about things like accessibility, design scalability, client device capabilities, etc.
- nine_k 4y ago
  How does a wire protocol relate to UX concerns like accessibility or design scalability? Client device capabilities are there, MQTT is neither rocket science nor a resource hog, since it was designed for underpowered IoT devices.