6 ms·

Cloudflare Pub/Sub: Programmable MQTT Message Brokers

128 points by elithrar 4y ago

0xy 4y ago
Cloudflare is really fast putting the pieces together to become a modern simpler AWS. By focusing on edge networks and programmability from the start they've been able to outmaneuver AWS in product development and pricing. AWS today is stable, reliable but expensive and rigid. A lot of their products feel like they've been taken out of the oven 15 minutes too early. AppSync, API Gateway, CloudFormation and others all feel lethargic, stagnant or not fit for purpose in the current world. I don't think AWS is in a position to compete with Cloudflare's strategy without massively slashing bandwidth prices and improving their product development. They also aren't really a credible edge platform. Lambda@Edge is really bad, CloudFront is slow and terrible compared to Akamai, Fastly and Cloudflare (and Amazon's retail business doesn't even use it -- because it sucks). Super interesting to watch. ($NET shareholder)
- astlouis44 4y ago
  Great take, I agree. Thoughts on a Fastly or Cloudflare acquisition by one of the tech giants?
  - 0xy 4y ago
    If Akamai were smart they'd buy Fastly yesterday, because they're rapidly falling behind. Cloudflare is harder to see because of antitrust concerns. With Fastly's current position, they should sell, and someone should buy them for the network and progress on edge compute alone.
  - deleted 4y ago
    [deleted]
- nicoburns 4y ago
  They have no non-function compute though, which is arguably AWS’s core feature.
  - jbverschoor 4y ago
    It's also the things that's easiest to migrate away from to a cheap-and-performant VPS or dedicated server
    - nicoburns 4y ago
      I guess from my perspective that's also why I want it in my cloud. I'm willing to pay for managed services, but don't want to be using services I can't easily migrate away from.
fasteo 4y ago
cloudflarer around here ? Did you wrote the MQTT broker from scratch or are you using some existing broker (Mosquitto , Nanomq, etc). Just curious.
- elithrar 4y ago
  (I'm the PM for this at Cloudflare & the blog author) We looked at a number of existing options — trying to strike a balance between languages we run at scale (tooling, ops, security), memory safety, and long-term maintainability as a growing company. We ended up writing an MQTT broker from scratch in Rust, with our own Durable Objects[1] for consistency and co-ordination across our hundreds of edge locations (and tens of thousands of machines). The MQTT spec itself[2] is very approachable, which made this undertaking easier still. One of the increasingly cool parts of Cloudflare is that we use more and more of what our customers use: our Broker service sits behind Spectrum[3], and we use Workers + DO for all of the metadata/session handling. The next set of new products will invariably start to build on Pub/Sub itself, too :-) [1]: https://developers.cloudflare.com/workers/learning/using-durable-objects/ https://developers.cloudflare.com/workers/learning/using-dur... [2]: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html [3]: https://developers.cloudflare.com/spectrum/ https://developers.cloudflare.com/spectrum/
  - fasteo 4y ago
    Thanks, make sense. Actually, I asked because it surprised me that D1 - sqlite - was not written the same way. [1] https://blog.cloudflare.com/introducing-d1/ https://blog.cloudflare.com/introducing-d1/
    - elithrar 4y ago
      D1 _does_ use Durable Objects under the hood! Like Pub/Sub, it's ultimately an implementation detail, albeit probably interesting to this audience.
      - fasteo 4y ago
rafaelturk 4y ago
Not clear for me if I can `push` messages from my backend to client? At lest for me this is a killer feature.
- elithrar 4y ago
  In the MQTT world, in which Pub/Sub is based, every publisher is a client, and every subscriber is a client. You can be both, too! Assuming your existing backend is in $SOME_CLOUD_PROVIDER, publishing messages to Pub/Sub is just a few lines of code in your favorite language - we have a couple of examples up in JS + Rust - e.g. https://developers.cloudflare.com/pub-sub/examples/connect-javascript/ https://developers.cloudflare.com/pub-sub/examples/connect-j... Subscribers receiving those messages can be anywhere you can run an MQTT client, which turns out to be... a lot of places :-) (I'm the PM for this, and we'll be adding more examples to our docs in the coming weeks)
deleted 4y ago
[deleted]
hack_tmmm 4y ago
This is such a wonderful news
VectorLock 4y ago
I like how Cloudflare is taking a serious run at AWS. And you can't even subscribe a program to a SNS topic and Cloudflare is giving us pub/sub.
- nullwarp 4y ago
  FWIW with AWS, despite the naming, the IoT Core MQTT broker is an excellent product for exactly this - it's just sold as IoT specific. It's really just an MQTT broker at it's core and could probably be split off into it's own product without the IoT branding.
- manigandham 4y ago
  AWS has several "pub/sub" like services: SNS, IoT MQTT, Managed Kafka, Kinesis, EventBridge, etc.
  - VectorLock 4y ago
    And they're all kinda fucky, like SNS handles a lot but you can't write a program to receive events.
    - manigandham 4y ago
      What do you mean by "receive events"? SNS can deliver to any HTTP endpoint.
      - sieabahlpark 4y ago
        People are butt hurt that SNS doesn't allow the "subscribe" portion. A webhook is a push action. They forget SQS exists...
      - VectorLock 4y ago
        Then you have to go through the whole process of setting up, routing, and registering an HTTP endpoint, rather than you would as a normal pub/sub simply have the program connect to the publisher and start receiving messages/events.
losvedir 4y ago
Wow, another neat Cloudflare release. Is there some sort of "innovation week" type thing going on right now? One question that the example given raises for me, is how to manage that function I see there in a robust, "proper software engineering" sort of way. This has been a challenge for me on AWS when trying out Lambdas. The individual pieces are cool, but the system that connects them all together is a little opaque. How do you version control the code? How do you keep track of which pieces work with which other pieces? We use terraform to manage the AWS resources, but it's kind of clunky, and not official. Is there some Cloudflare-supported way to maybe declaratively define your different resources? Someone who has built a somewhat complex app with a lot of workers, and durable objects, and different pieces of the ecosystem: how do you manage that? How do people review changes? Since it sounds like from another comment that Cloudflare dogfoods its own services internally, I'm curious if there's a write-up about how they actually do so? The little function code snippets give a nice taste of the code a worker might run, but that's still kind of a long way clearing up for me how you would actually use this in practice in a standard production environment.
- brightball 4y ago
  Platform Week
- deleted 4y ago
  [deleted]
- jffry 4y ago
  > Is there some sort of "innovation week" type thing going on right now? Yes. "Platform Week 2022": https://blog.cloudflare.com/platform-week-2022/ https://blog.cloudflare.com/platform-week-2022/
- piperswe 4y ago
  At least on my team at Cloudflare, great Terraform support is a basically required part of any API change. We have an official Terraform provider (https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs https://registry.terraform.io/providers/cloudflare/cloudflar...), and nearly every API feature is provisionable using that provider.
anonymousDan 4y ago
How does this compare to something like Ably?
knodi 4y ago
I love it!! There needs to be more message brokers as a service providers.
deleted 4y ago
[deleted]
shikivi 4y ago
Pub/Sub, Workers, KV, Durable Objects, R2... yes, these all sounds promising. Awesome tool for developers. But there is one huge obstacle in the middle that prevents most to actually try this stack. Logs and Account Access control. Cloudflare is a very powerful platform, you can't let everyone be Admin. If you run anything, you need to be able to see logs it to ship it somewhere. For some reason, Cloudflare thinks that these are enterprise feature. I got news for you, these are basic features for anyone who wants to build more than just a simple script for individual use. You work so hard to build these features but let only a small subset of users really do anything meaningful with it
- sitkack 4y ago
  Honest question, not a snark attack. But would it be possible to build those things on top of Cloudflare itself, esp the logging? For access control, you would have to have a separate process controlling the CF api for configuration.
  - okaybuy 4y ago
    Not so simple when a lot of the tooling that facilitates easy development is tied to CF API, are you meant to build your own cloudflared and wrangler just so you dont have to grant everyone in the org super admin? This is where AWS wins, AWS gives you everything from $0/month, IAM in the above case, but you don't stay at $0 for long, where as CF gives you partial bits but you remain at $0-low/pm for longer Its actually really ironic the same company that says SAML for all for free, can't even get basic RAC on their own admin controls
- jplevine 4y ago
  We just announced something called the Workers Analytics Engine today: https://blog.cloudflare.com/workers-analytics-engine/ https://blog.cloudflare.com/workers-analytics-engine/ More coming soon! Happy to chat more - jpl at cloudflare dot com
- imobbscf 4y ago
  What would you like to see? Disclaimer: Engineer on the IAM team at Cloudflare
michaelrhansen 4y ago
How many of these commenters work at Cloudflare? :)
haroldadmin 4y ago
I like that Cloudflare is steadily becoming a serious alternative to the existing cloud providers. If you folks can build a decent IAM tool for access control to Cloudflare resources, much more people will start using your services.
VirusNewbie 4y ago
huh, Fastly is doing pub/sub lately too with fanout now: https://www.fastly.com/blog/fastly-fanout-why-real-time-messaging-and-edge-computing-are-an-amazing-combination https://www.fastly.com/blog/fastly-fanout-why-real-time-mess...
eska 4y ago
Since you guys seem to be using Rust, what client library do you recommend for MQTTv5 support?
- cehrig 4y ago
  (Pub/Sub Eng at Cloudflare) We are testing the broker using the Paho C wrapper https://github.com/eclipse/paho.mqtt.rust https://github.com/eclipse/paho.mqtt.rust .. and have added a client example to our dev-docs https://developers.cloudflare.com/pub-sub/examples/connect-rust/ https://developers.cloudflare.com/pub-sub/examples/connect-r...

128 points by elithrar 4y ago

0xy 4y ago

Cloudflare is really fast putting the pieces together to become a modern simpler AWS. By focusing on edge networks and programmability from the start they've been able to outmaneuver AWS in product development and pricing. AWS today is stable, reliable but expensive and rigid. A lot of their products feel like they've been taken out of the oven 15 minutes too early. AppSync, API Gateway, CloudFormation and others all feel lethargic, stagnant or not fit for purpose in the current world. I don't think AWS is in a position to compete with Cloudflare's strategy without massively slashing bandwidth prices and improving their product development. They also aren't really a credible edge platform. Lambda@Edge is really bad, CloudFront is slow and terrible compared to Akamai, Fastly and Cloudflare (and Amazon's retail business doesn't even use it -- because it sucks). Super interesting to watch. ($NET shareholder)

astlouis44 4y ago
Great take, I agree. Thoughts on a Fastly or Cloudflare acquisition by one of the tech giants?
- 0xy 4y ago
  If Akamai were smart they'd buy Fastly yesterday, because they're rapidly falling behind. Cloudflare is harder to see because of antitrust concerns. With Fastly's current position, they should sell, and someone should buy them for the network and progress on edge compute alone.
- deleted 4y ago
  [deleted]
nicoburns 4y ago
They have no non-function compute though, which is arguably AWS’s core feature.
- jbverschoor 4y ago
  It's also the things that's easiest to migrate away from to a cheap-and-performant VPS or dedicated server
  - nicoburns 4y ago
    I guess from my perspective that's also why I want it in my cloud. I'm willing to pay for managed services, but don't want to be using services I can't easily migrate away from.

fasteo 4y ago

cloudflarer around here ? Did you wrote the MQTT broker from scratch or are you using some existing broker (Mosquitto , Nanomq, etc). Just curious.

elithrar 4y ago
(I'm the PM for this at Cloudflare & the blog author) We looked at a number of existing options — trying to strike a balance between languages we run at scale (tooling, ops, security), memory safety, and long-term maintainability as a growing company. We ended up writing an MQTT broker from scratch in Rust, with our own Durable Objects[1] for consistency and co-ordination across our hundreds of edge locations (and tens of thousands of machines). The MQTT spec itself[2] is very approachable, which made this undertaking easier still. One of the increasingly cool parts of Cloudflare is that we use more and more of what our customers use: our Broker service sits behind Spectrum[3], and we use Workers + DO for all of the metadata/session handling. The next set of new products will invariably start to build on Pub/Sub itself, too :-) [1]: https://developers.cloudflare.com/workers/learning/using-durable-objects/ https://developers.cloudflare.com/workers/learning/using-dur... [2]: https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html [3]: https://developers.cloudflare.com/spectrum/ https://developers.cloudflare.com/spectrum/
- fasteo 4y ago
  Thanks, make sense. Actually, I asked because it surprised me that D1 - sqlite - was not written the same way. [1] https://blog.cloudflare.com/introducing-d1/ https://blog.cloudflare.com/introducing-d1/
  - elithrar 4y ago
    D1 _does_ use Durable Objects under the hood! Like Pub/Sub, it's ultimately an implementation detail, albeit probably interesting to this audience.
    - fasteo 4y ago

rafaelturk 4y ago

Not clear for me if I can `push` messages from my backend to client? At lest for me this is a killer feature.

elithrar 4y ago
In the MQTT world, in which Pub/Sub is based, every publisher is a client, and every subscriber is a client. You can be both, too! Assuming your existing backend is in $SOME_CLOUD_PROVIDER, publishing messages to Pub/Sub is just a few lines of code in your favorite language - we have a couple of examples up in JS + Rust - e.g. https://developers.cloudflare.com/pub-sub/examples/connect-javascript/ https://developers.cloudflare.com/pub-sub/examples/connect-j... Subscribers receiving those messages can be anywhere you can run an MQTT client, which turns out to be... a lot of places :-) (I'm the PM for this, and we'll be adding more examples to our docs in the coming weeks)

deleted 4y ago

[deleted]

hack_tmmm 4y ago

This is such a wonderful news

VectorLock 4y ago

I like how Cloudflare is taking a serious run at AWS. And you can't even subscribe a program to a SNS topic and Cloudflare is giving us pub/sub.

nullwarp 4y ago
FWIW with AWS, despite the naming, the IoT Core MQTT broker is an excellent product for exactly this - it's just sold as IoT specific. It's really just an MQTT broker at it's core and could probably be split off into it's own product without the IoT branding.
manigandham 4y ago
AWS has several "pub/sub" like services: SNS, IoT MQTT, Managed Kafka, Kinesis, EventBridge, etc.
- VectorLock 4y ago
  And they're all kinda fucky, like SNS handles a lot but you can't write a program to receive events.
  - manigandham 4y ago
    What do you mean by "receive events"? SNS can deliver to any HTTP endpoint.
    - sieabahlpark 4y ago
      People are butt hurt that SNS doesn't allow the "subscribe" portion. A webhook is a push action. They forget SQS exists...
    - VectorLock 4y ago
      Then you have to go through the whole process of setting up, routing, and registering an HTTP endpoint, rather than you would as a normal pub/sub simply have the program connect to the publisher and start receiving messages/events.

losvedir 4y ago

Wow, another neat Cloudflare release. Is there some sort of "innovation week" type thing going on right now? One question that the example given raises for me, is how to manage that function I see there in a robust, "proper software engineering" sort of way. This has been a challenge for me on AWS when trying out Lambdas. The individual pieces are cool, but the system that connects them all together is a little opaque. How do you version control the code? How do you keep track of which pieces work with which other pieces? We use terraform to manage the AWS resources, but it's kind of clunky, and not official. Is there some Cloudflare-supported way to maybe declaratively define your different resources? Someone who has built a somewhat complex app with a lot of workers, and durable objects, and different pieces of the ecosystem: how do you manage that? How do people review changes? Since it sounds like from another comment that Cloudflare dogfoods its own services internally, I'm curious if there's a write-up about how they actually do so? The little function code snippets give a nice taste of the code a worker might run, but that's still kind of a long way clearing up for me how you would actually use this in practice in a standard production environment.

brightball 4y ago
Platform Week
deleted 4y ago
[deleted]
jffry 4y ago
> Is there some sort of "innovation week" type thing going on right now? Yes. "Platform Week 2022": https://blog.cloudflare.com/platform-week-2022/ https://blog.cloudflare.com/platform-week-2022/
piperswe 4y ago
At least on my team at Cloudflare, great Terraform support is a basically required part of any API change. We have an official Terraform provider (https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs https://registry.terraform.io/providers/cloudflare/cloudflar...), and nearly every API feature is provisionable using that provider.

anonymousDan 4y ago

How does this compare to something like Ably?

knodi 4y ago

I love it!! There needs to be more message brokers as a service providers.

deleted 4y ago

[deleted]

shikivi 4y ago

Pub/Sub, Workers, KV, Durable Objects, R2... yes, these all sounds promising. Awesome tool for developers. But there is one huge obstacle in the middle that prevents most to actually try this stack. Logs and Account Access control. Cloudflare is a very powerful platform, you can't let everyone be Admin. If you run anything, you need to be able to see logs it to ship it somewhere. For some reason, Cloudflare thinks that these are enterprise feature. I got news for you, these are basic features for anyone who wants to build more than just a simple script for individual use. You work so hard to build these features but let only a small subset of users really do anything meaningful with it

sitkack 4y ago
Honest question, not a snark attack. But would it be possible to build those things on top of Cloudflare itself, esp the logging? For access control, you would have to have a separate process controlling the CF api for configuration.
- okaybuy 4y ago
  Not so simple when a lot of the tooling that facilitates easy development is tied to CF API, are you meant to build your own cloudflared and wrangler just so you dont have to grant everyone in the org super admin? This is where AWS wins, AWS gives you everything from $0/month, IAM in the above case, but you don't stay at $0 for long, where as CF gives you partial bits but you remain at $0-low/pm for longer Its actually really ironic the same company that says SAML for all for free, can't even get basic RAC on their own admin controls
jplevine 4y ago
We just announced something called the Workers Analytics Engine today: https://blog.cloudflare.com/workers-analytics-engine/ https://blog.cloudflare.com/workers-analytics-engine/ More coming soon! Happy to chat more - jpl at cloudflare dot com
imobbscf 4y ago
What would you like to see? Disclaimer: Engineer on the IAM team at Cloudflare

michaelrhansen 4y ago

How many of these commenters work at Cloudflare? :)

haroldadmin 4y ago

I like that Cloudflare is steadily becoming a serious alternative to the existing cloud providers. If you folks can build a decent IAM tool for access control to Cloudflare resources, much more people will start using your services.

VirusNewbie 4y ago

huh, Fastly is doing pub/sub lately too with fanout now: https://www.fastly.com/blog/fastly-fanout-why-real-time-messaging-and-edge-computing-are-an-amazing-combination https://www.fastly.com/blog/fastly-fanout-why-real-time-mess...

eska 4y ago

Since you guys seem to be using Rust, what client library do you recommend for MQTTv5 support?

cehrig 4y ago
(Pub/Sub Eng at Cloudflare) We are testing the broker using the Paho C wrapper https://github.com/eclipse/paho.mqtt.rust https://github.com/eclipse/paho.mqtt.rust .. and have added a client example to our dev-docs https://developers.cloudflare.com/pub-sub/examples/connect-rust/ https://developers.cloudflare.com/pub-sub/examples/connect-r...