3 ms·

Crossbar.io: Real-Time Distributed RPC in Python (Interview)

62 points by sametmax 10y ago

sametmax 10y ago
I've been following crossbar for 2 years now, wrote some doc and tutorials for the team, but only this week have I finally used it in production. Made a real time dashboard for a big transport facility, coupled with vuejs and django. In short, it's a fantastic piece of software: RPC and Pub/Sub can't be easier and cleaner. The fact it also packs a process manager, a static file server and a WSGI servers makes it amazing. But we really need a framework around it because right now it's a lot of manual work. It would probably take a year or so to make something worth it and not "yet-another-microframework", but I've been studying the beast for some time and using it would allow to create a Python framework loaded out of the box with: - full async web (and non web) microservices - live settings, including routing - task queues, with PUB/SUB talking directly to the browser - task runner, including for building static assets - advanced cache busting strategies - hot reloading to help you in dev, or to deploy in prod - load balancing and fault tolerance strategies On top of the regular features. Additional stuff we could add, not related to crossbar: - configuration framework (there is literally no good general solution for configuration management out there. The less terrible being pyramid's and zope's) - life cycle and error handling helpers (this is somewhere async framework fails big time) with clean logging, event model and debug tooling. - wrappers offer a micro-framework like API that you can unfold into a full-scale framework when your needs grow. Aend not just one or the other. - clear story for DB. Qraphql is something to explore but it's a hard topic, with bad asyncio support for ORM and nosqlite db. - 2 pass template rendering. Meta template generating templates, then generated templated used on the fly as views. This allow many tricks like pregenerating translated page, URL routing, static assets name based on hash, etc. - administration web console, not for the database, but for the project itself. We really need something to compete with new gen frameworks in Python. Right now there is nothing trying to be on par with meteorjs and the likes. The trap is that Django is good enough right now, so investing the time and resource into dev something THIS new is hard to justify. And I read the code base, it's not easy to groke (async never is anyway, but here you have to manually deal with the event loop) so finding a team would be hard, and doing it alone even harder.
- StavrosK 10y ago
  I haven't seen crossbar.io, so I can't really comment, but the features that you describe sound fantastic for making web apps. Do you think django-channels gets us much of the way there?
  - sametmax 10y ago
    Django channel is a way to mitigate django limitations. In that sense it does work but it doesn't do a 10th of what crossbar do. Channels let you do websocket, but crossbar add a standardizes a protocol on top of it for rpc and pub sub. It propagate errors for you, allow wild cards in routing, load balancing for rpc, and allow php, c#, js and python code to talk to each others transparently. But the best feature is: crossbar route all messages for you and clients don't need to know about other clients, only about the crossbar. It takes a time to realize how important this is, but it changes the whole architecture of your app. Channels are basically only non blocking data pipes for django, with one for websocket. Useful, but nowhere as powerful. I've made a living with django for years and love the framework, but channels feel like a hack to me. I hope for a django 2.0 with backed in async/await.
- detaro 10y ago
  Last time I looked at it, properly authenticating the browser clients made it quite hard to use (you could get an auth-less prototype up quite quickly, and then had to replace basically everything with RPC calls to add the necessary security checks, unless we missed a key trick). I assume this has changed?
  - sametmax 10y ago
    Authentication is working quite well but involves a lot of manual work right now. Ex: for django you need to create an auth microservice calling django.contrib.auth authenticate function. Now as for pub/sub, I use uuid to namespace messages and rpc call to fetch the namespace if i want to limit their access. It's imperfect but you have the same issue with redis or bare socket.js. However, last year tavendo added meta events and rpc allowing you to react to sub and control them from the api, so you can roll the policy you want. Again, manual work. Hence the framework I wish we had. The thing is, tavendo targets IoT with crossbar and is developping it's acticity for the factories and office automation. If we want a mature web solution, somebody will have to code it.

62 points by sametmax 10y ago

I've been following crossbar for 2 years now, wrote some doc and tutorials for the team, but only this week have I finally used it in production. Made a real time dashboard for a big transport facility, coupled with vuejs and django. In short, it's a fantastic piece of software: RPC and Pub/Sub can't be easier and cleaner. The fact it also packs a process manager, a static file server and a WSGI servers makes it amazing. But we really need a framework around it because right now it's a lot of manual work. It would probably take a year or so to make something worth it and not "yet-another-microframework", but I've been studying the beast for some time and using it would allow to create a Python framework loaded out of the box with: - full async web (and non web) microservices - live settings, including routing - task queues, with PUB/SUB talking directly to the browser - task runner, including for building static assets - advanced cache busting strategies - hot reloading to help you in dev, or to deploy in prod - load balancing and fault tolerance strategies On top of the regular features. Additional stuff we could add, not related to crossbar: - configuration framework (there is literally no good general solution for configuration management out there. The less terrible being pyramid's and zope's) - life cycle and error handling helpers (this is somewhere async framework fails big time) with clean logging, event model and debug tooling. - wrappers offer a micro-framework like API that you can unfold into a full-scale framework when your needs grow. Aend not just one or the other. - clear story for DB. Qraphql is something to explore but it's a hard topic, with bad asyncio support for ORM and nosqlite db. - 2 pass template rendering. Meta template generating templates, then generated templated used on the fly as views. This allow many tricks like pregenerating translated page, URL routing, static assets name based on hash, etc. - administration web console, not for the database, but for the project itself. We really need something to compete with new gen frameworks in Python. Right now there is nothing trying to be on par with meteorjs and the likes. The trap is that Django is good enough right now, so investing the time and resource into dev something THIS new is hard to justify. And I read the code base, it's not easy to groke (async never is anyway, but here you have to manually deal with the event loop) so finding a team would be hard, and doing it alone even harder.

StavrosK 10y ago
I haven't seen crossbar.io, so I can't really comment, but the features that you describe sound fantastic for making web apps. Do you think django-channels gets us much of the way there?
- sametmax 10y ago
  Django channel is a way to mitigate django limitations. In that sense it does work but it doesn't do a 10th of what crossbar do. Channels let you do websocket, but crossbar add a standardizes a protocol on top of it for rpc and pub sub. It propagate errors for you, allow wild cards in routing, load balancing for rpc, and allow php, c#, js and python code to talk to each others transparently. But the best feature is: crossbar route all messages for you and clients don't need to know about other clients, only about the crossbar. It takes a time to realize how important this is, but it changes the whole architecture of your app. Channels are basically only non blocking data pipes for django, with one for websocket. Useful, but nowhere as powerful. I've made a living with django for years and love the framework, but channels feel like a hack to me. I hope for a django 2.0 with backed in async/await.
detaro 10y ago
Last time I looked at it, properly authenticating the browser clients made it quite hard to use (you could get an auth-less prototype up quite quickly, and then had to replace basically everything with RPC calls to add the necessary security checks, unless we missed a key trick). I assume this has changed?
- sametmax 10y ago
  Authentication is working quite well but involves a lot of manual work right now. Ex: for django you need to create an auth microservice calling django.contrib.auth authenticate function. Now as for pub/sub, I use uuid to namespace messages and rpc call to fetch the namespace if i want to limit their access. It's imperfect but you have the same issue with redis or bare socket.js. However, last year tavendo added meta events and rpc allowing you to react to sub and control them from the api, so you can roll the policy you want. Again, manual work. Hence the framework I wish we had. The thing is, tavendo targets IoT with crossbar and is developping it's acticity for the factories and office automation. If we want a mature web solution, somebody will have to code it.