4 ms·

Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC an

Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view functions that Crossbar calls when messages arrive (meaning users wont touch those directly) or perhaps going directly to Django with scripts (Django management commands)? About your wish list: I understand you'd build these features on top of, or relying on crossbar.io? This sounds like something regular Django wouldn't have a place in. I'd very much like a system with features these features. Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate.

sametmax 10y ago
> Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view functions that Crossbar calls when messages arrive (meaning users wont touch those directly) or perhaps going directly to Django with scripts (Django management commands)? There are a lot of things to get Django integration because Django is a lot of things, but I can see 3 points worth mentioning quickly: - auth. Crossbar has it's own version of auth, which means you need to create a WAMP auth service and call django.auth.contrib's authenticate() function in it to get authentification in it. It also means you need to create some manual hook for logging out. - pub/sub and rpc: crossbar has an HTTP bridge, and it's probably the best way to get Django in the loop for all events. However, you must remember than you may have many Django processes in production, and only one will receive each pub. - orm. Django's ORM is blocking, meaning you have to run all calls to it into loop.run_in_executor. It's very annoying and there is no way around it. > I understand you'd build these features on top of, or relying on crossbar.io? Yes. Although the webpart would be handled by something else, but connected with an integrated wrapper. > This sounds like something regular Django wouldn't have a place in. Indeed. Django being blocking, you can't really make it the center piece of such a work. You could, however, provide adapters so you can integrate with existing Django project to ease transitions of existing code bases. > I'd very much like a system with features these features. Maybe I should start a kickstarter or something like, gather a team and work on it. I gave it a try in short sprints, but a few hours does not cut it. After a while I ended up opening several tickets on the official asyncio repository because as soon as you work that close to the event loop, you run into edge cases you must take in consideration. > Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate. We can't blame Django for that though. It has been a useful design for a long time, plus the design is still holding quite well today. It's just starting to show limitions because we compare it to much more recent techs. Still, I agree. AppConfig and Django channels are hacks written as a workaround of the framework design. But let's not kid ourself, the amount of work that went into Django is enormous, and it's still the framework making me the most productive to this day. Any alternative would have a hard time just catching up with what it can do before even starting to offer killer features that Django doesn't have. Some other issues: - async is hard. A good async framework should aim at making common task as easy as possible, including debugging. This is really hard, and few developpers spend time on that. - async is hard (again). A good async framework should aim at providing a solid even management framework, with a clear lifecycle and robusts hooks. This is really hard, and few developpers spend time on that. - configuration is un unsolved problem in Python. There is no such thing as a "standard configuration framework" and we all end up parsing config files, reinventing live settings in database, or dealing with redis/etcd. But configuration is a really, really hard problem : you have serialisation, access rights, value sharing, data validation, hierarchy, handling of different data source and several layers of conf depending of where in the life cycle you are in an app. This is a project on it's own. - crossbar code base is not made to be integrated. It's a design decision from Tavendo's team: provide a piece of infrastructure (like nginx, redis, etc) and not a library or a framework. So you would probably have to make some heavy mocking and monkey patching to bend it to a programmable API. Or, god save us, making a fork.
- jarpineh 10y ago
  Thanks for the tips about Django with Crossbar. It doesn't sound entirely smooth sailing ;) Be that as it may, we got some IoT things starting and it might fit the bill. I'd certainly back such a kickstarter ;) I took a look at asyncio just a few days ago and got dizzy fast. As you said, async is hard. I just wish it could be more explicit without having all that boilerplate. These days doing much more functional programming I'm thinking that configuration should be data instead of global state we got now. I think that Django is still great and keeps getting better. I'm not about to replace it. I just hope it wasn't so hard to use functionality and data in Django projects by less conventional means (such as when async ops, long running processing or keeping track of changes is required).

sametmax 10y ago

> Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view functions that Crossbar calls when messages arrive (meaning users wont touch those directly) or perhaps going directly to Django with scripts (Django management commands)? There are a lot of things to get Django integration because Django is a lot of things, but I can see 3 points worth mentioning quickly: - auth. Crossbar has it's own version of auth, which means you need to create a WAMP auth service and call django.auth.contrib's authenticate() function in it to get authentification in it. It also means you need to create some manual hook for logging out. - pub/sub and rpc: crossbar has an HTTP bridge, and it's probably the best way to get Django in the loop for all events. However, you must remember than you may have many Django processes in production, and only one will receive each pub. - orm. Django's ORM is blocking, meaning you have to run all calls to it into loop.run_in_executor. It's very annoying and there is no way around it. > I understand you'd build these features on top of, or relying on crossbar.io? Yes. Although the webpart would be handled by something else, but connected with an integrated wrapper. > This sounds like something regular Django wouldn't have a place in. Indeed. Django being blocking, you can't really make it the center piece of such a work. You could, however, provide adapters so you can integrate with existing Django project to ease transitions of existing code bases. > I'd very much like a system with features these features. Maybe I should start a kickstarter or something like, gather a team and work on it. I gave it a try in short sprints, but a few hours does not cut it. After a while I ended up opening several tickets on the official asyncio repository because as soon as you work that close to the event loop, you run into edge cases you must take in consideration. > Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate. We can't blame Django for that though. It has been a useful design for a long time, plus the design is still holding quite well today. It's just starting to show limitions because we compare it to much more recent techs. Still, I agree. AppConfig and Django channels are hacks written as a workaround of the framework design. But let's not kid ourself, the amount of work that went into Django is enormous, and it's still the framework making me the most productive to this day. Any alternative would have a hard time just catching up with what it can do before even starting to offer killer features that Django doesn't have. Some other issues: - async is hard. A good async framework should aim at making common task as easy as possible, including debugging. This is really hard, and few developpers spend time on that. - async is hard (again). A good async framework should aim at providing a solid even management framework, with a clear lifecycle and robusts hooks. This is really hard, and few developpers spend time on that. - configuration is un unsolved problem in Python. There is no such thing as a "standard configuration framework" and we all end up parsing config files, reinventing live settings in database, or dealing with redis/etcd. But configuration is a really, really hard problem : you have serialisation, access rights, value sharing, data validation, hierarchy, handling of different data source and several layers of conf depending of where in the life cycle you are in an app. This is a project on it's own. - crossbar code base is not made to be integrated. It's a design decision from Tavendo's team: provide a piece of infrastructure (like nginx, redis, etc) and not a library or a framework. So you would probably have to make some heavy mocking and monkey patching to bend it to a programmable API. Or, god save us, making a fork.

jarpineh 10y ago
Thanks for the tips about Django with Crossbar. It doesn't sound entirely smooth sailing ;) Be that as it may, we got some IoT things starting and it might fit the bill. I'd certainly back such a kickstarter ;) I took a look at asyncio just a few days ago and got dizzy fast. As you said, async is hard. I just wish it could be more explicit without having all that boilerplate. These days doing much more functional programming I'm thinking that configuration should be data instead of global state we got now. I think that Django is still great and keeps getting better. I'm not about to replace it. I just hope it wasn't so hard to use functionality and data in Django projects by less conventional means (such as when async ops, long running processing or keeping track of changes is required).