3 ms·
looking at the release notes and the source, it looks like the asyncio interface is still using the old drivers, like psycopg2 for postgresql, and using a threa
by zzzeek 4y ago
looking at the release notes and the source, it looks like the asyncio interface is still using the old drivers, like psycopg2 for postgresql, and using a threadpool.
we at SQLAlchemy came up with a way to interface asyncio frontend and backend (like asyncpg for the driver) while maintaining all the "in the middle" code as synchronous style. the dark secret is that for this approach you have to use greenlet to propagate the "await" operations. It's been in our release for 18 months now with 1M downloads a day and there have been no problems reported with it, so i continue to wonder why nobody else seems to want to look at this approach. one downside which nobody has complained about yet is that it makes profiling tools like cProfile harder to use, but seems not to have come up yet.
there's also another approach, which is that you rewrite all your "in the middle" code as pure asyncio, then create a dummy task to implement your synchronous API. dark secret for that one is you had to rewrite everything and I'm not sure if there's other performance implications for having awaits all throughout code that's running only one task per thread.
- baq 4y agothanks for sqlalchemy. it's a shining star in the Python package ecosystem. > interface asyncio frontend and backend (like asyncpg for the driver) while maintaining all the "in the middle" code as synchronous style. the dark secret is that for this approach you have to use greenlet to propagate the "await" operations. It's been in our release for 18 months now with 1M downloads a day and there have been no problems reported with it, so i continue to wonder why nobody else seems to want to look at this approach. do you have a blog post or some other kind of writeup to read about it? maybe nobody else had the same idea and it works so well nobody is aware that it can be done...
- tmpz22 4y agoIs it a shining star or a rough dirty pragmatic get-the-job-done tool with a heart of gold thats approachable to newcomers and grizzled veterans alike. The trenches are too dirty for shining stars.
- zzzeek 4y agoI was a bit ranty but the gist at https://gist.github.com/zzzeek/d9c98c43553e43e76600f03dfb5e1af2 https://gist.github.com/zzzeek/d9c98c43553e43e76600f03dfb5e1... lays out pretty much everything that happened.
- snapcaster 4y agoJust want to say thanks for SQLAlchemy work! Love the package and use it in every one of my work and personal projects
- syastrov 4y agoSomeone mentioned it on a Django forum thread about async ORM [0]. It seems to have been ignored. Super cool idea and execution! I hope it gets some more exposure. [0] https://forum.djangoproject.com/t/asynchronous-orm/5925/51 https://forum.djangoproject.com/t/asynchronous-orm/5925/51
- bredren 4y agoMike, thanks for this reply and your work on SQLAlchemy. My employer uses your framework, but not yet the newest versions featuring async. Previously, and still in my personal projects, I have been a Django and Django ORM person. I got away with by and large avoiding SQL statements in building web apps and have sometimes struggled to understand why SQL statements seemed much more forward-facing in SQLAlchemy in comparison. Then I listened to your recent Talk Python podcast episode with M. Kennedy [1] and it helped me understand where the framework is coming from with regards to expectations for SQL capability from a developer that wants to use SQLAlchemy. I believe the focus was much more so on the dark secrets you mention above, though and colored in how you discovered those. If anyone wants to learn more about async or SQLAlchemy, I highly recommend listening. As an aside, it was also cool to hear you talking about your project because for me, it humanized you as the maintainer. You sounded pretty chill, which was for some reason not what I was expecting. I hope you can make it back on to that podcast with less of a gap between appearances next time. [1] https://talkpython.fm/episodes/show/344/sqlalchemy-2.0 https://talkpython.fm/episodes/show/344/sqlalchemy-2.0