7 ms·
Tworoutines in Python
- gok 8y agoIsn't it trivial to accidentally make this deadlock?
- deleted 8y ago[deleted]
- erdewit 8y agoNot really, it starts a sub-event loop that still handles all the events sequentially. What can be a problem is infinite recursion, if inside an event handler a new nested event loop is started.
- mehrdadn 8y agoSounds like Visual Basic's DoEvents() all over again?
- aaaaaaaaaab 8y agoThey are running the event loop recursively so it shouldn’t deadlock like an ordinary queue.
- sifoobar 8y agoOf all the methods for dealing with async calls I've come across, I strongly prefer the Smalltalk process flavor. The general idea is that main runs in a fiber (or process in Smalltalk lingo) out of the box. There's a way of starting new fibers, yielding to the next; and blocking calls yield automagically. And that's it. But talk is cheap, and Smalltalk is not coming back anytime soon; which is why I'm building a new language [0] around the same ideas. https://gitlab.com/sifoo/snigl https://gitlab.com/sifoo/snigl
- deleted 8y ago[deleted]
- nightfly 8y agoIsn't that pretty much how Erlang does things?
- ekimekim 8y agoYep, and golang, and python with gevent (linked in the article). The general approach is often called "green threads" or "userspace threads".
- sifoobar 8y agoGo runs NxM threading behind the scenes, it's far from the same thing. By choice, but it's not what I'm aiming for. And it needs to be built into the core (as opposed to gevent), otherwise it wont run as fast as it could and can't integrate deeply enough with IO to be seamless.
- seabrookmx 8y agoWhen doing network calls golang will yield in a similar fashion as described in the thread above. However this is standard library magic and and user code will indeed run in the context of a goroutine which is NxM threading.
- sifoobar 8y agoErlang throws plenty of other ideas into the mix though, I prefer my concerns separated. It assumes multiple threads for example, in plenty of applications that's needless complexity. You end up sending messages between processes when all you wanted to do was encapsulate some state.
- dnautics 8y agoElixir solves this with Agent.
- Walkman 8y agoI implemented something like this not long ago: class Api(AsyncApi): def __init__(self, token): try: self._loop = asyncio.get_event_loop() except RuntimeError: # when running in a thread, get_event_loop doesn't create another one self._loop = asyncio.new_event_loop() asyncio.set_event_loop(self._loop) self._session = aiohttp.ClientSession(loop=self._loop) super().__init__(token, self._loop, self._session) @lru_cache(maxsize=None) def __getattribute__(self, name): attr = super().__getattribute__(name) if name.startswith("_") or not asyncio.iscoroutinefunction(attr): return attr def call_sync(*args, **kwargs): coro = attr(*args, **kwargs) return self._loop.run_until_complete(coro) return call_sync It can run when there is no loop or even when a loop is already running.
- jakobegger 8y agoI'm not familiar with async in Python, but creating a blocking wrapper around an async call is something I've had to do quite often in various circumstances. Theoretically, you shouldn't need to do that, but in practice there's always some big legacy component that you can't rewrite to make it asynchronous. In my experience it's such a common pattern that the language should support it. But in many languages it ends up being quite complex or inelegant. Makes me feel like there's a bit of a schism between language designers and users: designers want an architecture that encourages clean code, users want duct tape.
- jacob019 8y agoThis is why I love gevent. Synchronous code becomes asynchronous automatically and it's rarely necessary to reason about the asynchronicity.
- ekimekim 8y agoI'll add something I've observed to that: In asyncio and similar systems, the choice between "is this code syncronous or asyncronous" is made at declaration time - either you define a sync function or an async one, and you can only use it in that way. In gevent and similar systems, the same choice is made at call time - there's no distinction between sync and async functions, instead you can call syncronously with "foo()" or asyncronously with "future = gevent.spawn(foo)" (I'm taking some liberties by calling the returned greenlet object a future but it can be used as such).
- meowface 8y agoExactly. gevent has been a joy to work with since day one. asyncio (even with the new syntax additions) has been a complete pain for me since day one. I still use gevent for everything - even in brand new Python 3.7 projects - and I see no reason to stop. Great API, great performance, "just works".
- ekimekim 8y agoAs much as I love gevent I can't claim in good conscience that it always "just works" for me. It tends to just work until it doesn't, and then you're stuck investigating how a dependency of a dependency just switched to using a network library implemented in C that therefore isn't monkey-patchable and causes process-wide stalls. I've gotten good at diagnosing these issues over many years as a heavy gevent user, and it doesn't stop me highly reccomending gevent to anyone who will listen, but it's a caveat that should be mentioned for anyone new starting out.
- jacob019 8y agoWhat libraries have you run into trouble with?
- spamizbad 8y agoThat's an clever way to go about solving the sync/async situation. I work in an ecosystem that has both Python 2 and Python 3.6 async applications, so my code has to play nice with both. I often end up writing APIs as such: from __future__ import [Python 3 niceties] class BaseApi(object): def _method_that_does_io(self, *args, **kwargs): raise NotImplementedError def do_a_thing(self, stuff): return self._method_that_does_io(stuff) and then create a SyncAPI and AsyncAPI respectively in different files, each subclassing BaseApi implementing the respective io-bound helper-methods. So Python 2 people can do: from SomeAPI.sync import SomeAPI and Python 3 people can run: from SomeAPI.py3async import SomeAPI The synchronous code is written in such a manner it's both 2 and 3 compatible, so every flavor can be happy.
- Xophmeister 8y agoBut how can that work when the async methods have a different signature and calling convention to their sync counterparts? Your “public” method in the async version would have to handle all the async set up and submission of your chain of coroutines to the loop. That’s possible, but it would have to live in isolation (i.e., no other async code could run simultaneously, which kind of defeats the purpose) and it means for a lot of nasty plumbing, which doesn’t seem like it should be the class’ job (in a separation of responsibilities sense).
- spamizbad 8y agoOh you can do it with identical signatures. Let’s say we are making http requests, with the sync code using requests and the async code using aiohttp. Your public methods all call a private _request method with a verb, a url, and optionally something to be seriaized to json. Your async _request method does need to do a small bit of code around sessions: make one first if one hasn’t been made for the class instance yet (or override your __init__ and take an existing one as an arg) but at the end of the day it’s just returning a coroutine that’ll ultimately give you a deserialized response. Sync can just be more direct and return a dict. Both have the same signature.
- yoklov 8y agoPrior to the introduction of async/await (or promises) there were a number of places we used a similar technique inside Firefox to avoid callback spaghetti. For us it caused no end of problems, and were glad to see it go (well, there are still probably some places using it...). It's not surprising to me that it's discouraged.
- veli_joza 8y agoCould you point out what the problems were, and what technique did you replace it with?
- yoklov 8y agoA long tail of subtle event and reentrancy bugs, where an event loop wasn't getting an event it needed (leading to dropped or mishandled OS events), or a function that doesn't expect to need to be reentrant was suddenly reentrant. Most of these could be worked around, but some of them are unreasonably difficult given OS event apis. We replaced it with just making the calls that might need to be asynchronous use `async`/`await`.
- csytan 8y agoIPython has a pretty neat feature where you can await at the top level: https://ipython.readthedocs.io/en/stable/whatsnew/version7.html#autowait-asynchronous-repl https://ipython.readthedocs.io/en/stable/whatsnew/version7.h...
- pickdenis 8y agoNim's `multisync` macro is another cool approach to this. It allows the function to be called isomorphically regardless of synchronicity, determining the latter using the type system. In the following example, the `writeResponse` proc can be called from either a synchronous environment on a `Stream` or from an asynchronous environment using `AsyncStream`. https://github.com/sid-code/nmoo/blob/master/src/nmoo/sidechannel.nim#L17 https://github.com/sid-code/nmoo/blob/master/src/nmoo/sidech... There is a bit of glue code holding this together, but I find it really neat. There's no event loop hack, just static polymorphism.
- dom96 8y agoNice to see that others are finding this useful. :)
- rs86 8y agoWhy Python? Couldn't hire real programmers?
- dang 8y agoPlease don't do this here.
- carapace 8y agoIn any event, Functional Reactive Programming is thought-provoking. Here's a link to "Elm: Concurrent FRP for Functional GUIs", Evan Czaplicki's thesis (although I think Elm-lang recently simplified their system.) https://www.seas.harvard.edu/sites/default/files/files/archived/Czaplicki.pdf https://www.seas.harvard.edu/sites/default/files/files/archi...
- andrewstuart 8y agoI thought this post my have been my Twolang gaining in public attention https://github.com/bootrino/twolang https://github.com/bootrino/twolang but it seems not.
- lukeplato 8y agowhat kind of website is this?
- snicker7 8y agoI love the interface, but I don't like the idea of monkey-patching the standard library. Can't one instead spawn a separate thread and run the event loop there?
- alehander42 8y agoNim's multisync macro is another nice approach: on compile time it transforms the same function to a sync and async variant
- jlkjfasdfwm2 8y agoI hope Python & Node will evolve from async/await, because we can do better. Take a look at Lua coroutines
- infinity0 8y agoor just learn Haskell and realise that this whole concept is just monadic bind, written simply as >>=
- BerislavLopac 8y agoI'm not sure, but it seems an overly complex approach. If all you want is to execute an async function from a sync context, run and wait for it: >>> import asyncio >>> async def foo(): ... return 'foo' ... >>> bar = asyncio.run(asyncio.wait_for(foo(), timeout=15)) >>> print(bar) foo >>>