9 ms·
Asyncio: A library with too many sharp corners
- TZubiri 1y agoOnly experience with this thing was when working with a node dev that was trying out python. What a mess, he didnt quite get how processes or threads worked, just learned node's model event / promise models and brought them to python
- zdragnar 1y agoIn all fairness, that's a pretty common experience with any transition between dislike platforms. I'll never forget one Java developer who transitioned to JS frontend work (pre-typescript days). Try/catch everywhere. There's a semi-accurate mindset that "good developers are good in any language" as if software engineers are fungible resources, but the transition time to a novel paradigm isn't really well appreciated.
- mbb70 1y agoI have seen "you can write Java in any language" proven true many times. The culture and idioms of Java impose great constraints than syntax. Not meant as a knock on Java or Java developers.
- zahlman 1y agoOne of the biggest problems with Python is how much Java is written in it. Including in the standard library (the most egregious examples being `logging` and `unittest`). And another big problem is how much C is written in it, even when the project also contains actual C.
- bsder 1y ago> In the traditional model of concurrent programming using threads, there is no clean way to do cancellation. The only solution is the one being complained about--for a thread to voluntarily check if it needs to be cancelled. Java deprecated stop() for good reasons. "Those who cannot remember the past are condemned to repeat it."
- krferriter 1y agoYou can't always control what the thread is doing, like what code is being executed. Even if you put flags and ensure your code is periodically checking them, your code is not necessarily the only thing being executed by the thread. It may be calling into external code. You need a way to interrupt it.
- zahlman 1y agoNo, that's exactly why you must not have "a way to interrupt it". Doing so could leave said external code in an invalid state and there would be no plausible way to do anything about it.
- Spivak 1y ago> You can't always control what the thread is doing, like what code is being executed. If this is the case then you can't use threads for this workload. If you're calling into some external code that could be doing whatever then one of the things it could be doing is holding locks. If you want a unit of execution you can murder with prejudice then you need the operating system's help which means a process. On both Linux and Windows the situations where you can safely kill a thread is when you have complete control and knowledge of what's running in it, the very opposite of when people want to be able to kill threads.
- zbentley 1y agoAgree. If genuine pre-emptive cancellation from the outside is needed, all languages support the primitive which allows that: a process. Many of the tradeoffs that come from using a process rather than a thread descend directly from that property of support for arbitrary-point cancellation.
- mikert89 1y agothe real problem is python, great at the start of a project, by the end i am dying to rewrite in java.
- echelon 1y agoPython needs a version 4 to make a clean break: - runtime optionally type checked interfaces - make depending on system python an antipattern - a reasonable package manager with lockfiles and transitive dependency versions - package org namespaces - steps towards hermetic builds with less linking against system libraries and dependencies - cleanup and slimming down of the standard library. Removal of cruft. Less "batteries included" so people lean more on packages and enforcing good practices with packages.
- Jabrov 1y agoThe latest versions of Python 3 already have or can do most of these things.
- echelon 1y agoThey do not enforce it. It's not about "can do". It's about defaults and enforcing stricter standards. Python is too flexible. It produces sloppy code. And in no world are Python builds and dependencies solved. It's a major headache.
- 7bit 1y agoIf you want Python to be like TypeScript, why don't you just write a typescript then?
- Philpax 1y agoPersonally, I do; my only interactions with Python are unwilling (work or AI-related code). Given that I don't have much of a choice in the matter, I'd like to see Python improve as a language and as an ecosystem. I hope that one day I will not feel dread upon encountering it in the wild.
- Philpax 1y agoThis seems like a microcosm of the general design problems that pervade Python. It is a fundamentally quirky language.
- alberth 1y agoAny recommended alternative?
- pinoy420 1y ago[dead]
- rangerelf 1y agoNim?
- jasonm23 1y agoCrystal.
- pansa2 1y agoNim is often recommended as an alternative to Python, but how similar is it? I know it has Python-like significant indentation, but I'd have thought syntax is the least important consideration when choosing a language?
- nilslindemann 1y agohttps://github.com/nimpylib/nimpylib https://github.com/nimpylib/nimpylib shows it can be made nearly identical.
- cb321 1y agoThis answer's @pansa2's question in a way that more leans on one aspect of Nim's greater syntactic & code transformation flexibility (which is great!), but I suspect pansa2 was more interested in differences in "idiomatic Nim" from Python for which there are more details here: https://github.com/nim-lang/Nim/wiki/Nim-for-Python-Programmers https://github.com/nim-lang/Nim/wiki/Nim-for-Python-Programm...
- btown 1y agoI really wish the community had coalesced around gevent. - no async/await, instead every possible thing that could block in the standard library is monkey-patched to yield to an event loop - this means that you can write the same exact code for synchronous and concurrent workflows, and immediately get levels of concurrency only bounded by memory limits - you'll never accidentally use a synchronous API in an async context and block your entire event loop (well, you can, if you spin on a tight CPU-bound loop, but that's a problem in asyncio too) - the ecosystem doesn't need to implement libraries for asyncio and blocking code, everything just works There's a universe where the gevent patches get accepted into Python as something like a Python 4, breaking some niche compatibility but creating the first ever first-class language with green-threading at its core. But in this universe, we're left with silly things like "every part of the Django ecosystem dealing with models must be rewritten with 'a' prefixes or be abandoned" and it's a sad place indeed.
- jeeyoungk 1y agoI like `gevent` but I think it may have been too hacky of a solution to be incorporated to the main runtime. "creating the first ever first-class language with green-threading at its core." ... isn't that what Go is? I think out of all languages I use extensively, Go is the only one that doesn't suffer from the sync/async function coloring nightmare.
- deleted 1y ago[deleted]
- pansa2 1y ago> I like `gevent` but I think it may have been too hacky of a solution to be incorporated to the main runtime gevent's monkey-patching might be hacky, but an official implementation of stackful coroutines (similar to Lua's) need not have been. Instead, stackless coroutines were chosen - maybe on merit, but also maybe because C#'s async/await keywords were in-vogue at the time and Python copied them.
- dontlaugh 1y ago
- pansa2 1y ago> asyncio has so many sharp corners and design issues [...] bordering on being fundamentally broken > other languages and libraries did what asyncio does significantly better > it is baffling how the library made it out of provisional status with such glaring flaws Unfortunately this seems common in Python 3.x. The hype-cycle focuses on a feature from another language(s), and Python rushes to add its own version without sufficient attention to its design, nor how the feature complements the rest of the language (if at all). Async, type hints and pattern matching all fit this description. `match` is probably the worst offender - it seems to have been added mainly so that novices choosing a language will not veto Python because it's missing the current feature-du-jour. [0] Type hints are at least having their flaws addressed - but the flaws can't be removed due to back-compat. This leads to a C++-like situation of "this is the modern way, these other approaches are deprecated". So much for "one way to do it". [0] https://discuss.python.org/t/pep-8012-frequently-asked-questions/487/2 https://discuss.python.org/t/pep-8012-frequently-asked-quest...
- privatelypublic 1y agoI don't feel calling modern Match/Case syntax a "du-jour" feature is plausible. Are you referencing something else?
- pansa2 1y ago> I don't feel calling modern Match/Case syntax a "du-jour" feature is plausible I do. To be clear: I'm not saying that (well-designed versions of) pattern-matching will become less useful over time, but that they will become less fashionable. Python will then, no doubt, add another language construct to appeal to those who choose languages solely based on a checklist of features.
- gshulegaard 1y agoI feel like this started/greatly accelerated when Guido stepped down as BDFL. Python at is on a path where the essence of what made it popular (readable, well designed, productive) is being crushed under the weight of it’s popularity. The language now feels bloated and needlessly complex in areas that were previously limited, but simple. I recently chased down a bug where something was accidentally made a class variable because a type hint was left off it by accident and it clicked for me that Python is not the same language I loved at the start of my career.
- dagenix 1y agoIn my experience, the key to using asyncio is to use anyio. Anyio is an interface that you can use ontop of asyncio and fixes most of its shortcomings. https://anyio.readthedocs.io https://anyio.readthedocs.io
- dade_ 1y agoAsyncio is too dull. Twisted for the win! My favourite insane package of wonders.
- lormayna 1y agoAnd the best way to write unmaintainable and undebuggable spaghetti code.
- dcreater 1y agoHow does httpx compare?
- zahlman 1y agoasyncio is for general concurrency; httpx is an http client. There is no comparison to make.
- o11c 1y agoFor the (brief) C comparison, I'm really not sure why this talks about `pthread_kill` but not `pthread_cancel` ...
- SkiFire13 1y ago`pthread_cancel` still has most of the issues described.
- o11c 1y agoIt's standard, it's persistent, and it automatically handled by well-formed code. What's missing?
- SkiFire13 1y agoIt will still leave resources in unknown and possibly inconsistent state, moreover you need to use `pthread_cleanup_push` to register everything that may need to be done to clean up the thread... for languages with RAII this basically means the destructor of every local variable, which isn't really reasonable, so you end up with ton of code where it won't work. You can claim it's "standard", but de-facto it really isn't and most code is not "well-formed" from its point of view.
- o11c 1y agoCancellation is just an exception, so destructors work automatically. Any implementation of a language that uses RAII without integrating cancellation into it is fundamentally broken and should be avoided at all costs. `pthread_cleanup_push` (or `__attribute__((cleanup))` is only needed if you're implementing it by hand. Since the set of functions that can be cancelled is documented (and pretty obvious - roughly, any blocking function but notably excluding mutex functions), there's no "unknown" involved. At most there is "I insisted on writing C but refused to take the effort to make my code correct in a threaded environment." Notably, `malloc` is not a cancellation point so this all is much easier to deal with than signals.
- anal_reactor 1y agoSir, this is Wendy's. I have a feeling that the core issue is that author is trying to implement complex communication mechanisms with a tool that was designed for much more basic usage. I don't understand why half of the article is about cancellations while cancellations as a concept are wrong. Have you ever tried to cancel peeing mid-pee? It just doesn't work. Not sure if the other half of the article is worth reading if first half has completely wrong premise. Or does this guy live in some utopian future where it's reasonable to assume that everything can be cancelled at any point.
- drkrab 1y ago“Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.” – Virding’s first rule of programming
- Starlevel004 1y agoOh cool. I wrote this. 100% hit rate on random posts of mine ending up on HN.
- lormayna 1y agoAs somebody that has written couple of services with Twisted in the late '00s, I feel asyncio a breath of fresh air. This not means that it's perfect, but it's a lot more usable than Twisted. I know that it's not possible, but I would like to have in Python something similar to goroutines, waitgroups and channels that Go has.
- rich_sasha 1y agoA big issue with asyncio is IME the documentation. The main document is a tutorial-style tour of the features, scarcely mentioning the important details. These are often hidden in comments on seemingly unrelated features or, more likely, reddit and SE. Some Python documentation indeed includes a tutorial, but also has a comprehensive documentation. This is not my experience with asyncio
- snakepit 1y agoI'm working on an ML platform in Elixir. Orchestrating CPU-intensive Python processes is central to the project, so I created a session based pooler called `snakepit` [1]. The latest revision is building a batteries-included gRPC Python bridge that enables streaming, bi-directional tool use, as well as an innovative variables feature for experimental ML inspired by ideas from DSPy's team. One of the later project goals: A Python client that can manage the Elixir orchestrator that manages pools of Python, in a distributed environment. Maybe I'll call that submodule `snakepits`. In this embodiment, it will be an effective albeit much more sophisticated replacement for `asyncio` for some use cases. [1] https://github.com/nshkrdotcom/snakepit https://github.com/nshkrdotcom/snakepit