5 ms·

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to ge

Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/async dichotomy.

y3sh 8y ago
And that even with async you're still bound by the GIL
- jnwatson 8y ago
  Huh? With async, there's typically only ever 1 thread running, so there's no contention for the GIL.
jnwatson 8y ago
Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.
- weberc2 8y ago
  > The event loop will literally print out all the un-awaited coroutines when it exits IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error. > you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time I don't run my production servers in debug mode, and even when I do manage to find the problem, I have limited options for solving it. Usually it amounts to refactoring out the offending code into a separate process or service. An extreme counterpoint is a language like Go which 1) Is roughly 100X faster in single-threaded, CPU-bound execution anyway 2) Allows for additional optimizations that simply aren't possible in Python (mostly involving reduced allocations and improved cache coherence) 3) Has a runtime that balances CPU load across all available cores This isn't a "shit on Python" post; only that concurrency really isn't Python's strong suit (yet).
  - mixmastamyk 8y ago
    These are not really an issue in vfx production and other things Python is used for.
    - weberc2 8y ago
      It’s a problem for lots of things Python is used for, but maybe not vfx (whatever that is).
      - mixmastamyk 8y ago
        They are using it for things it’s good at, for others they use java. So this subthread is largely a waste of time.
        weberc2 8y ago
        Who is “they”? What is your point?
        mixmastamyk 8y ago
        They is Netflix, and other post-production oriented users. You know, what this article and discussion is about?
        weberc2 8y ago
        It's not at all obvious that "they" refers to "netflix and other post-production oriented users", and your argument is a tautology "Python is good at the things that Python is good at". Obviously. The rest of us are debating what those things are or are not.
        mixmastamyk 7y ago
        The subject is well-trodden, there's not much to debate. Python is not good at threading, but works well in multiprocessing situations. Netflix is using it in the later situation, and not the former. Async is unlikely to be a use case either.
Too 8y ago
The missing await is a a very common fault indeed, they should have used another keyword like 'not_await' for that scenario to make the decision explicit. Pycharm at least will warn you if you call an awaitable without 'await' and without assigning it to a variable. If you assign it to a variable and pass it into another function that doesn't expect an awaitable, it's up to you to have added sufficient type annotations and run your code through some static checker like mypy. Running python at scale without mypy is kindof doomed to failed to begin with.

y3sh 8y ago

And that even with async you're still bound by the GIL

jnwatson 8y ago
Huh? With async, there's typically only ever 1 thread running, so there's no contention for the GIL.

jnwatson 8y ago

Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.

weberc2 8y ago
> The event loop will literally print out all the un-awaited coroutines when it exits IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error. > you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time I don't run my production servers in debug mode, and even when I do manage to find the problem, I have limited options for solving it. Usually it amounts to refactoring out the offending code into a separate process or service. An extreme counterpoint is a language like Go which 1) Is roughly 100X faster in single-threaded, CPU-bound execution anyway 2) Allows for additional optimizations that simply aren't possible in Python (mostly involving reduced allocations and improved cache coherence) 3) Has a runtime that balances CPU load across all available cores This isn't a "shit on Python" post; only that concurrency really isn't Python's strong suit (yet).
- mixmastamyk 8y ago
  These are not really an issue in vfx production and other things Python is used for.
  - weberc2 8y ago
    It’s a problem for lots of things Python is used for, but maybe not vfx (whatever that is).
    - mixmastamyk 8y ago
      They are using it for things it’s good at, for others they use java. So this subthread is largely a waste of time.
      - weberc2 8y ago
        Who is “they”? What is your point?
        mixmastamyk 8y ago
        They is Netflix, and other post-production oriented users. You know, what this article and discussion is about?
        weberc2 8y ago
        It's not at all obvious that "they" refers to "netflix and other post-production oriented users", and your argument is a tautology "Python is good at the things that Python is good at". Obviously. The rest of us are debating what those things are or are not.
        mixmastamyk 7y ago
        The subject is well-trodden, there's not much to debate. Python is not good at threading, but works well in multiprocessing situations. Netflix is using it in the later situation, and not the former. Async is unlikely to be a use case either.

Too 8y ago

The missing await is a a very common fault indeed, they should have used another keyword like 'not_await' for that scenario to make the decision explicit. Pycharm at least will warn you if you call an awaitable without 'await' and without assigning it to a variable. If you assign it to a variable and pass it into another function that doesn't expect an awaitable, it's up to you to have added sufficient type annotations and run your code through some static checker like mypy. Running python at scale without mypy is kindof doomed to failed to begin with.