7 ms·

> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support Because it's a distributed environme

• wongarsu 7y ago
  In fact it's precisely Python's deficiency in multithreading that lead to it having one of the best ecosystems for every other form of concurrency, like green threads and multiprocess applications.
• pletnes 7y ago
  If you’re doing (data analysis|simulations|Image processing) you can offload computation to numpy, which releases the GIL. This allows nice multicore speedups with python and threading. The same holds for various CPU intensive standard library functions implemented in C. The GIL issue is real, but posts like this one confused me for years. Please, don’t exaggerate GIL issues.
  • weberc2 7y ago
    Not everything is amenable to numpy and it's pretty easy to make performance worse by throwing numpy at every problem. For example, if your array contains Python objects that are part of an operation, you've likely just introduced a significant performance regression. Worse, there's no way to detect these regressions except to have performance tests. Please, don't understate GIL issues.
    • xapata 7y ago
      Why would you make a NumPy array of objects? Use a list until it makes sense to create an array.
      • weberc2 7y ago
        Because you’re using Numpy via an intermediate library like pandas and you have composite data in one column?
        • xapata 7y ago
          Ah. Pandas is the problem. Unfortunately, you need to understand how its features are implemented to use it well. Still, my main trouble with Pandas is unnecessary memory bloat, not compute inefficiency. That caveat is somewhat true for all programming abstractions, but well-designed interfaces make the more efficient techniques more obvious and beautiful, while the inefficient or risky techniques are made esoteric and ugly.
          • weberc2 7y ago
            Pandas isn't the problem, the problem is assuming that "$LIBRARY releases the GIL so things will be fast!". It's a pennywise, pound-foolish approach to performance. Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] or something and all of a sudden your program has a difficult to debug performance regression. In general, the "just rewrite the slow parts in C!" motto is terrible advice because it's unlikely that it will actually make your code appreciably slower, and if it does, it's very likely to be defeated unexpectedly as soon as requirements change. Using FFI to make things faster can work, but only if you've really considered your problem and you're quite sure you can safely predict relevant changes to requirements.
            • xapata 7y ago
              > Someone will write a function assuming the user is only going to pass in a list of ints and someone else will extend that function to take a list of Tuple[str, int] There are plenty of pitfalls in leaky abstractions. Establishing that fast numeric calculations only work with specific numeric types seems to help. One thing you seem to be encountering, that I've seen a few times, is that people don't realize NumPy and core Python are almost orthoganal. The best practices for each are nearly opposite. I try to make it clear when I'm switching from one to the other by explaining the performance optimization (broadly) in comments. Regardless, any function that receives a ``list`` of ints will need to convert to an ndarray if it wants NumPy speed. If the function interface is modified later, I think it's fair to expect the editor to understand why.
• weberc2 7y ago
  Even if your app is IO bound, Python's concurrency is painful. Because it's not statically typed, it's too easy to forget an `await` (causing your program to get a Promise[Foo] when you meant to get a Foo) or to overburden your event loop and such things are difficult to debug (we've had several production outages because of these class of bugs). Never mind the papercuts that come about from dealing with the sync/async dichotomy.
  • y3sh 7y ago
    And that even with async you're still bound by the GIL
    • jnwatson 7y ago
      Huh? With async, there's typically only ever 1 thread running, so there's no contention for the GIL.
  • jnwatson 7y ago
    Both problems have built-in debug solutions in recent versions of python. The event loop will literally print out all the un-awaited coroutines when it exits, and you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time.
    • weberc2 7y ago
      > The event loop will literally print out all the un-awaited coroutines when it exits IIRC, I've only ever seen "unawaited coroutine found" (or similar) errors; I've never seen anything that points to a specific unawaited coroutine. In either case, a bug in prod is still many times worse than compile time type error. > you can enable debug on the event loop and have it print out every time a coroutine takes longer than a configurable amount of time I don't run my production servers in debug mode, and even when I do manage to find the problem, I have limited options for solving it. Usually it amounts to refactoring out the offending code into a separate process or service. An extreme counterpoint is a language like Go which 1) Is roughly 100X faster in single-threaded, CPU-bound execution anyway 2) Allows for additional optimizations that simply aren't possible in Python (mostly involving reduced allocations and improved cache coherence) 3) Has a runtime that balances CPU load across all available cores This isn't a "shit on Python" post; only that concurrency really isn't Python's strong suit (yet).