8 ms·

I see a lot of comments that are talking about how python does not have a go-like concurrency story. Fyi - python ASGI frameworks like fastapi/Starlette are th

I see a lot of comments that are talking about how python does not have a go-like concurrency story. Fyi - python ASGI frameworks like fastapi/Starlette are the same developer experience as go. They also compete on techempower benchmarks. Also used in production by Uber, Microsoft,etc. A queue based system is used for a very different tradeoff of persistence vs concurrency. It's similar to saying that the usecase for Kafka doesn't exist because go can do concurrency. Running "python -m asyncio" launches a natively async REPL. https://www.integralist.co.uk/posts/python-asyncio/#running-async-code-in-the-repl https://www.integralist.co.uk/posts/python-asyncio/#running-... Go play with it ;)

pinkbeanz 7y ago
I think a hard part with lots of these “what do I use x for” examples is it starts with the tool and the discusses the problem that it solves. I find it more helpful to start with a problem, and discuss the various tools that address it, in different ways. Forget email, say you have a app that scans links in comments for maliciousness. You rely on an internal api for checking against a known blacklist, which follows shortened links first, and an external api from a third party. You want the comment to appear to submit instantly to the poster but are comfortable waiting for it to appear for everyone else. What are your options? You could certainly use message queues and workers. If you’re cloud native maybe you leverage lambdas. Maybe you spin up an independent service that does the processing and inserting into the database in the background, and all you need to do is send a simple HTTP request on an internal network. Your solution depends on your throughout requirements, the size of your team and their engineering capabilities, what existing solutions you have in place. Everything has its pros and cons. Pretending that celery/redis is useless and would be solved if everyone just used Java ignores the fact that celery and redis are widely popular and drive many successful applications and use cases.
- StavrosK 7y ago
  While I agree with the rest of your comment, the sentence "if you’re cloud native maybe you leverage lambdas" made me irrationally angry.
  - CalRobert 7y ago
    Can you explain why? I use lambdas often and they seem to solve the problems they're meant for well.
    - StavrosK 7y ago
      It wasn't the lambdas, it was the combination of "cloud-native", which is a very salesmany term, and "leverage", which is my pet hate word. It's exactly as useful as "use", only much more pretentious. I'm just easily triggered with language :P More off-topic (or, rather, on-topic), I find lambdas great for things like a static website that needs a few functions. I especially like how Netlify uses them, they seem to fit that purpose exactly.
      - normalnorm 7y ago
        > I'm just easily triggered with language :P Me too! It makes me irrationally angry when people regurgitate linguistic clichés. I was already mad with: "python does not have a go-like concurrency story" when it would be enough (and 1000x less cringe) to say: "python does not have go-like concurrency" I think these mindless clichés make language really ugly and dysfunctional, and even worse they are thought-stoppers, because they make the reader/listener feel like something smart is being said, because they recognize the "in-group" lingo. In my experience, people get really offended when you point this out. It's kind of an HN taboo to discuss this. Which is also interesting in itself. Going forward we should pay more attention to our communication use cases. Btw: I wonder if we can stack several of these clichés. For example: "leverage" + "use case" = "leverage case".
        StavrosK 7y ago
        I agree, I think Orwell's "Politics and the English Language" is spot on here. I try to use simpler language whenever possible, I agree that people think that using longer words makes them sound smart but is just worse for communication. I've found it's a taboo to discuss anything even slightly personal. People are averse to feeling bad, so criticism needs to be extremely subtle in order to not offend. > Btw: I wonder if we can stack several of these clichés. For example: "leverage" + "use case" = "leverage case". I hate you for even thinking of this.
- lmeyerov 7y ago
  We've been experimenting with nice compromises for using pydata on-the-fly compute with caching. Basic tension is indiv requests get blocked by their blocking APIs, while caches (eg, IP-pinned) are best when same-python-app-thread. Right now, we do hypercorn multiproc -> per-proc quart/asyncio/aiohttp IP-pinned event loop -> Apache arrow in-app cache -> on-gpu rapids.ai cache. But not happy w event loop due to pandas/rapids blocking if heavy datasets by concurrent users. (Taking us back to celery, redis, etc, which we don't want due to extra data movement..) Maybe we can get immutable arrow buffs shared across python proc threads.. Ideas welcome!
BiteCode_dev 7y ago
Besides, serious projects in Go do use additional tools for creating task queues because they need to handle various stages of persistence, error handling, serialization, workflow, etc. Which are not stuff you want to write by hand yourself. If you don't need all that, then it's not a problem in Python either. You don't even need asyncio, you can just use a ThreadPool or a ProcessPool, dump stuff with pickle/shelve/sqlite3 and be on your way.
Carpetsmoker 7y ago
I was using gevent in Python about 10 years ago, and from memory, it's roughly similar to goroutines. It's not exactly the same of course, but just like goroutines it's pretty easy to just spawn a few jobs off, wait for them to finish, and get the results. It's not in the standard library, and there are probably other options too now (not a heavy Python user any more), but Python has had easy parallelism for at least a decade (probably longer).
- eloff 7y ago
  Gevent is like goroutines with GOMAXPROCS=1. Which is to say not nearly as useful. It gives you concurrency without parrallelism, because Python never did shake the GIL. Which goes to show some technical debt will haunt you forever.
  - antoinealb 7y ago
    To nuance your comment, you can still get some form of parallelism, "just not" thread parallelism in Python. You can still spawn multiple handler process, or have threaded code in a C extension.
    - zepto 7y ago
      It’s arguable whether either of those are ‘in python’.
      - quacker 7y ago
        The multiprocessing module in the standard library is absolutely a Python-native way to do parallelism: with Pool(5) as p: print(p.map(f, [1, 2, 3])) This runs f(1), f(2), and f(3) in parallel, using a pool of five processes. https://docs.python.org/3.6/library/multiprocessing.html https://docs.python.org/3.6/library/multiprocessing.html
        goostavos 7y ago
        Whoa! It never occurred to me that `Pool` could be used with context manager. I've always typed out `.close()` manually like a sucker.
jakearmitage 7y ago
> same developer experience as go You can't say that when Python shoves Pip/Poetry/VirtualEnv/Black/Flake and what not into people. In contrast, Go has built-in package management and gofmt. Python is essentially gate-keeping new devs.
- morelisp 7y ago
  Go may be unique in being the only ecosystem built after Python that can't claim it avoided Python's packaging disasters.
  - mdtusz 7y ago
    Cargo works pretty well too.
    - takeda 7y ago
      Cargo is not from Go
      - jessaustin 7y ago
        As a name for a package manager, "cargo" certainly appears more suitable for "go" than for "rust".
  - takeda 7y ago
    I believe Python suffers from no leadership in that space (everyone creates their own packaging, every tutorial advocates something different, many tutorials are outright wrong). There was also a bad decision of using Python code for installation (setup.py) instead of a declarative language. Most of that issues are actually fixed in setuptools if you put all settings in setup.cfg and just call empty setup() in setup.py. Like here: https://github.com/takeda/example_python_project/blob/master/setup.cfg https://github.com/takeda/example_python_project/blob/master...
  - throwaway894345 7y ago
    How do you figure? Go's packaging is wayyyy better than Python's. I've done considerable work with each and while Go's ecosystem has warts here and there, it's far from disastrous. I can't say that about Python. If nothing else, Go lets you distribute a static binary with everything built in, including the runtime. Python's closest analog is PEX files, but these don't include the runtime and often require you to have the right `.so` files installed on your system, and they also don't work with libraries that assume they are unpacked to the system packages directory or similar. In general, it also takes much longer to build a PEX file than to compile a Go project. Unfortunately, PEX files aren't even very common in the Python ecosystem.
throwaway894345 7y ago
> Fyi - python ASGI frameworks like fastapi/Starlette are the same developer experience as go. Can you provide some context for this statement? I've used Python asyncio extensively in Fargate (no ASGI frontend), and the developer experience is far from Go; however, I don't see how an ASGI framework can fix this. It seems like it offers the same general course-grained parallelism that you get from a containerized environment like Fargate except that Fargate abstracts over a cluster while ASGI frameworks presumably just abstract over a handful of CPUs. For example, we have a large data structure that we have to load and process for each request. We want to parallelize the processing of that structure, but the costs to pickle it for a multiprocessing approach are much too large. We've considered the memory mapped file approach, but it has its own issues. We're also looking at stateful-server solutions, like Dask, but now we're talking about running infrastructure. In Go, we could just fork a few goroutines and be on our way.
- sandGorgon 7y ago
  you should ask this exact same question here - https://github.com/tiangolo/fastapi/ https://github.com/tiangolo/fastapi/ i dont claim to have expertise in your business domain, but you should get the answer here.
zaro 7y ago
I've played with the concurrency in Python, and it's simply not worth it. Much better to use Node.js or Go where the async story is not an afterthought. Of course if you are stuck with Python it's better than nothing.
solidasparagus 7y ago
The difference is that it doesn't play well with large parts of the existing Python ecosystem. Concurrency is a pain if you are writing a library or trying to integrate with existing code. Python async is powerful, but: (1) it's undersupported (e.g. you can in theory download s3 files using async botocore, but in practice it is hard to use because of strict botocore version dependencies) (2) it isn't natural - once you're in the event loop it makes sense, but using an event loop alongside normal python code is confusing at best. (3) it got introduced too late. The best async primitives are only available in pretty recent versions of python3. The difference with go is that it go has these primitives built in from the beginning and using them doesn't introduce interoperability problems.
- jatone 7y ago
  and on top of it go is performant enough that 90% of the time you don't need them to begin with.
  - hackworks 7y ago
    Most common use case for concurrency is for IO intensive workloads. When it comes down to IO, programming language hardly matters. If you have CPU intensive workload, an optimizing compiler can help. Well, for that you have C or other languages with more mature and aggressive compilers.

pinkbeanz 7y ago

I think a hard part with lots of these “what do I use x for” examples is it starts with the tool and the discusses the problem that it solves. I find it more helpful to start with a problem, and discuss the various tools that address it, in different ways. Forget email, say you have a app that scans links in comments for maliciousness. You rely on an internal api for checking against a known blacklist, which follows shortened links first, and an external api from a third party. You want the comment to appear to submit instantly to the poster but are comfortable waiting for it to appear for everyone else. What are your options? You could certainly use message queues and workers. If you’re cloud native maybe you leverage lambdas. Maybe you spin up an independent service that does the processing and inserting into the database in the background, and all you need to do is send a simple HTTP request on an internal network. Your solution depends on your throughout requirements, the size of your team and their engineering capabilities, what existing solutions you have in place. Everything has its pros and cons. Pretending that celery/redis is useless and would be solved if everyone just used Java ignores the fact that celery and redis are widely popular and drive many successful applications and use cases.

StavrosK 7y ago
While I agree with the rest of your comment, the sentence "if you’re cloud native maybe you leverage lambdas" made me irrationally angry.
- CalRobert 7y ago
  Can you explain why? I use lambdas often and they seem to solve the problems they're meant for well.
  - StavrosK 7y ago
    It wasn't the lambdas, it was the combination of "cloud-native", which is a very salesmany term, and "leverage", which is my pet hate word. It's exactly as useful as "use", only much more pretentious. I'm just easily triggered with language :P More off-topic (or, rather, on-topic), I find lambdas great for things like a static website that needs a few functions. I especially like how Netlify uses them, they seem to fit that purpose exactly.
    - normalnorm 7y ago
      > I'm just easily triggered with language :P Me too! It makes me irrationally angry when people regurgitate linguistic clichés. I was already mad with: "python does not have a go-like concurrency story" when it would be enough (and 1000x less cringe) to say: "python does not have go-like concurrency" I think these mindless clichés make language really ugly and dysfunctional, and even worse they are thought-stoppers, because they make the reader/listener feel like something smart is being said, because they recognize the "in-group" lingo. In my experience, people get really offended when you point this out. It's kind of an HN taboo to discuss this. Which is also interesting in itself. Going forward we should pay more attention to our communication use cases. Btw: I wonder if we can stack several of these clichés. For example: "leverage" + "use case" = "leverage case".
      - StavrosK 7y ago
        I agree, I think Orwell's "Politics and the English Language" is spot on here. I try to use simpler language whenever possible, I agree that people think that using longer words makes them sound smart but is just worse for communication. I've found it's a taboo to discuss anything even slightly personal. People are averse to feeling bad, so criticism needs to be extremely subtle in order to not offend. > Btw: I wonder if we can stack several of these clichés. For example: "leverage" + "use case" = "leverage case". I hate you for even thinking of this.
lmeyerov 7y ago
We've been experimenting with nice compromises for using pydata on-the-fly compute with caching. Basic tension is indiv requests get blocked by their blocking APIs, while caches (eg, IP-pinned) are best when same-python-app-thread. Right now, we do hypercorn multiproc -> per-proc quart/asyncio/aiohttp IP-pinned event loop -> Apache arrow in-app cache -> on-gpu rapids.ai cache. But not happy w event loop due to pandas/rapids blocking if heavy datasets by concurrent users. (Taking us back to celery, redis, etc, which we don't want due to extra data movement..) Maybe we can get immutable arrow buffs shared across python proc threads.. Ideas welcome!

BiteCode_dev 7y ago

Besides, serious projects in Go do use additional tools for creating task queues because they need to handle various stages of persistence, error handling, serialization, workflow, etc. Which are not stuff you want to write by hand yourself. If you don't need all that, then it's not a problem in Python either. You don't even need asyncio, you can just use a ThreadPool or a ProcessPool, dump stuff with pickle/shelve/sqlite3 and be on your way.

Carpetsmoker 7y ago

I was using gevent in Python about 10 years ago, and from memory, it's roughly similar to goroutines. It's not exactly the same of course, but just like goroutines it's pretty easy to just spawn a few jobs off, wait for them to finish, and get the results. It's not in the standard library, and there are probably other options too now (not a heavy Python user any more), but Python has had easy parallelism for at least a decade (probably longer).

eloff 7y ago
Gevent is like goroutines with GOMAXPROCS=1. Which is to say not nearly as useful. It gives you concurrency without parrallelism, because Python never did shake the GIL. Which goes to show some technical debt will haunt you forever.
- antoinealb 7y ago
  To nuance your comment, you can still get some form of parallelism, "just not" thread parallelism in Python. You can still spawn multiple handler process, or have threaded code in a C extension.
  - zepto 7y ago
    It’s arguable whether either of those are ‘in python’.
    - quacker 7y ago
      The multiprocessing module in the standard library is absolutely a Python-native way to do parallelism: with Pool(5) as p: print(p.map(f, [1, 2, 3])) This runs f(1), f(2), and f(3) in parallel, using a pool of five processes. https://docs.python.org/3.6/library/multiprocessing.html https://docs.python.org/3.6/library/multiprocessing.html
      - goostavos 7y ago
        Whoa! It never occurred to me that `Pool` could be used with context manager. I've always typed out `.close()` manually like a sucker.

jakearmitage 7y ago

> same developer experience as go You can't say that when Python shoves Pip/Poetry/VirtualEnv/Black/Flake and what not into people. In contrast, Go has built-in package management and gofmt. Python is essentially gate-keeping new devs.

morelisp 7y ago
Go may be unique in being the only ecosystem built after Python that can't claim it avoided Python's packaging disasters.
- mdtusz 7y ago
  Cargo works pretty well too.
  - takeda 7y ago
    Cargo is not from Go
    - jessaustin 7y ago
      As a name for a package manager, "cargo" certainly appears more suitable for "go" than for "rust".
- takeda 7y ago
  I believe Python suffers from no leadership in that space (everyone creates their own packaging, every tutorial advocates something different, many tutorials are outright wrong). There was also a bad decision of using Python code for installation (setup.py) instead of a declarative language. Most of that issues are actually fixed in setuptools if you put all settings in setup.cfg and just call empty setup() in setup.py. Like here: https://github.com/takeda/example_python_project/blob/master/setup.cfg https://github.com/takeda/example_python_project/blob/master...
- throwaway894345 7y ago
  How do you figure? Go's packaging is wayyyy better than Python's. I've done considerable work with each and while Go's ecosystem has warts here and there, it's far from disastrous. I can't say that about Python. If nothing else, Go lets you distribute a static binary with everything built in, including the runtime. Python's closest analog is PEX files, but these don't include the runtime and often require you to have the right `.so` files installed on your system, and they also don't work with libraries that assume they are unpacked to the system packages directory or similar. In general, it also takes much longer to build a PEX file than to compile a Go project. Unfortunately, PEX files aren't even very common in the Python ecosystem.

throwaway894345 7y ago

> Fyi - python ASGI frameworks like fastapi/Starlette are the same developer experience as go. Can you provide some context for this statement? I've used Python asyncio extensively in Fargate (no ASGI frontend), and the developer experience is far from Go; however, I don't see how an ASGI framework can fix this. It seems like it offers the same general course-grained parallelism that you get from a containerized environment like Fargate except that Fargate abstracts over a cluster while ASGI frameworks presumably just abstract over a handful of CPUs. For example, we have a large data structure that we have to load and process for each request. We want to parallelize the processing of that structure, but the costs to pickle it for a multiprocessing approach are much too large. We've considered the memory mapped file approach, but it has its own issues. We're also looking at stateful-server solutions, like Dask, but now we're talking about running infrastructure. In Go, we could just fork a few goroutines and be on our way.

sandGorgon 7y ago
you should ask this exact same question here - https://github.com/tiangolo/fastapi/ https://github.com/tiangolo/fastapi/ i dont claim to have expertise in your business domain, but you should get the answer here.

zaro 7y ago

I've played with the concurrency in Python, and it's simply not worth it. Much better to use Node.js or Go where the async story is not an afterthought. Of course if you are stuck with Python it's better than nothing.

solidasparagus 7y ago

The difference is that it doesn't play well with large parts of the existing Python ecosystem. Concurrency is a pain if you are writing a library or trying to integrate with existing code. Python async is powerful, but: (1) it's undersupported (e.g. you can in theory download s3 files using async botocore, but in practice it is hard to use because of strict botocore version dependencies) (2) it isn't natural - once you're in the event loop it makes sense, but using an event loop alongside normal python code is confusing at best. (3) it got introduced too late. The best async primitives are only available in pretty recent versions of python3. The difference with go is that it go has these primitives built in from the beginning and using them doesn't introduce interoperability problems.

jatone 7y ago
and on top of it go is performant enough that 90% of the time you don't need them to begin with.
- hackworks 7y ago
  Most common use case for concurrency is for IO intensive workloads. When it comes down to IO, programming language hardly matters. If you have CPU intensive workload, an optimizing compiler can help. Well, for that you have C or other languages with more mature and aggressive compilers.