8 ms·

I love Python so much. It's my favourite language. It's so easy. I don't even use the dynamic nature of Python - monkey patching etc. I wish more people wrote

I love Python so much. It's my favourite language. It's so easy. I don't even use the dynamic nature of Python - monkey patching etc. I wish more people wrote algorithms in Python for understandability then compiled them into a compiled language for performance. I find that Python resembles pseudocode because it is syntactically sparse, unlike C++. In Python I don't need to worry about ownership or memory. It probably needs Pointers to be a low level system language. I just wish Python had parallelism.

divbzero 5y ago
By parallelism do you mean something different from multiprocessing? [1] [1]: https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html
- triceratops 5y ago
  Pretty sure they're referring to the GIL.[1] 1. https://realpython.com/python-gil/ https://realpython.com/python-gil/
- samsquire 5y ago
  I want Thread to run in parallel. I've tried multiprocessing but I found it buggy for my use case. I tried to create a worker queue and submit items to worker threads with JoinableQueue but eventually the system deadlocks and no progress is made. I'm not sure what the problem is. It never finishes. It's a sentence correlator using multiprocessing - it generates correlations of words in your sentences. https://github.com/samsquire/notebook/blob/master/sentence-correlator.py https://github.com/samsquire/notebook/blob/master/sentence-c... https://github.com/samsquire/notebook/blob/master/workers.py https://github.com/samsquire/notebook/blob/master/workers.py Maybe somebody can catch what I'm doing wrong. edit: it may be because i've forgotten to run task_done on joinable queue. edit2: yes it was that! it works!
  - aw1621107 5y ago
    Out of curiosity, is using multiprocessing.Pool’s map/etc.() methods not an option? It’s a higher-level API which would remove the need to manually manage workers/a queue.
- dekhn 5y ago
  yes, for many of us, we really do want a CPython that runs multiple threads of the same interpreter, with memory sharing. That said after decades of CPython not supporting it, I've rewritten most of my code to use multiprocessing with granular parallelism.
quietbritishjim 5y ago
(Standard reponse) Python does support parallelism, so long as you're calling into a C library to do the work, because they (usually) release the GIL while doing the work. That sounds like a cop out but using C libraries is very common to do CPU intensive work anyway. Examples include numpy (e.g. you can call np.dot from multiple threads in Python and they will genuinely use multiple cores) and all the rest of the scipy stack, and even modules in the standard library (e.g. when using zipfile, even to process an in-memory buffer, the GIL is released).
- amelius 5y ago
  Not really. You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. Unless of course you translate all of your data to C too, at which point you are really doing everything in C (data and code).
  - incrudible 5y ago
    > You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. You probably do not want to do that in the first place. The reason that the GIL still exists is that the overhead of fine-grained locking for Python datastructures would negate the speedup from multithreading.
    - amelius 5y ago
      Which is why Python doesn't really support parallelism. Other language (like GoLang) do support parallelism but they have a more advanced runtime and don't use a GIL.
      - incrudible 5y ago
        It's not specific to Python. If you want to do fine-grained "parallelism" on a set of complex shared datastructures, your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. That's not necessarily going to result in a speedup over a lock-free single-threaded implementation (that you didn't implement to compare against). In the case of Python, the single-threaded implementation was already there, the implementation that removed the GIL in favor of fine-grained locking failed to deliver the goods. If you want to actually take advantage of parallelism, you want data structures that are amenable to it, which is basically sections of arrays without a lot of potential read/write conflicts over multiple threads. If you're at that point, you probably want to use C/C++/Rust anyway, you don't want speed up something that's already 100x slower (the interpreter) by parallelizing it. Python offers that with C-extensions like NumPy.
adsharma 5y ago
Please try py2many, which targets this use case. Go, rust, C++ are among the 7 supported backends. http://github.com/adsharma/py2many http://github.com/adsharma/py2many
pjmlp 5y ago
You can do low level coding via ctypes, and on Micro/CircuitPython you can even mess with Assembly, just like on some old BASIC dialects.
pansa2 5y ago
> I just wish Python had parallelism. Would support for parallelism have benefits other than increasing speed? If not, I'd rather see an increase in Python's single-threaded performance - which is what Guido and his colleagues at Microsoft are currently working on.
- samsquire 5y ago
  The problem with interpreting the Python way, using a switch loop ceval.c is that there is plenty of overhead with that loop. Branch prediction fails. It could be every opcode is compared before reaching the actual opcode in the switch. I wonder if Python will ever take the JVM approach of having a TemplateInterpreter and mapping each bytecode to assembly directly. Of course you would have to do it for each platform. http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview.html#Interpreter|outline http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview....
lmm 5y ago
I used to take your view, but actually there are plenty of languages (e.g. ML-family languages) where you get the best of both worlds - light, clean syntax, but also type safety and decent performance.
- tenacious_tuna 5y ago
  Can you share some examples? I'm very interested in a typesafe alternative to python, but low friction is also necessary.
  - adsharma 5y ago
    Why do you think python3 is not typesafe?
    - packetlost 5y ago
      Because it's not? It has a type system but it's still duck-typed at the end of the day, and no amount of Mypy will change that.
      - adsharma 5y ago
        It should still be possible to use the python syntax and a reimplemented stdlib to build a statically typed, typesafe variant. I think you object to the name python3, because there is a duck typed interpreter that's popular.
        packetlost 5y ago
        You fundamentally cannot build a statically typed variant of Python without cutting portions of the stdlib, most notably `mock.patch`. You could do a good majority of it, probably enough to be completely usable, but at what point does it stop being Python? Go use Nim if you want statically typed Python-ish syntax.
        5y ago
O_H_E 5y ago
That is Julia for you: https://julialang.org/blog/2012/02/why-we-created-julia/ https://julialang.org/blog/2012/02/why-we-created-julia/ \s? :D I understand that you (and I for that matter) can't stop using python altogether rn. Just wanted to point out that these two desirable qualities where build into Julia from the get go.
- kortex 5y ago
  Julia just isn't a replacement for Python yet. Recently did a deep dive researching various capabilities that would make Julia at least useful as a microservice/RPC target for a more I/O heavy language like Python. gRPC support is really poor. There's some OpenAPI/Swagger tooling, but it's not great. Likewise for ZeroRPC. The best candidate is probably the WebApi library [0] but even then, it does not inspire a ton of confidence for production. From cruising github issues and the Julia forums, nothing in this space (rpc/io/microservice) feels really bulletproof. Combined with no solid native executable support, it's just a nonstarter for me. My gestalt sensation is the community is still too small, the tooling too unpolished, to be ready for anything production-worthy. If there were a bulletproof 0rpc library, I think it would go a huge way towards growing the community and mindshare through successive approximation. But the other problem is many of the folks on the forum seem totally disinterested in this problem. [0] https://github.com/JuliaWeb/JuliaWebAPI.jl https://github.com/JuliaWeb/JuliaWebAPI.jl

divbzero 5y ago

By parallelism do you mean something different from multiprocessing? [1] [1]: https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html

triceratops 5y ago
Pretty sure they're referring to the GIL.[1] 1. https://realpython.com/python-gil/ https://realpython.com/python-gil/
samsquire 5y ago
I want Thread to run in parallel. I've tried multiprocessing but I found it buggy for my use case. I tried to create a worker queue and submit items to worker threads with JoinableQueue but eventually the system deadlocks and no progress is made. I'm not sure what the problem is. It never finishes. It's a sentence correlator using multiprocessing - it generates correlations of words in your sentences. https://github.com/samsquire/notebook/blob/master/sentence-correlator.py https://github.com/samsquire/notebook/blob/master/sentence-c... https://github.com/samsquire/notebook/blob/master/workers.py https://github.com/samsquire/notebook/blob/master/workers.py Maybe somebody can catch what I'm doing wrong. edit: it may be because i've forgotten to run task_done on joinable queue. edit2: yes it was that! it works!
- aw1621107 5y ago
  Out of curiosity, is using multiprocessing.Pool’s map/etc.() methods not an option? It’s a higher-level API which would remove the need to manually manage workers/a queue.
dekhn 5y ago
yes, for many of us, we really do want a CPython that runs multiple threads of the same interpreter, with memory sharing. That said after decades of CPython not supporting it, I've rewritten most of my code to use multiprocessing with granular parallelism.

quietbritishjim 5y ago

(Standard reponse) Python does support parallelism, so long as you're calling into a C library to do the work, because they (usually) release the GIL while doing the work. That sounds like a cop out but using C libraries is very common to do CPU intensive work anyway. Examples include numpy (e.g. you can call np.dot from multiple threads in Python and they will genuinely use multiple cores) and all the rest of the scipy stack, and even modules in the standard library (e.g. when using zipfile, even to process an in-memory buffer, the GIL is released).

amelius 5y ago
Not really. You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. Unless of course you translate all of your data to C too, at which point you are really doing everything in C (data and code).
- incrudible 5y ago
  > You can't build large data structures (not arrays, but trees, graphs, etc) with structural sharing between threads. You probably do not want to do that in the first place. The reason that the GIL still exists is that the overhead of fine-grained locking for Python datastructures would negate the speedup from multithreading.
  - amelius 5y ago
    Which is why Python doesn't really support parallelism. Other language (like GoLang) do support parallelism but they have a more advanced runtime and don't use a GIL.
    - incrudible 5y ago
      It's not specific to Python. If you want to do fine-grained "parallelism" on a set of complex shared datastructures, your cores might look busy, but really you're just wasting most of the time locking/unlocking stuff. That's not necessarily going to result in a speedup over a lock-free single-threaded implementation (that you didn't implement to compare against). In the case of Python, the single-threaded implementation was already there, the implementation that removed the GIL in favor of fine-grained locking failed to deliver the goods. If you want to actually take advantage of parallelism, you want data structures that are amenable to it, which is basically sections of arrays without a lot of potential read/write conflicts over multiple threads. If you're at that point, you probably want to use C/C++/Rust anyway, you don't want speed up something that's already 100x slower (the interpreter) by parallelizing it. Python offers that with C-extensions like NumPy.

adsharma 5y ago

Please try py2many, which targets this use case. Go, rust, C++ are among the 7 supported backends. http://github.com/adsharma/py2many http://github.com/adsharma/py2many

pjmlp 5y ago

You can do low level coding via ctypes, and on Micro/CircuitPython you can even mess with Assembly, just like on some old BASIC dialects.

pansa2 5y ago

> I just wish Python had parallelism. Would support for parallelism have benefits other than increasing speed? If not, I'd rather see an increase in Python's single-threaded performance - which is what Guido and his colleagues at Microsoft are currently working on.

samsquire 5y ago
The problem with interpreting the Python way, using a switch loop ceval.c is that there is plenty of overhead with that loop. Branch prediction fails. It could be every opcode is compared before reaching the actual opcode in the switch. I wonder if Python will ever take the JVM approach of having a TemplateInterpreter and mapping each bytecode to assembly directly. Of course you would have to do it for each platform. http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview.html#Interpreter|outline http://openjdk.java.net/groups/hotspot/docs/RuntimeOverview....

lmm 5y ago

I used to take your view, but actually there are plenty of languages (e.g. ML-family languages) where you get the best of both worlds - light, clean syntax, but also type safety and decent performance.

tenacious_tuna 5y ago
Can you share some examples? I'm very interested in a typesafe alternative to python, but low friction is also necessary.
- adsharma 5y ago
  Why do you think python3 is not typesafe?
  - packetlost 5y ago
    Because it's not? It has a type system but it's still duck-typed at the end of the day, and no amount of Mypy will change that.
    - adsharma 5y ago
      It should still be possible to use the python syntax and a reimplemented stdlib to build a statically typed, typesafe variant. I think you object to the name python3, because there is a duck typed interpreter that's popular.
      - packetlost 5y ago
        You fundamentally cannot build a statically typed variant of Python without cutting portions of the stdlib, most notably `mock.patch`. You could do a good majority of it, probably enough to be completely usable, but at what point does it stop being Python? Go use Nim if you want statically typed Python-ish syntax.
        5y ago

O_H_E 5y ago

That is Julia for you: https://julialang.org/blog/2012/02/why-we-created-julia/ https://julialang.org/blog/2012/02/why-we-created-julia/ \s? :D I understand that you (and I for that matter) can't stop using python altogether rn. Just wanted to point out that these two desirable qualities where build into Julia from the get go.

kortex 5y ago
Julia just isn't a replacement for Python yet. Recently did a deep dive researching various capabilities that would make Julia at least useful as a microservice/RPC target for a more I/O heavy language like Python. gRPC support is really poor. There's some OpenAPI/Swagger tooling, but it's not great. Likewise for ZeroRPC. The best candidate is probably the WebApi library [0] but even then, it does not inspire a ton of confidence for production. From cruising github issues and the Julia forums, nothing in this space (rpc/io/microservice) feels really bulletproof. Combined with no solid native executable support, it's just a nonstarter for me. My gestalt sensation is the community is still too small, the tooling too unpolished, to be ready for anything production-worthy. If there were a bulletproof 0rpc library, I think it would go a huge way towards growing the community and mindshare through successive approximation. But the other problem is many of the folks on the forum seem totally disinterested in this problem. [0] https://github.com/JuliaWeb/JuliaWebAPI.jl https://github.com/JuliaWeb/JuliaWebAPI.jl