4 ms·

My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is

by btilly 3y ago

My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem. Also, no matter how much you wish it otherwise, retrofitting concurrency on an existing project guarantees that you'll wind up with subtle concurrency bugs. You might not encounter them often, and they're hard to spot, but they'll be there. Furthermore existing libraries that expect to be single-threaded are now a potential source of concurrency bugs. And there is no particular reason to expect the authors of said libraries to have either the skills or interest to track those bugs down and fix them. Nor do I expect that multi-threaded enthusiasts who use those libraries in unwise ways will recognize the potential problems. Particularly not in a dynamic language like Python that doesn't have great tool support for tracking down such bugs in an automated way. As a result if "no GIL" ever gets merged, I expect that the whole Python ecosystem will get much worse as well. But that's no skin off of my back - I've learned plenty of languages. I can simply learn one that hasn't (yet) made this category of mistake.

coldtea 3y ago
>My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem. It's 2023 already. 1988 called.
- btilly 3y ago
  Did you know that Coverity actively REMOVED checks for concurrency bugs? It turns out that when the programmer doesn't understand what the tool says, managers believe the programmer and throw out the tool. Coverity was finding itself in situations where they were finding real bugs, and being punished for it by losing the sale. So they removed the checks for those bugs. I'll revisit my opinion of multithreaded code when things like that stop happening. In the meantime there are models of how to run code on multiple threads that work well enough with different primitives. See Erlang, Go, and Rust for three of them. Also, if you squint sideways, microservices. (Though most people set up microservices in a way that makes debugging problematic. Topic for another day.)
  - biorach 3y ago
    > Did you know that Coverity actively REMOVED checks for concurrency bugs? Source?
    - btilly 3y ago
      https://cseweb.ucsd.edu/~dstefan/cse227-spring20/papers/bessey:a-few.pdf https://cseweb.ucsd.edu/~dstefan/cse227-spring20/papers/bess...
      - yjftsjthsd-h 3y ago
        Specifically, on the last page of that: > As an example, for many years we gave up on checkers that flagged concurrency errors; while finding such errors was not too difficult, explaining them to many users was. (And thanks; I was also wondering about that)
- yjftsjthsd-h 3y ago
  This is very witty and all, but what does it mean?
  - DonHopkins 3y ago
    I think he was subtly Rick Rolling you. https://en.wikipedia.org/wiki/Never_Gonna_Give_You_Up https://en.wikipedia.org/wiki/Never_Gonna_Give_You_Up On 12 March 1988, "Never Gonna Give You Up" reached number one in the American Billboard Hot 100 chart after having been played by resident DJ, Larry Levan, at the Paradise Garage in 1987. The single topped the charts in 25 countries worldwide.
    - coldtea 3y ago
      Actually it just meant that this is a tired old argument when C/C++ programmers were new to multithreaded code. Now we have languages and language facilities (consider Rust, Haskell, and others) to make it much safer. Same with green threads and what Go and now Java does.
      - btilly 3y ago
        And yet, the actual proposals for how to do multithreading in Python are essentially the same as the old C/C++ approaches. And since that hasn't changed, there is no reason to expect better results.
samsquire 3y ago
My deep interest is multithreaded code. For a software engineer working on business software, I'm not sure if they should be spending too much time debugging multithreaded bugs because they are operating at the wrong level of abstraction from my perspective for business operations. I'm looking for an approach to writing concurrent code with parallelism that is elegant and easy to understand and hard to introduce bugs. This requires alternative programming approaches and in my perspective, alternative notations. One such design uses monotonic state machines which can only move in one direction. I've designed a syntax and written a parser and very toy runtime for the notation. https://github.com/samsquire/ideas5#56-stateful-circle-programs-on-rails https://github.com/samsquire/ideas5#56-stateful-circle-progr... https://github.com/samsquire/ideas4#558-state-machine-formulation https://github.com/samsquire/ideas4#558-state-machine-formul... The idea is inspired by LMAX Disruptor and queuing systems.
- btilly 3y ago
  And your approach can be built into a system that does multi-threading away from Python, thereby achieving parallelism without requiring that Python supports it as well. That's basically what all machine learning code written in Python does. It calls out to libraries that can themselves parallelize, use the GPU, etc. And then gets the answer back. You get parallelism without any special Python support.
- zo1 3y ago
  Just to add a bit of my opinion after reading your comment in the context of this thread and not to the merit of your idea. You are precisely the type of person I'd keep very very far away from multithreading in any business software project and also why I advocate the GIL to stay. If you want to do that, go solo in your own time, or try apply for a research position in some giant tech Co.
iskander 3y ago
>My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem. Out of curiosity, have you done any Rust programming and used Rayon? It's hard to convey how easy and impactful multi-threading can be if properly enclosed in a safe abstraction.
- kortex 3y ago
  Python is dynamic AF and Rust's whole shtick is compile-time safety. Python was built from the ground up to be dynamic and "easy", Rust was meticulously designed to be strict and use types to enforce constraints. It's hard to convey how difficult it would be to retrofit python to be able to truly "enclose multithreading in a safe abstraction".
- btilly 3y ago
  I have only read about and played a tiny bit with Rust. But as I noted at https://news.ycombinator.com/item?id=36342081 https://news.ycombinator.com/item?id=36342081, I see it as fundamentally different than the way people want to add multithreading to Python. People want to lock code in Python. But Rust locks data with its compile-time checked ownership model. See https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.h... for more.
hklzr100 3y ago
This is a good point. With tsan and helgrind multithreading becomes manageable in C/C++, nothing like that exists (or will exist) for Python.

by btilly 3y ago

coldtea 3y ago

>My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem. It's 2023 already. 1988 called.

btilly 3y ago
Did you know that Coverity actively REMOVED checks for concurrency bugs? It turns out that when the programmer doesn't understand what the tool says, managers believe the programmer and throw out the tool. Coverity was finding itself in situations where they were finding real bugs, and being punished for it by losing the sale. So they removed the checks for those bugs. I'll revisit my opinion of multithreaded code when things like that stop happening. In the meantime there are models of how to run code on multiple threads that work well enough with different primitives. See Erlang, Go, and Rust for three of them. Also, if you squint sideways, microservices. (Though most people set up microservices in a way that makes debugging problematic. Topic for another day.)
- biorach 3y ago
  > Did you know that Coverity actively REMOVED checks for concurrency bugs? Source?
  - btilly 3y ago
    https://cseweb.ucsd.edu/~dstefan/cse227-spring20/papers/bessey:a-few.pdf https://cseweb.ucsd.edu/~dstefan/cse227-spring20/papers/bess...
    - yjftsjthsd-h 3y ago
      Specifically, on the last page of that: > As an example, for many years we gave up on checkers that flagged concurrency errors; while finding such errors was not too difficult, explaining them to many users was. (And thanks; I was also wondering about that)
yjftsjthsd-h 3y ago
This is very witty and all, but what does it mean?
- DonHopkins 3y ago
  I think he was subtly Rick Rolling you. https://en.wikipedia.org/wiki/Never_Gonna_Give_You_Up https://en.wikipedia.org/wiki/Never_Gonna_Give_You_Up On 12 March 1988, "Never Gonna Give You Up" reached number one in the American Billboard Hot 100 chart after having been played by resident DJ, Larry Levan, at the Paradise Garage in 1987. The single topped the charts in 25 countries worldwide.
  - coldtea 3y ago
    Actually it just meant that this is a tired old argument when C/C++ programmers were new to multithreaded code. Now we have languages and language facilities (consider Rust, Haskell, and others) to make it much safer. Same with green threads and what Go and now Java does.
    - btilly 3y ago
      And yet, the actual proposals for how to do multithreading in Python are essentially the same as the old C/C++ approaches. And since that hasn't changed, there is no reason to expect better results.

samsquire 3y ago

My deep interest is multithreaded code. For a software engineer working on business software, I'm not sure if they should be spending too much time debugging multithreaded bugs because they are operating at the wrong level of abstraction from my perspective for business operations. I'm looking for an approach to writing concurrent code with parallelism that is elegant and easy to understand and hard to introduce bugs. This requires alternative programming approaches and in my perspective, alternative notations. One such design uses monotonic state machines which can only move in one direction. I've designed a syntax and written a parser and very toy runtime for the notation. https://github.com/samsquire/ideas5#56-stateful-circle-programs-on-rails https://github.com/samsquire/ideas5#56-stateful-circle-progr... https://github.com/samsquire/ideas4#558-state-machine-formulation https://github.com/samsquire/ideas4#558-state-machine-formul... The idea is inspired by LMAX Disruptor and queuing systems.

btilly 3y ago
And your approach can be built into a system that does multi-threading away from Python, thereby achieving parallelism without requiring that Python supports it as well. That's basically what all machine learning code written in Python does. It calls out to libraries that can themselves parallelize, use the GPU, etc. And then gets the answer back. You get parallelism without any special Python support.
zo1 3y ago
Just to add a bit of my opinion after reading your comment in the context of this thread and not to the merit of your idea. You are precisely the type of person I'd keep very very far away from multithreading in any business software project and also why I advocate the GIL to stay. If you want to do that, go solo in your own time, or try apply for a research position in some giant tech Co.

iskander 3y ago

>My point of view is that anyone who wants to write multithreaded code, shouldn't be trusted to. Making it easier for people to justify this kind of footgun is a problem. Out of curiosity, have you done any Rust programming and used Rayon? It's hard to convey how easy and impactful multi-threading can be if properly enclosed in a safe abstraction.

kortex 3y ago
Python is dynamic AF and Rust's whole shtick is compile-time safety. Python was built from the ground up to be dynamic and "easy", Rust was meticulously designed to be strict and use types to enforce constraints. It's hard to convey how difficult it would be to retrofit python to be able to truly "enclose multithreading in a safe abstraction".
btilly 3y ago
I have only read about and played a tiny bit with Rust. But as I noted at https://news.ycombinator.com/item?id=36342081 https://news.ycombinator.com/item?id=36342081, I see it as fundamentally different than the way people want to add multithreading to Python. People want to lock code in Python. But Rust locks data with its compile-time checked ownership model. See https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html https://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.h... for more.

hklzr100 3y ago

This is a good point. With tsan and helgrind multithreading becomes manageable in C/C++, nothing like that exists (or will exist) for Python.