36 ms·
I have seen and written Python code that spawns various threads with shared mutable state. Is it possible that some day the same code would run in parallel? Tha
by lvass 5y ago
I have seen and written Python code that spawns various threads with shared mutable state. Is it possible that some day the same code would run in parallel? That could be a terrible (very) breaking change. I'm not against allowing in-process parallel execution but please let it require a new API.
- juanbyrge 5y agoTranslation: I have written buggy, racy software that has specific dependencies on thread timing. Please do not make significant improvements to Python because it will reveal these bugs in my software, and I will be forced to fix the bugs and use proper synchronization.
- lvass 5y agoIs that how you call things that have been working flawlessly and solving people's problems for over 10 years? Is needlessly breaking things that work an improvement to you?
- zelphirkalt 5y agoYou could probably simply lock the Python version you use for such code. No breakage there. If you must upgrade to a newer Python version, then you will have to repair broken code.
- doubled112 5y agoThis went really well for the Python 2 -> 3 upgrade
- lazide 5y agoIt did buy a decade or so (or more really) - not like the python2 distribution you downloaded and distribute with your program back then is going to get tracked down and shot in the head by Guido anytime soon. If you’re relying on whatever python version is distributed with whatever machine it happens to be on, there are a huge number of problems you’re already going to have.
- deleted 5y ago[deleted]
- lazide 5y agoIf you make something that works because of an explicit memory and concurrency model (and not like there are other options at the time), it is indeed legit to worry about a major shift to those models that would cause problems. Even if those changes are better for other ways of solving problems.
- deleted 5y ago[deleted]
- ajkjk 5y agoIsn't that.. Already the case? The gil doesn't prevent thread switching in the middle of python code.
- lvass 5y agoIt's concurrent not parallel. The switch won't happen inside the execution of one opcode including some dictionary update operations so it's safe in many cases where parallel execution isn't.
- zelphirkalt 5y agoIt is probably a bad practice to not acquire a mutex for that concurrent dictionary update. The code should be improved in that regard, with or without any potential Python language change.
- lvass 5y agoIf needed, I'll probably just change it to asyncio, probably saner than inspecting everything for new parallelism bugs which can be incredibly subtle.
- ajkjk 5y agoIf performance isn't hugely important you could make blanket-locking wrappers around common data structures and swap them in-place for all of your global state. .. but, as I said, removing the GIL will almost certainly be opt-in.
- ajkjk 5y agoYes, single-instruction operations would be fine, but if you're writing multithreaded code you are probably doing things that the GIL doesn't protect all the time. Like dict-updates on classes that implement __set__, or `if not a[x]: a[x] = y` sorts of two-phased checks, or just like, anything else. You can't get very far with global state without reckoning with concurrency, GIL or not. I assume that a change to relax the GIL will both allow you to opt-out of it, and allow you to use locking versions of primitive data-structures, anyway; it's not like it's going to just vanish overnight with no guardrails.
- pkulak 5y agoThat was my first thought as well. I use Python to whip up quick scripts and enjoy not having to worry about shared memory, even when I'm using concurrency. I'd hate to lose that.