7 ms·
How to use a Python multiprocessing module
- jmakov 3y agoYou use it by installing ray.io
- samsquire 3y agoI use Python's multiprocessing to run A* in parallel with queues communicating between workers and the main thread for code generation. https://replit.com/@Chronological/SlidingPuzzle3 https://replit.com/@Chronological/SlidingPuzzle3 (the code shows multiprocessing Processes connected by queues, the README.md explains what the program does) Some algorithms aren't easily parallelisable (at least I haven't worked out how to paralellise them!). My original attempts at paralellising A* didn't work because of communication overhead. But I sharded my data that it searches - each thread searches a different subset of neighbours as A* is a graph search. This lets me paralellise, with 8 threads (when I boosted the repl.it from 2 vcpu to 8vcpu) It would run in 20-40 milliseconds as opposed to 100-300 milliseconds. mgaunard [0] recently told me that there is something called an algorithmic skeleton for parelllisation problems. I've not used any though. https://en.wikipedia.org/wiki/Algorithmic_skeleton https://en.wikipedia.org/wiki/Algorithmic_skeleton I have no exposure to high performance computing unfortunately. Just a deep interest in parallelism. [0]: https://news.ycombinator.com/item?id=36792796 https://news.ycombinator.com/item?id=36792796
- pvitz 3y agoProgram synthesis through optimal control or directly through algorithms like A* sounds fun and interesting. If I may, I would suggest to either comment your code or add more details to your README.md, because it is currently quite difficult to understand what exactly is being done here and how.
- samsquire 3y agoThanks for your reply and words. It's path finding between output states of a program. I was inspired by path finding in video games. The path is the discovered program. It's like having a game map where each neighbour of a cell is the potential things you can do at that point in time. For this reason it can be used for unit AI. I think of assembly (and regular) programming as a sliding image puzzle game. My design is based on this intuitive idea that most programming is just moving things around - it's logistics. How much time do you spend marshalling data? REST APIs? To get registers to the value for a CALL instruction, the calling convention, is just moving the right data into the right locations. Imagine you have a start state and an end state. It's a turing machine. State is processor register values and memory values in each memory location. There is a series of state transitions between the start state and end state which is created by an instruction, these are intermediary states. Instructions can be MOV instructions or CALL instructions (function calls) I dynamically generate neighbours of starting from the start state which are potential instructions we can run that get us nearer to the end state. The program searches the space of possible instructions. It infers hidden states caused by functions. The distance heuristic is the number of values that in registers and memory that match the end state. My dream: if you knew the type of every API call, you could provide a start state and an end state and the computer would work out the program based on exploring potential instructions that get us to the target state. This is a much simpler form of program synthesis compared to LLMs. I'll update the README and document the code. I also have a dream of a real time strategy game where you program the computer by telling unit what to do. In most RTS games you can only build, attack but we can do more.
- vorticalbox 3y agoWhen I need to process sometimes millions of documents from a production mongoDB instance I normally push all the objectID into rabbitMQ then spin up N workers that each fetch a single message at a time.
- rdedev 3y agoIs there a reason why you use rabbitmq instead of just using the whats available in the language (I understand if those primitives are hard to work with)
- tempay 3y agoI'd guess they're using multiple machines.
- vorticalbox 3y agoAs said people using multiple ec2s to process the data but the biggest reason is so I don't have to run it all in one go. Rabbit is used by a few of our services so is always up, means I can run for a few hours then come back to it another day. Also error handling, if one fails I just don't ack the message, after everything is processed I only have failed ones. Can slowly process the messages to see what is wrong
- samsquire 3y agoThanks for sharing, I think this topic is very interesting. I played about with Kafka consumer groups to get the same effect.
- paulddraper 3y ago> then spin up N workers And how do you do that? Because...that is also what the multiprocessing module does.
- kastden 3y agoCheck out ProcessPoolExecutor [0] (and ThreadPoolExecutor) too for an easy way to spin up a bunch of tasks and wait for them to complete. [0] https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor https://docs.python.org/3/library/concurrent.futures.html#pr...
- gpderetta 3y agoCoincidence: I literally read that doc-page and wrote some ThreadPoolExecutor code 5 minutes ago to workaround the lack of a specific async operation in asyncio.
- evomassiny 3y agoAgreed ! Plus, the ability submit a bunch of tasks, and to block until _one_ task completed (akin to `epoll_wait` of tokio's `select`) makes it quite useful. I don't know of a use case or `mutiprocessing.Pool` which is not covered by `concurrent.futures.ProcessPoolExecutor`; so I wonder why both exist
- hangonhn 3y agoIn general, the Executor or even Queue abstractions are much better and safer ways of doing multithreading and multiprocessing. These days, I rarely ever directly create threads. It's fine for situations when you must but parallelizing work can usually be done better, safer, and easier with executors.
- culebron21 3y agoThis covers just the most basic things that can be looked up in the manual (IDK if it's ChatGPT output or not, doesn't matter). It doesn't answer * why you need multiprocessing (rather than threads) * what are costs and benefits of them * how to process a sequence of tasks maintaining results in order * ...or without maintaining the order * how to handle exceptions in the processes * how to gracefully exit by Ctrl+C or other exceptions in the main process * how to minimize data transmission between processes, because it's costy etc. None of this is covered.
- benterix 3y agoYeah, that's struck me as well. "Let's get down to some work" without explaining why it should (or should not!) be done. And in this case it's crucial.
- nerdponx 3y agoIt's low quality content marketing for Red Hat. Equivalent to all those Medium articles written by boot camp grads, except it's a Red Hat ad instead of a "hire me" ad.
- maegul 3y agoI mean par for the course for tech blogs/articles right.
- benterix 3y agoAgreed, but the bar for HN submissions is slightly higher.
- velosol 3y agoI don't think I can do all of those justice right this moment but let's get a few here so people can link to an HN thread instead of the red hat article ;) MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL). Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs. Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable). Exceptions: Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed. For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals. Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.
- AndrewOMartin 3y agoThis is a decent article, but for me the best discussion on parelleism in Python is Raymond Hettinger's talk "Keynote on Concurrency, PyBay 2017" https://www.youtube.com/watch?v=Bv25Dwe84g0 https://www.youtube.com/watch?v=Bv25Dwe84g0 It includes a very important observation that lots of problems are small enough that they can be solved with a single CPU core, and that lots of problems are so big you need full on Cloud Computing Distributed Processing with effectively unlimited number of cores, and there are relatively few problems that are too big for 1 core but can be made tractable by going up to around 10 cores. (This is the "Martelli Model of Scalability") Also this... Some people, when confronted with a problem, think "I know, I'll use parallelism." problems two they two Now two two two two two....
- kzrdude 3y agoHettinger is a great teacher but I think he's been part of core python's status-quoism, that quote is just in line with that. And that culture is being changed now (faster cpython and maybe even nogil!)
- ospdfhnnioniop 3y ago[dead]
- akasakahakada 3y agojust delete multiprocessing.py and replace it with mpi4py
- benterix 3y agoYou might be right but you haven't provided any arguments. I remember MPI from my C times and it definitely had its quirks.
- ggm 3y agoOne observation I'd like to make is that feeding a thread farm from a single input stream demands some understanding of locks, queues and mutex. The examples are a bit piss weak/simple. One master and one slave. Really I expected 3-4 consumers so you can understand how they interact over q.put() and q.get() and q.join() and q.task_done() On a large(ish) memory and core host using dnspython I've been getting significantly more dns throughout over log file processing and a dict() cache shared amongst the threads avoiding dns stalls. 20+ threads. It's still buggy, better examples would be helpful. I know: threads not processes. They're documented as neighbour concepts. I'd have done process forking if I could find good lock and shared mem/mmap use Documented.
- itamarst 3y agoReminder that on Linux multiprocessing is broken out of the box, and you need to configure it to not freeze your process at random: https://pythonspeed.com/articles/python-multiprocessing/ https://pythonspeed.com/articles/python-multiprocessing/ (This will be fixed in future Python versions, 3.14 maybe.)
- ptx 3y agoTLDR: If you fork the process after starting threads you're gonna have a bad time, so use spawn instead of fork, like this: from multiprocessing import set_start_method set_start_method("spawn")
- deleted 3y ago[deleted]
- paulddraper 3y agoIf I understand correctly, this is about combining multiprocessing and multithreading?
- tj_astro 3y agoI've found https://superfastpython.com/learning-paths/#Multiprocessing_Learning_Path https://superfastpython.com/learning-paths/#Multiprocessing_... to be an invaluable resource for advanced multiprocessing.
- amelius 3y agoAnyone else run into problems with Multiprocessing + Numpy? I can somehow only reliably do multiprocessing with numpy when I set: export OPENBLAS_NUM_THREADS=1 Of course, this is a workaround and not a true solution.
- nerdponx 3y agoThis is probably a known bad interaction and might have to do with spawn vs fork as mentioned elsewhere in this thread.
- amelius 3y agoYes, I can remember that I fiddled with spawn versus fork, but still ended with the OPENBLAS_NUM_THREADS=1 solution.
- czbond 3y agoDask can also help with Numpy sets.
- amelius 3y agoWell, it's complicated as I'm not doing multiprocessing with Numpy, actually :) One part of my program uses Numpy, and the other part does multiprocessing. But Python's multiprocessing module causes interference between them (which took quite some time and effort to figure out).
- phkx 3y agoThe models for curve fitting in my lab were using lambdas, which cannot be serialized by multiprocessing/pickle (at least that was the state about 5 years ago). The pathos.multiprocessing module [1] served as a drop-in replacement, which was able to handle lambda with its own dill serializer. Saved me quite a bit of refactoring. [1] https://github.com/uqfoundation/pathos https://github.com/uqfoundation/pathos
- tedivm 3y agoI built a wrapper library called QuasiQueue around the multiprocess modules that makes it super easy to use a multiprocess queue to process data. https://github.com/tedivm/quasiqueue https://github.com/tedivm/quasiqueue I've built a few multiprocess programs with it and it's been really useful, so I thought I'd share.
- koromak 3y agoDoes anyone have a good overview of threading on AWS? Every time I read about it I get confused, it seems like there are wildly different rules depending on the context. Lambda is different than ECS which is different from EC2
- yazantapuz 3y agoI have several Python scripts that run simulations of cpu schedulers. I have used this module for a while, but resorted to just use gnu parallel.
- gjvc 3y agocareful you don't ever accidentally try to pass a large result (for example, a pandas dataframe...) back from the workers to the controller process :-) The communication happens via pickle and the memory usage is huuuge. If you can, think map/reduce, and return the smallest-sized answer.
- efxhoy 3y agoI needed to pass big dataframes between workers and the main thread. Turned out that the easiest way for me was having them write parquet files and reading them in the main thread.
- dec0dedab0de 3y agoI like using multiprocessing pools whenever I need to plow through something quickly, or if I don't already have a task queue like celery set up. However, this article feels off to me. Everything I read from redhat always feels a bit wrong for some reason. I don't know what it is exactly. It might just be that they used "a" instead of "the" in the title. But Ive had the same feeling with other manuals and whatnot from redhat.
- barbazoo 3y agoThe official documentation is great too: https://docs.python.org/3/library/multiprocessing.html https://docs.python.org/3/library/multiprocessing.html
- crabbone 3y agoJust use Erlang or Go. Even Java... Python doesn't hold a candle to any of those languages when it comes to multitasking. The most important advise anyone could give on using Python's anything, especially stuff that has to do with concurrency is to stock up on mood enhancing drugs, else they'll probably cry themselves to death out of despair. I don't understand why a company like RHEL would publish such a low-effort article. But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language... well, never use plain multiprocessing.Process. Use multirpocessing.pool.Pool: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool https://docs.python.org/3/library/multiprocessing.html#multi... And if you wonder why would you do that... well, obviously the later builds on the former. And, trust me, it doesn't improve the former in any way. But, it saves you some work. Virtually any task you will end up doing will be done with process pool in less code. Most importantly though, it's a "recognizable pattern" -- it's something that someone who ate from the garbage pile known as Python has learned to recognize in code and to treat accordingly. Code randomly peppered with mention of Process or, worse yet, extending that class is a sign of lack of experience, and produces an urge to delete that nonsense code and write it afresh because nobody's got the time to deal with novice's nonsense.
- ke88y 3y ago> Just use Erlang or Go. Even Java... Python doesn't hold a candle to any of those languages when it comes to multitasking. 1000%. Pretty much any other language. > But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language Sometimes the drag of dealing with Python multiprocessing is easier and incurs less debt than splitting the project across multiple languages and dealing with (de)serialization of complicated data structures in way that doesn't introduce bottlenecks. And there are at least a few domains where there isn't really a viable alternative to Python. But a good rule of thumb is that if you're not going to spend at least a couple full days dealing with the issues listed above -- or if you don't absolutely need to use a big chunk of complicated Python code -- you're probably better off picking a different language.