3 ms·
careful you don't ever accidentally try to pass a large result (for example, a pandas dataframe...) back from the workers to the controller process :-) The com
by gjvc 3y ago
careful you don't ever accidentally try to pass a large result (for example, a pandas dataframe...) back from the workers to the controller process :-) The communication happens via pickle and the memory usage is huuuge. If you can, think map/reduce, and return the smallest-sized answer.
- efxhoy 3y agoI needed to pass big dataframes between workers and the main thread. Turned out that the easiest way for me was having them write parquet files and reading them in the main thread.