4 ms·
Gotten burned by that. Have a python utility that calls something that's very CPU intensive (GDAL) and I literally have to wrap this python process in java for
by pookha 3y ago
Gotten burned by that. Have a python utility that calls something that's very CPU intensive (GDAL) and I literally have to wrap this python process in java for real concurrency. Not a python expert by any means but I'd love to know what other people do when this happens. I'm assuming it's something revolving around containers and messaging (blech)...
- ska 3y agoIs your problem that the c part of GDAL isn't concurrent and you are calling it multiple times, or are you blocking on something you shouldn't? Python calling a concurrent c thing is super common. For more complex things there is multiprocessing, but that essentially is simplification of IPC.
- pookha 3y agoHad a massive amount of images that I was trying to process for a specific purpose. I wound up spinning up multiple python processes within a java wrapper using concurrent linked queues. Got the job done but it's not the proudest moment of my life.
- seunosewa 3y agoWhy did you need the Java wrapper? You can orchestrate stuff like that within Python. The multiprocessing module.
- giantrobot 3y agoYou could have done this with the multiprocessing module. It interfaces like the threading library but it's launching new processes for each "thread". I've done a shitload of image processing in Python with it. You'll end up with a 100% Python version of what you did.
- ska 3y agoMultiprocessing does basically the same thing, but in python.
- antod 3y agoI thought Python calling a native lib was the one time you could multithread your way around the GIL if you were cpu limited? Are the GDAL bindings not releasing the lock or something?
- totony 3y agoPython works around this by having multithreading through multiprocessing and IPC. its made pretty easy to use with the futures builtin module but is finicky/slow/hard to support.