3 ms·
> a python process for a typical app can easily be 2GB without very careful memory consideration. Sorry, this is not true. I have a Python app that uses 2 thre
by hermitdev 7y ago
> a python process for a typical app can easily be 2GB without very careful memory consideration.
Sorry, this is not true. I have a Python app that uses 2 threads. It's memory doesnt even exceed 200 MB.
Yes, Python threads generally suck because of the GIL, but they dont cause memory bloat like what you're describing.
For context, I use Python extensively for extract/transform/load (ETL) work. I deal with quite large files. My loaders all run in mear linear time relative to the number of records in the file, and none use more than 300MB RAM. This is against Python 3.7.3.
If your multithreaded Python app is using 2 GB of RAM, it's not because of the threads. Best look elsewhere. Maybe you're caching something large in thread local storage?
- mplanchard 7y agoGP was talking about processes explicitly in comparison to threads, so multiprocessing not multithreading. I believe this was related to their previous comment about the GIL making true multithreaded performance impossible: > - The Global Interpreter lock - you can't avoid forking processes in order to handle concurrent python code. > - Threads are cheaper than processes. A default java thread carries 2MB overhead, a python process for a typical app can easily be 2GB without very careful memory consideration. And it is indeed true that forking can be much more memory intensive than threading.