3 ms·

Really? That sounds like an aweful lot. When I execute this: python3 -c 'import time; time.sleep(60)' And then pmap the process id of that process, I get

• lazide 5y ago
  Also generally no one spins up distinct, new processes for the ‘co-ordinated distinct process work queue’ when they can just fork(), which should be way faster and pretty much every platform uses copy-on-write for this, so also has minimal memory overhead (at least initially)
  • jesboat 5y ago
    The problem is (perhaps amusingly) with refcounting. As the processes run, they'll each be doing refcount operations on the same module/class/function/etc objects which causes the memory to be unshared.
    • lazide 5y ago
      Only where there is memory churn. If you’re in a tight processing loop (checksumming a file? Reading data in and computing something from it?) then the majority of objects are never referenced or dereferenced from the baseline. Also, since the copy on write generally is memory page by memory page, even if you were doing a lot of that, if most of those ref counts are in a small number of pages, it’s not likely to really change much. It would be good to get real numbers here of course. I couldn’t find anyone obviously complaining about obvious issues with it in Python after a cursory search though.
    • byroot 5y ago
      That was solved years ago by moving the refcounts into different pages: https://instagram-engineering.com/copy-on-write-friendly-python-garbage-collection-ad6ed5233ddf https://instagram-engineering.com/copy-on-write-friendly-pyt...
• deleted 5y ago
  [deleted]