6 ms·

A library to assist writing memory-unsafe code in "pure" Python

54 points by albertzeyer 2y ago

Galanwe 2y ago
There are some real world use cases for these. I do use some similar helpers to e.g. create pickable weakrefs for passing large objects to forking multiprocessing pools as arguments, or expose large objects in shared memories. It's not portable against non-CPython implementations, but as long as it's well documented and known, it's fine.
- dabacaba 2y ago
  Sharing Python objects between processes is not safe. Python assumes that access to objects is protected by GIL, but this is not the case when using multiple processes (even if the processes only read from the objects, they still modify the refcounts). Sharing memory buffers can be done safely using mmap.
  - Galanwe 2y ago
    Sharing objects is completely fine upon a forking multiprocessing pool. Each worker process ends up with a (lazy) copy of the parent address space, including the whole python interpreter state. When you think about it, you already rely on that to access imported modules, global variables, etc in your worker processes. The GIL and reference counting has nothing to care about it either, as they are both copied from the parent process as well, thus you can freely continue to read global state (and even modify, though it will be local to the worker process). That is, if you have large objects to share with your worker processes, and you don't need to get them back, you can freely just assign them as global variables before starting your multiprocessing pool, and have your workers read them, at zero transfer cost, and safely. This _trick_ has been used since decades in Python. Now the annoying thing is, that global variable mumbo jumbo is kind of dirty and ad-hoc to maintain. Using custom pickable weakrefs (using something similar to the trick with id() in this repo), you can create proxy weak objects to transfer to your workers, thus achieving the same effect, while keeping a "function argument" interface, instead of proxy8ng through global variables.
    - ptx 2y ago
      Do you have any publicly available code demonstrating this pattern?
      - Galanwe 2y ago
        I don't actually, but it can be explained in a few lines of code. Consider the following two simple functions: def ref(obj): return id(obj) def deref(addr): import ctypes return ctypes.cast(addr, ctypes.py_object).value Basically, this relies on an implementation detail of `id()` in CPython: the unique id of an object is its memory address. `ref()` returns a reference to an object (think `&` in C), and `deref()` dereferences it back (think `*` in C). This is close to the standard `weakref` module in essence, but weakref is a black box. Now even though the callstack is cleared upon fork of the worker processes, you still have the parent objects available, and properly tracked and refcounted, as you can check from `gc.get_objects()`. This is in fact a feature of `gc` as explained in the doc (https://docs.python.org/3/library/gc.html https://docs.python.org/3/library/gc.html): > If a process will fork() without exec(), avoiding unnecessary copy-on-write in child processes will maximize memory sharing and reduce overall memory usage. This requires both avoiding creation of freed “holes” in memory pages in the parent process and ensuring that GC collections in child processes won’t touch the gc_refs counter of long-lived objects originating in the parent process. To accomplish both, call gc.disable() early in the parent process, gc.freeze() right before fork(), and gc.enable() early in child processes. Now whenever you want to send large objects to a `multiprocessing.Pool` or `concurrent.futures.ProcessPoolExecutor`, you can avoid expensive pickling by just sending these references. class BigObject: pass def child(rbo): bo = deref(rbo) return bo.compute_something() def parent(): bo1 = BigObject() bo2 = BigObject() with Pool(2) as pool: result = pool.map(child, [ref(bo1), ref(bo2)]) In a real codebase though, there are some caveats around this. You cannot take the reference of just anything, there are temporaries, cached small integers, etc. You will need some form of higher level wrapper around `ref()` to properly choose when and what to reference or to copy. Also it may be inconvenient to have your child functions explicitely dereference their parameters, it will force you to write _dereference wrappers_ around your original functions. A good strategy I've used is to create a proxy class that stores a reference and override `__getstate__`/`__setstate__` for pickling itself as reference and unpickling itself as a proxy. That way, you can transparently pass these proxies to your original functions without any modification.
  - hot_gril 2y ago
    What do you mean by sharing objects between processes in Python? By default it sends a pickled copy to each subprocess, but there are special things like Manager I haven't used.
    - Galanwe 2y ago
      The `multiprocessing` and `concurrent.futures` pools support a fork strategy on Linux (which is the default). What that means is that your worker processes run in an interpreter whose state is the same as the parent. Effectively you will then inherit imported modules, global variables, etc. The mechanisms to send objects across threads do rely on pickling objects in a queue, yes. But if you know you will use a forking pool, you can just put your variables as globals before the fork and access them in your workers. Essentially, you can change this: def worker(obj): print(obj) def parent(): objs = [obj1, obj2] with Pool(2) as pool: pool.map(worker, objs) # each obj is pickled and sent to the worker through a queue To this: OBJS = [] def worker(idx): print(OBJS[idx]) def parent(): OBJS.append(obj1) OBJS.append(obj2) with Pool(2) as pool: pool.map(worker, range(len(OBJS))) del OBJS[0] del OBJS[1] > there are special things like Manager I haven't used. Managers are an abstraction over a shared memory, and objects that can be safely stored in it. From my experience it's cumbersome to use and very limited in functionality.
      - hot_gril 2y ago
        I tried this on a Linux machine in Py3 earlier, which I think is equivalent to your example: from multiprocessing import Pool shared = {} def f(x): global shared # idk if this matters but tried without it too shared[x] = x print(shared) if __name__ == '__main__': with Pool(5) as p: p.map(f, [1, 2, 3]) print(shared) It printed {1: 1}, {2: 2}, {3: 3} then {} at the end. So the parent and other children didn't see the change made by each child. Doesn't seem like they're RW sharing.
        Galanwe 2y ago
        Yes, as mentioned, the workers get a copy of the interpreter states. Every modification made in a worker to these objects stay local to the worker. These tricks are for sending data to a worker, not receiving back data.
Olesya000 2y ago
[dead]
owlstuffing 2y ago
Sergeant Frank Tree: You shouldn't touch the ordnance at all. But more specifically, you should never pull this hand-operating lever to the rear. Ward Douglas: Never.

54 points by albertzeyer 2y ago

There are some real world use cases for these. I do use some similar helpers to e.g. create pickable weakrefs for passing large objects to forking multiprocessing pools as arguments, or expose large objects in shared memories. It's not portable against non-CPython implementations, but as long as it's well documented and known, it's fine.

dabacaba 2y ago
Sharing Python objects between processes is not safe. Python assumes that access to objects is protected by GIL, but this is not the case when using multiple processes (even if the processes only read from the objects, they still modify the refcounts). Sharing memory buffers can be done safely using mmap.
- Galanwe 2y ago
  Sharing objects is completely fine upon a forking multiprocessing pool. Each worker process ends up with a (lazy) copy of the parent address space, including the whole python interpreter state. When you think about it, you already rely on that to access imported modules, global variables, etc in your worker processes. The GIL and reference counting has nothing to care about it either, as they are both copied from the parent process as well, thus you can freely continue to read global state (and even modify, though it will be local to the worker process). That is, if you have large objects to share with your worker processes, and you don't need to get them back, you can freely just assign them as global variables before starting your multiprocessing pool, and have your workers read them, at zero transfer cost, and safely. This _trick_ has been used since decades in Python. Now the annoying thing is, that global variable mumbo jumbo is kind of dirty and ad-hoc to maintain. Using custom pickable weakrefs (using something similar to the trick with id() in this repo), you can create proxy weak objects to transfer to your workers, thus achieving the same effect, while keeping a "function argument" interface, instead of proxy8ng through global variables.
  - ptx 2y ago
    Do you have any publicly available code demonstrating this pattern?
    - Galanwe 2y ago
      I don't actually, but it can be explained in a few lines of code. Consider the following two simple functions: def ref(obj): return id(obj) def deref(addr): import ctypes return ctypes.cast(addr, ctypes.py_object).value Basically, this relies on an implementation detail of `id()` in CPython: the unique id of an object is its memory address. `ref()` returns a reference to an object (think `&` in C), and `deref()` dereferences it back (think `*` in C). This is close to the standard `weakref` module in essence, but weakref is a black box. Now even though the callstack is cleared upon fork of the worker processes, you still have the parent objects available, and properly tracked and refcounted, as you can check from `gc.get_objects()`. This is in fact a feature of `gc` as explained in the doc (https://docs.python.org/3/library/gc.html https://docs.python.org/3/library/gc.html): > If a process will fork() without exec(), avoiding unnecessary copy-on-write in child processes will maximize memory sharing and reduce overall memory usage. This requires both avoiding creation of freed “holes” in memory pages in the parent process and ensuring that GC collections in child processes won’t touch the gc_refs counter of long-lived objects originating in the parent process. To accomplish both, call gc.disable() early in the parent process, gc.freeze() right before fork(), and gc.enable() early in child processes. Now whenever you want to send large objects to a `multiprocessing.Pool` or `concurrent.futures.ProcessPoolExecutor`, you can avoid expensive pickling by just sending these references. class BigObject: pass def child(rbo): bo = deref(rbo) return bo.compute_something() def parent(): bo1 = BigObject() bo2 = BigObject() with Pool(2) as pool: result = pool.map(child, [ref(bo1), ref(bo2)]) In a real codebase though, there are some caveats around this. You cannot take the reference of just anything, there are temporaries, cached small integers, etc. You will need some form of higher level wrapper around `ref()` to properly choose when and what to reference or to copy. Also it may be inconvenient to have your child functions explicitely dereference their parameters, it will force you to write _dereference wrappers_ around your original functions. A good strategy I've used is to create a proxy class that stores a reference and override `__getstate__`/`__setstate__` for pickling itself as reference and unpickling itself as a proxy. That way, you can transparently pass these proxies to your original functions without any modification.
- hot_gril 2y ago
  What do you mean by sharing objects between processes in Python? By default it sends a pickled copy to each subprocess, but there are special things like Manager I haven't used.
  - Galanwe 2y ago
    The `multiprocessing` and `concurrent.futures` pools support a fork strategy on Linux (which is the default). What that means is that your worker processes run in an interpreter whose state is the same as the parent. Effectively you will then inherit imported modules, global variables, etc. The mechanisms to send objects across threads do rely on pickling objects in a queue, yes. But if you know you will use a forking pool, you can just put your variables as globals before the fork and access them in your workers. Essentially, you can change this: def worker(obj): print(obj) def parent(): objs = [obj1, obj2] with Pool(2) as pool: pool.map(worker, objs) # each obj is pickled and sent to the worker through a queue To this: OBJS = [] def worker(idx): print(OBJS[idx]) def parent(): OBJS.append(obj1) OBJS.append(obj2) with Pool(2) as pool: pool.map(worker, range(len(OBJS))) del OBJS[0] del OBJS[1] > there are special things like Manager I haven't used. Managers are an abstraction over a shared memory, and objects that can be safely stored in it. From my experience it's cumbersome to use and very limited in functionality.
    - hot_gril 2y ago
      I tried this on a Linux machine in Py3 earlier, which I think is equivalent to your example: from multiprocessing import Pool shared = {} def f(x): global shared # idk if this matters but tried without it too shared[x] = x print(shared) if __name__ == '__main__': with Pool(5) as p: p.map(f, [1, 2, 3]) print(shared) It printed {1: 1}, {2: 2}, {3: 3} then {} at the end. So the parent and other children didn't see the change made by each child. Doesn't seem like they're RW sharing.
      - Galanwe 2y ago
        Yes, as mentioned, the workers get a copy of the interpreter states. Every modification made in a worker to these objects stay local to the worker. These tricks are for sending data to a worker, not receiving back data.

Olesya000 2y ago

[dead]

owlstuffing 2y ago

Sergeant Frank Tree: You shouldn't touch the ordnance at all. But more specifically, you should never pull this hand-operating lever to the rear. Ward Douglas: Never.