3 ms·

replaced the print() for sys.stderr.write() to make it compatible with python2. the program still deadlocks on python3 but it works perfectly on python2, anyon

replaced the print() for sys.stderr.write() to make it compatible with python2. the program still deadlocks on python3 but it works perfectly on python2, anyone know what changed between implementations that could be triggering this issue?

itamarst 4y ago
From memory, the reason it deadlocks is: 1. Writing to stderr grabs a lock. 2. Part of the multiprocessing code (perhaps not present in Python 2) also grabs this lock. 3. If you fork at the right moment (which is quite likely with the loop) the lock is held by a thread that is now dead, and so now you're waiting for a lock to release that will never be released.
crabbone 4y ago
I don't believe OP's deadlock is related to a rare event of dead thread holding a lock. The problem is that any calls to multiprocessing objects if not guarded by if __name__ == '__main__' will result in infinite loop / fork bomb kind of situation. For some reason on OP's computer that probably causes an appearance of the program hanging, but really, it will just crash after a while exhausting some resource.
- itamarst 4y ago
  If I add a `if __name__ == '__main__':` I get the same result. Here's the gdb stack trace: #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x17bc7c0) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x17bc7c0) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x17bc7c0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 #3 0x00007ff326c9cc5f in do_futex_wait (sem=sem@entry=0x17bc7c0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 #4 0x00007ff326c9ccf8 in __new_sem_wait_slow64 (sem=0x17bc7c0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 #5 0x00007ff326c9cd71 in __new_sem_wait (sem=<optimized out>) at ./nptl/sem_wait.c:42 #6 0x000000000042765b in PyThread_acquire_lock_timed (lock=0x17bc7c0, microseconds=-1, intr_flag=0) at ../Python/thread_pthread.h:483 #7 0x0000000000625813 in _enter_buffered_busy (self=0x7ff326f330f0) at ../Modules/_io/bufferedio.c:281 #8 0x000000000045f9b3 in buffered_flush (self=0x7ff326f330f0, args=<optimized out>) at ../Modules/_io/bufferedio.c:825 #9 0x0000000000524185 in method_vectorcall_NOARGS (func=func@entry=<method_descriptor at remote 0x7ff326f611d0>, args=args@entry=0x7fffa8bc7a38, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/descrobject.c:436 #10 0x000000000053bd5a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffa8bc7a38, callable=<method_descriptor at remote 0x7ff326f611d0>, tstate=0x17bf410) at ../Include/cpython/abstract.h:118 #11 PyObject_VectorcallMethod (name=<optimized out>, args=0x7fffa8bc7a38, nargsf=<optimized out>, kwnames=0x0) at ../Objects/call.c:828 #12 0x000000000060283a in _PyObject_CallMethodIdNoArgs (name=0x8f6120 <PyId_flush.lto_priv.2>, self=<optimized out>) at ../Include/cpython/abstract.h:243 #13 _io_TextIOWrapper_flush_impl (self=0x7ff326f40040) at ../Modules/_io/textio.c:3038 #14 _io_TextIOWrapper_flush (self=0x7ff326f40040, _unused_ignored=<optimized out>) at ../Modules/_io/clinic/textio.c.h:685
  - crabbone 4y ago
    Well, it's hard for me to tell from this stack trace what's happening here. I don't recognize PyObject_VectorcallMethod. Is this a new thing? Last time I wrote any C code for Python I don't recall seeing / using this. Anyways, another common pitfall in multiprocessing is attempting to serialize multithreading / multiprocessing primitives s.a. locks, variables or mutexes. My memory may fail me, but, I think, it may result in deadlock too. I think, multiprocessing code tries to guard against it, but there are some weird rules for when it's OK for serialized objects to have those primitives (I think, initialization in __init__ is fine, but not so much otherwise or something like that), but the check isn't very good / just a heuristic... But, really, I don't remember this part well.
JosephRedfern 4y ago
This post links to another post by the same author which goes into this in a little more detail: https://pythonspeed.com/articles/python-multiprocessing/ https://pythonspeed.com/articles/python-multiprocessing/

itamarst 4y ago

From memory, the reason it deadlocks is: 1. Writing to stderr grabs a lock. 2. Part of the multiprocessing code (perhaps not present in Python 2) also grabs this lock. 3. If you fork at the right moment (which is quite likely with the loop) the lock is held by a thread that is now dead, and so now you're waiting for a lock to release that will never be released.

crabbone 4y ago

I don't believe OP's deadlock is related to a rare event of dead thread holding a lock. The problem is that any calls to multiprocessing objects if not guarded by if __name__ == '__main__' will result in infinite loop / fork bomb kind of situation. For some reason on OP's computer that probably causes an appearance of the program hanging, but really, it will just crash after a while exhausting some resource.

itamarst 4y ago
If I add a `if __name__ == '__main__':` I get the same result. Here's the gdb stack trace: #0 __futex_abstimed_wait_common64 (private=<optimized out>, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x17bc7c0) at ./nptl/futex-internal.c:57 #1 __futex_abstimed_wait_common (cancel=true, private=<optimized out>, abstime=0x0, clockid=0, expected=0, futex_word=0x17bc7c0) at ./nptl/futex-internal.c:87 #2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x17bc7c0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=<optimized out>) at ./nptl/futex-internal.c:139 #3 0x00007ff326c9cc5f in do_futex_wait (sem=sem@entry=0x17bc7c0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111 #4 0x00007ff326c9ccf8 in __new_sem_wait_slow64 (sem=0x17bc7c0, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183 #5 0x00007ff326c9cd71 in __new_sem_wait (sem=<optimized out>) at ./nptl/sem_wait.c:42 #6 0x000000000042765b in PyThread_acquire_lock_timed (lock=0x17bc7c0, microseconds=-1, intr_flag=0) at ../Python/thread_pthread.h:483 #7 0x0000000000625813 in _enter_buffered_busy (self=0x7ff326f330f0) at ../Modules/_io/bufferedio.c:281 #8 0x000000000045f9b3 in buffered_flush (self=0x7ff326f330f0, args=<optimized out>) at ../Modules/_io/bufferedio.c:825 #9 0x0000000000524185 in method_vectorcall_NOARGS (func=func@entry=<method_descriptor at remote 0x7ff326f611d0>, args=args@entry=0x7fffa8bc7a38, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ../Objects/descrobject.c:436 #10 0x000000000053bd5a in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffa8bc7a38, callable=<method_descriptor at remote 0x7ff326f611d0>, tstate=0x17bf410) at ../Include/cpython/abstract.h:118 #11 PyObject_VectorcallMethod (name=<optimized out>, args=0x7fffa8bc7a38, nargsf=<optimized out>, kwnames=0x0) at ../Objects/call.c:828 #12 0x000000000060283a in _PyObject_CallMethodIdNoArgs (name=0x8f6120 <PyId_flush.lto_priv.2>, self=<optimized out>) at ../Include/cpython/abstract.h:243 #13 _io_TextIOWrapper_flush_impl (self=0x7ff326f40040) at ../Modules/_io/textio.c:3038 #14 _io_TextIOWrapper_flush (self=0x7ff326f40040, _unused_ignored=<optimized out>) at ../Modules/_io/clinic/textio.c.h:685
- crabbone 4y ago
  Well, it's hard for me to tell from this stack trace what's happening here. I don't recognize PyObject_VectorcallMethod. Is this a new thing? Last time I wrote any C code for Python I don't recall seeing / using this. Anyways, another common pitfall in multiprocessing is attempting to serialize multithreading / multiprocessing primitives s.a. locks, variables or mutexes. My memory may fail me, but, I think, it may result in deadlock too. I think, multiprocessing code tries to guard against it, but there are some weird rules for when it's OK for serialized objects to have those primitives (I think, initialization in __init__ is fine, but not so much otherwise or something like that), but the check isn't very good / just a heuristic... But, really, I don't remember this part well.

JosephRedfern 4y ago

This post links to another post by the same author which goes into this in a little more detail: https://pythonspeed.com/articles/python-multiprocessing/ https://pythonspeed.com/articles/python-multiprocessing/