5 ms·

Serious question: Once I have been building Qt Widgets application in Python with PySide bindings. I needed to do some simple processing in several SLOTs and d

Serious question: Once I have been building Qt Widgets application in Python with PySide bindings. I needed to do some simple processing in several SLOTs and decided to use lambdas as an easy replacement. Kids, do not even think about attempting to do this at home, seriously. I have run into some extremely evasive random crashes and stuff. I did not dig to the bottom of the issue, although my guess is that Python lambdas are executed in main (constructing) thread context and that causes race conditions. My question: how do Golang's goroutines interact with native threads spawned by Qt?

ffjffsfr 10y ago
I have some experience building PyQt apps and can confirm random crashes (segfaults, sigaborts and such). I suspect this has nothing to do with lambdas. It may likely be something happening between Python and QT - remember that all things happening in QT/C++ world are invisible to Python. For example if your Python routine returns None (as many Python methods, functions do) instead of some other object that is expected by C++ (say QNetworkRequest) it may crash and die without any traceback. I spend hours on things like this - QT method must return something but in some cases was not returning expected object and whole thing was crashing. Python being Python doesn't care about return types, there is no Python traceback, you're left in the dark groping, browsing codebase trying to find what might get wrong, no debugger will help you. Working with QT bindings is very difficult. Debuging things is not easy. I'm not sure if using bindings is worth it. I suspect doing things in plain C++ QT might be easier to debug and maintain.
- blub 10y ago
  In C++ one gets a compilation error if the wrong type is returned from a function. Qt is generally type safe, but it uses quite a lot of pointer types and also a rather unusual for C++ parent-child resource cleanup rule for QObjects (e.g. all widgets). When something does crash, C++ doesn't offer any tracebacks, but if one has a core dump or can reproduce the crash under a debugger, a traceback can be easily obtained in the majority of cases.
  - ffjffsfr 10y ago
    > but if one has a core dump or can reproduce the crash under a debugger, a traceback can be easily obtained in the majority of cases. how do you create core dump? starting program under some debugger? I tried debugging with valgrind but the output contained so much noise it took my ages to find something. I also had to recompile QT used by PyQT because QT because QT requires some flags to be able to debug. What's the best way to debug things like this?
    - gpderetta 10y ago
      It should create a coredump on crash by default (either in the current directory or in a OS specific system directory, try /var/cores for example). If it doesn't, try 'ulimit -c unlimited' in your shell to enable coredumps for that session. Valgrind is not really useful for debugging crashes. Running under gdb would work though.
      - dom0 10y ago
        IMHO Valgrind is immensely useful for debugging crashes.
    - blub 10y ago
      A core dump is a file that's generated by the OS and includes a snapshot of the process state. It's useful when e.g. one is not able to run the program under a debugger. Enabling it depends on the OS, I would just search for enable core dumps on "my OS". If one can run the target program under a debugger, a core dump is not so helpful because it's a static snapshot, whereas a debugger can modify program state. I am guessing that one has to run the python process itself under the debugger and pass the parameters such as the script path. There will probably be a signal or exception condition at one point and one can take a look at the call stack. The call stack will probably be messy, as it will include python internals. An alternative way to debug this is to understand the lifetimes and preconditions of C++ widgets and signals and slots and review one's python code. Unless there's a bug in Qt itself, PyQt and Qt might have a misunderstanding about lifetimes and something gets cleaned up too soon or there's a dangling refrence to a cleaned up object
- xioxox 10y ago
  I hit some crashes with PyQt which were fixed if the Python slots were defined with "@slot". I think this may be now fixed.
  - ffjffsfr 10y ago
    You mean PyQtSlot? http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html#PyQt5.QtCore.pyqtSlot http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html#Py...
witty_username 10y ago
IIRC this occurs when Qt keeps a reference to something but Python deletes the object; and thus segfault. I've fixed crashes by keeping stuff assigned to variables to keep it from being GCed.
sirtaj 10y ago
Usually these kinds of errors are due to weird interaction of the python GC and the parent/child lifecycle used internally by QObject (and derived classes). In this case, you're depending on the Qt side (signal/slot management) to keep your lambda alive along with its entire call stack. But python doesn't know anything about QObject pointers, and will happily GC it. The next time that signal is fired, crash is extremely likely. This used to be a much bigger problem in PyQt and PySide 4 and earlier, but it appears to have gotten better recently in PyQt at least.
- dekhn 10y ago
  Yeah, I recently found some sample code written with a weird (non-standard) mechanism of moving data between Python and Qt and it had all sorts of problems. For example, they had a Python thread that was running a timer and when Qt shut down, it tried to deliver a Python callback afterwards and things got messed up. In general, the best way to work with Qt is to let it assume control, and it use its mechanisms for lifecycle-management. I converted the code so that it used a QThread-derived Python class and QTimer. Works great. You basically have to understand the Deep Magic that makes Python work under the hood, and ditto for Qt, too.
pstch 10y ago
This is a very interesting question (to which I don't know the answer), I'm curious how the coroutines will interact with the Qt events. Sidenote: It's also possible to build Qt applications in Python using AsyncIO (using quamash), by integrating the Qt event loop as an AsyncIO event loop. This can be helpful to avoid race conditions, as much more interactions can be handled without thread concurrency.
- icefo 10y ago
  Thanks to quamash you can use the asyncio event loop to build a Qt app that call other async stuff at the same time ! I managed to use pyqt and autobahn|python (wamp protocol, RPC and pub/sub) in the same app thanks to quamash. I set up an example app here https://github.com/icefo/PyQt5-Wamp https://github.com/icefo/PyQt5-Wamp
therecipe 10y ago
author here; The main golang thread lives in the same thread as Qt's main event loop. If you now want to "cross" threads from a goroutines (which may lives in another thread), you can use Qt's signals and slots to safely "cross" threads. Take a look at the "quick/bridge" examples to see how Go is interacting with QML, and therefore has to "cross" threads several time. edit: can't comment any further currently, I will try to answer everything once I'm unblocked.
- dang 10y ago
  Ugh, sorry about that. New accounts are rate limited to try to cut down on spammers and trolls. It's really bad that this also cuts down on participation by the creator of a project showing up to comment on it, which is one of the best things that can happen here. We've marked your account legit so it won't happen again. Welcome to HN!
  - therecipe 10y ago
    Thank you very much :)
sevensor 10y ago
I've run into this problem too. I think it has more to do with object lifetimes. Suppose you define your lambda in a method of a QWidget that's later destroyed. The underlying C++ object has been cleaned up already, but the signal's reference to the lambda is keeping the Python part of the QWidget alive. Then when the slot is triggered you get weird crashy behavior.

ffjffsfr 10y ago

I have some experience building PyQt apps and can confirm random crashes (segfaults, sigaborts and such). I suspect this has nothing to do with lambdas. It may likely be something happening between Python and QT - remember that all things happening in QT/C++ world are invisible to Python. For example if your Python routine returns None (as many Python methods, functions do) instead of some other object that is expected by C++ (say QNetworkRequest) it may crash and die without any traceback. I spend hours on things like this - QT method must return something but in some cases was not returning expected object and whole thing was crashing. Python being Python doesn't care about return types, there is no Python traceback, you're left in the dark groping, browsing codebase trying to find what might get wrong, no debugger will help you. Working with QT bindings is very difficult. Debuging things is not easy. I'm not sure if using bindings is worth it. I suspect doing things in plain C++ QT might be easier to debug and maintain.

blub 10y ago
In C++ one gets a compilation error if the wrong type is returned from a function. Qt is generally type safe, but it uses quite a lot of pointer types and also a rather unusual for C++ parent-child resource cleanup rule for QObjects (e.g. all widgets). When something does crash, C++ doesn't offer any tracebacks, but if one has a core dump or can reproduce the crash under a debugger, a traceback can be easily obtained in the majority of cases.
- ffjffsfr 10y ago
  > but if one has a core dump or can reproduce the crash under a debugger, a traceback can be easily obtained in the majority of cases. how do you create core dump? starting program under some debugger? I tried debugging with valgrind but the output contained so much noise it took my ages to find something. I also had to recompile QT used by PyQT because QT because QT requires some flags to be able to debug. What's the best way to debug things like this?
  - gpderetta 10y ago
    It should create a coredump on crash by default (either in the current directory or in a OS specific system directory, try /var/cores for example). If it doesn't, try 'ulimit -c unlimited' in your shell to enable coredumps for that session. Valgrind is not really useful for debugging crashes. Running under gdb would work though.
    - dom0 10y ago
      IMHO Valgrind is immensely useful for debugging crashes.
  - blub 10y ago
    A core dump is a file that's generated by the OS and includes a snapshot of the process state. It's useful when e.g. one is not able to run the program under a debugger. Enabling it depends on the OS, I would just search for enable core dumps on "my OS". If one can run the target program under a debugger, a core dump is not so helpful because it's a static snapshot, whereas a debugger can modify program state. I am guessing that one has to run the python process itself under the debugger and pass the parameters such as the script path. There will probably be a signal or exception condition at one point and one can take a look at the call stack. The call stack will probably be messy, as it will include python internals. An alternative way to debug this is to understand the lifetimes and preconditions of C++ widgets and signals and slots and review one's python code. Unless there's a bug in Qt itself, PyQt and Qt might have a misunderstanding about lifetimes and something gets cleaned up too soon or there's a dangling refrence to a cleaned up object
xioxox 10y ago
I hit some crashes with PyQt which were fixed if the Python slots were defined with "@slot". I think this may be now fixed.
- ffjffsfr 10y ago
  You mean PyQtSlot? http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html#PyQt5.QtCore.pyqtSlot http://pyqt.sourceforge.net/Docs/PyQt5/signals_slots.html#Py...

witty_username 10y ago

IIRC this occurs when Qt keeps a reference to something but Python deletes the object; and thus segfault. I've fixed crashes by keeping stuff assigned to variables to keep it from being GCed.

sirtaj 10y ago

Usually these kinds of errors are due to weird interaction of the python GC and the parent/child lifecycle used internally by QObject (and derived classes). In this case, you're depending on the Qt side (signal/slot management) to keep your lambda alive along with its entire call stack. But python doesn't know anything about QObject pointers, and will happily GC it. The next time that signal is fired, crash is extremely likely. This used to be a much bigger problem in PyQt and PySide 4 and earlier, but it appears to have gotten better recently in PyQt at least.

dekhn 10y ago
Yeah, I recently found some sample code written with a weird (non-standard) mechanism of moving data between Python and Qt and it had all sorts of problems. For example, they had a Python thread that was running a timer and when Qt shut down, it tried to deliver a Python callback afterwards and things got messed up. In general, the best way to work with Qt is to let it assume control, and it use its mechanisms for lifecycle-management. I converted the code so that it used a QThread-derived Python class and QTimer. Works great. You basically have to understand the Deep Magic that makes Python work under the hood, and ditto for Qt, too.

pstch 10y ago

This is a very interesting question (to which I don't know the answer), I'm curious how the coroutines will interact with the Qt events. Sidenote: It's also possible to build Qt applications in Python using AsyncIO (using quamash), by integrating the Qt event loop as an AsyncIO event loop. This can be helpful to avoid race conditions, as much more interactions can be handled without thread concurrency.

icefo 10y ago
Thanks to quamash you can use the asyncio event loop to build a Qt app that call other async stuff at the same time ! I managed to use pyqt and autobahn|python (wamp protocol, RPC and pub/sub) in the same app thanks to quamash. I set up an example app here https://github.com/icefo/PyQt5-Wamp https://github.com/icefo/PyQt5-Wamp

therecipe 10y ago

author here; The main golang thread lives in the same thread as Qt's main event loop. If you now want to "cross" threads from a goroutines (which may lives in another thread), you can use Qt's signals and slots to safely "cross" threads. Take a look at the "quick/bridge" examples to see how Go is interacting with QML, and therefore has to "cross" threads several time. edit: can't comment any further currently, I will try to answer everything once I'm unblocked.

dang 10y ago
Ugh, sorry about that. New accounts are rate limited to try to cut down on spammers and trolls. It's really bad that this also cuts down on participation by the creator of a project showing up to comment on it, which is one of the best things that can happen here. We've marked your account legit so it won't happen again. Welcome to HN!
- therecipe 10y ago
  Thank you very much :)

sevensor 10y ago

I've run into this problem too. I think it has more to do with object lifetimes. Suppose you define your lambda in a method of a QWidget that's later destroyed. The underlying C++ object has been cleaned up already, but the signal's reference to the lambda is keeping the Python part of the QWidget alive. Then when the slot is triggered you get weird crashy behavior.