4 ms·

They did (it's me :)) and unfortunately in this case rpath can't be used, because that particular Python driver uses ctypes (https://docs.python.org/3/library/c

by hynek 4y ago

They did (it's me :)) and unfortunately in this case rpath can't be used, because that particular Python driver uses ctypes (https://docs.python.org/3/library/ctypes.html https://docs.python.org/3/library/ctypes.html) to open the binary drivers which oversimplified means that there is no binary top modify. I hope I make it in the preamble clear that this is bad and one should not have to deal with this – but it happens in practice and I hope such a summary is useful. For posterity: if you want to wrap / use a C library in Python, you should go for CFFI (Cython works too and is overall faster, but has other downsides). This PyCon US video is a great up-to-date summary: https://www.youtube.com/watch?v=gROGDQakzas https://www.youtube.com/watch?v=gROGDQakzas

throwaway09223 4y ago
You can still do it. rpath doesn't just apply to executable binaries, it applies to all shared objects including libraries. So, what you can do is modify the dyld that the python/ctypes module will eventually dlopen(). You should be able to find it by rooting around in the files installed by the module in question. rpath is evaluated not just at binary execution, but also upon any dynamic linking -- including dlopen(). So it will work for both ctypes and CFFI. Happy to provide further pointers if you need!
kosinus 4y ago
Maybe it's possible to patch the dependency? For example, for Node.js / npm, there are automated ways to do this, like [patch-package](https://www.npmjs.com/package/patch-package https://www.npmjs.com/package/patch-package). Does Python / pip have something similar?
- hynek 4y ago
  You mean at the Python driver level? Unfortunately, that doesn't work with ctypes. I've tried it by adding the DYLD_LIBRARY_PATH to os.environ before calling ctypes.LibraryLoader.LoadLibrary (https://docs.python.org/3/library/ctypes.html#ctypes.LibraryLoader https://docs.python.org/3/library/ctypes.html#ctypes.Library...) and it didn't work. I suspect ctypes gets somehow initialized much sooner and adding environment variables in your apps doesn't help. TBH I didn't research it further, since the problems of the post are more general and it can happen that you trip into them regardless of runtime.
  - orf 4y ago
    Here’s the relevant source: https://github.com/python/cpython/blob/8dd2766d99f8f51ad62dc0fde8282483590c6cd0/Lib/ctypes/__init__.py#L458 https://github.com/python/cpython/blob/8dd2766d99f8f51ad62dc... I believe you should just be able to replace the library name with an absolute path to the library, and remove the need for the lookup at all?
    - hynek 4y ago
      Right, but that would take some really deep patching of code I don't control, just so it works in development. If this were a production issue, I would probably fork the driver and do what you're suggesting (it's not like it's actively maintained or something :|).
      - orf 4y ago
        Why not fork it? If only to have the code under your control in case it abruptly becomes inaccessible. Then the fix is a single if statement. Seems worth it to me.
1vuio0pswjnm7 4y ago
Out of curiosity, what are the downsides of Cython?
- hynek 4y ago
  Conceptually, Cython is mainly for accelerating Python code, and can _also_ access C code. Meanwhile CFFI is specifically for calling C code and nothing else. I recommend the video for the differences. One concrete thing that pops to my mind is that Cython doesn't support Py_LIMITED_API which means that you need to ship a lot more binary wheels. At least the issue is still open (https://github.com/cython/cython/issues/2542 https://github.com/cython/cython/issues/2542) and Cython projects IME need new wheels for each minor Python release. Compare that to cffi projects that (musl & pypy aside) only have to ship wheels for one Python version / architecture: https://pypi.org/project/argon2-cffi-bindings/#files https://pypi.org/project/argon2-cffi-bindings/#files

by hynek 4y ago

throwaway09223 4y ago

You can still do it. rpath doesn't just apply to executable binaries, it applies to all shared objects including libraries. So, what you can do is modify the dyld that the python/ctypes module will eventually dlopen(). You should be able to find it by rooting around in the files installed by the module in question. rpath is evaluated not just at binary execution, but also upon any dynamic linking -- including dlopen(). So it will work for both ctypes and CFFI. Happy to provide further pointers if you need!

kosinus 4y ago

Maybe it's possible to patch the dependency? For example, for Node.js / npm, there are automated ways to do this, like [patch-package](https://www.npmjs.com/package/patch-package https://www.npmjs.com/package/patch-package). Does Python / pip have something similar?

hynek 4y ago
You mean at the Python driver level? Unfortunately, that doesn't work with ctypes. I've tried it by adding the DYLD_LIBRARY_PATH to os.environ before calling ctypes.LibraryLoader.LoadLibrary (https://docs.python.org/3/library/ctypes.html#ctypes.LibraryLoader https://docs.python.org/3/library/ctypes.html#ctypes.Library...) and it didn't work. I suspect ctypes gets somehow initialized much sooner and adding environment variables in your apps doesn't help. TBH I didn't research it further, since the problems of the post are more general and it can happen that you trip into them regardless of runtime.
- orf 4y ago
  Here’s the relevant source: https://github.com/python/cpython/blob/8dd2766d99f8f51ad62dc0fde8282483590c6cd0/Lib/ctypes/__init__.py#L458 https://github.com/python/cpython/blob/8dd2766d99f8f51ad62dc... I believe you should just be able to replace the library name with an absolute path to the library, and remove the need for the lookup at all?
  - hynek 4y ago
    Right, but that would take some really deep patching of code I don't control, just so it works in development. If this were a production issue, I would probably fork the driver and do what you're suggesting (it's not like it's actively maintained or something :|).
    - orf 4y ago
      Why not fork it? If only to have the code under your control in case it abruptly becomes inaccessible. Then the fix is a single if statement. Seems worth it to me.

1vuio0pswjnm7 4y ago

Out of curiosity, what are the downsides of Cython?

hynek 4y ago
Conceptually, Cython is mainly for accelerating Python code, and can _also_ access C code. Meanwhile CFFI is specifically for calling C code and nothing else. I recommend the video for the differences. One concrete thing that pops to my mind is that Cython doesn't support Py_LIMITED_API which means that you need to ship a lot more binary wheels. At least the issue is still open (https://github.com/cython/cython/issues/2542 https://github.com/cython/cython/issues/2542) and Cython projects IME need new wheels for each minor Python release. Compare that to cffi projects that (musl & pypy aside) only have to ship wheels for one Python version / architecture: https://pypi.org/project/argon2-cffi-bindings/#files https://pypi.org/project/argon2-cffi-bindings/#files