6 ms·

I'm going to ask a 'self-taught' question, so my terms may be incorrect. Has anyone solved the compiled libraries / dynamic libraries / linking problems for mu

I'm going to ask a 'self-taught' question, so my terms may be incorrect. Has anyone solved the compiled libraries / dynamic libraries / linking problems for multiplatform apps? Everything works just fine with these python>native app tools until you hit some esoteric library that's not pure python.

willcipriano 3y ago
Solved as in you don't have to do anything and it just works or solved as in there is a way to fix the problem but it requires some effort?
- anonymouse008 3y ago
  Ha! Yeah I want the magic wand. I have deep wounds from pre-chatGPT times of trying to manually figure out libraries. As in otooling each library, error, etc then recompiling into various places until things 'worked.' I'm appreciative of the experience, though now I think with LLMs the problem could be solved with some simple queries. So I guess you're right, it requires effort, but I was a 10x engineer in the wrong direction and never want to experience that again ;)
  - willcipriano 3y ago
    What works for me is figuring out the platforms I need to support and creating a build pipeline that can handle those requirements. If you join onto a project I don't want you playing around with libraries on your local machine. If you are I consider that a failure of the build script/makefile/docker compose file/etc. First it's lost productivity, second you might get it wrong and have misleading results, causing more lost productivity. A lot of places do this nonsense "spend hours configuring your machine" method for projects. The only reason I can think of doing that is "job security" and who wants to keep a job with horrible tooling? But yeah, it takes effort to do.
    - anonymouse008 3y ago
      That's definitely working smarter. I'm coming around to understanding how all the tooling works - the lost weeks learning the basics happen less often now. In a way, that time was a chapter in 'job security.' Ran out of money, had to let everyone go, and picked up the pieces to fight another day (self-taught under threat of survival). By the grace of God I'm a reformed 'idea-guy' and am now a builder.
KeplerBoy 3y ago
esoteric as in a whole lot of essential libraries?
- deleted 3y ago
  [deleted]
- stuaxo 3y ago
  This needs solving. It makes it really hard to distribute certain python libraries.
  - robertlagrant 3y ago
    Do wheels not solve this? I know not everyone ships wheels, but are they not the answer?
schemescape 3y ago
For Node, I’ve seen packages just bundle a copy of the shared library for each supported platform. I would assume you could do the same for Python.
- aragilar 3y ago
  That's what wheels do.
BiteCode_dev 3y ago
Nuitka deals pretty well with those in general: https://nuitka.net/ https://nuitka.net/
- Jack5500 3y ago
  Until you require pytorch ;)
  - eyegor 3y ago
    What kind of madman tries to deploy pytorch? If it's a product, convert the finished model to onnx. Otherwise it's probably for a dev and you can tell them to have fun because installing all the gpu libs sucks no matter what. In theory mambaforge should make most of these scenarios less painful (mostly portable faster version of anaconda), but you'll still need the right cuda runtime, rocm, etc.
    - eternityforest 3y ago
      What makes this stuff so hard? I could probably find a Python tutorial to render a spinning cube in like 5 minutes, but as soon as you want to do ML you spend an hour just getting libraries to work? Is there ever going to be standardization, the way a WebGL shader just works everywhere?
  - BiteCode_dev 3y ago
    True, I never think about the machine learning stack as something I want a stand alone executable for.
pjc50 3y ago
Python remains a mess for these use cases because it does not have packages. It has a very specific filesystem layout that it calls a package, but this is not a first class citizen; it has a packaging format called "wheel" that is now used for distribution, and that can contain native libraries. That currently mostly handles the distribution of compiled libraries for multiple platforms, but only within the traditional venv-style setup. Being able to ship interpreter+venv+launcher would go a long way towards providing a native-ish experience. A step beyond that would be being able to distribute a "virtual filesystem" containing a whole tree of python compiled as .pyc, appended to a native executable containing the interpreter and linked against the required native libraries already.
- piyh 3y ago
  Venvs not being able to be moved to other directories is such a PITA.
  - plonk 3y ago
    They are though, at least on Windows, I’ve moved them before. The whole Lib/ tree was independent of the venv’s path, it’s only the “activate” script that hard-codes a full path. Since venvs on Linux symlink to the system’s Python executable, they should also work after moving.
    - ShamelessC 3y ago
      > it’s only the “activate” script that hard-codes a full path. So…they aren’t then.
      - plonk 3y ago
        They are in the sense that anything one would want to do with a venv, they can do with trivial effort. You don't need the activate script, you can just set PATH, PYTHONHOME and/or PYTHONPATH and be done with it.
    - dragonwriter 3y ago
oefrha 3y ago
PyInstaller bundles native dependencies just fine. Well, for Windows and macOS at least, never packaged as “native app” for Linux.
- Frost1x 3y ago
  Eh it depends, some python packages do things like pull data samples from my experience and it can be quite a mess detangling the package to give PyInstaller everything it needs to bundle stuff up. It also takes a lot of additional configuration for a lot of packages. As much as I hate how containers are often misused, this is one of the reasons people obsess over them from my experience. The ability to bundle all your crap regardless of how many layers of package management idiosyncrasies your application ultimately woul have to deal with pretty much transparently.
simonw 3y ago
Python wheels really can help here, but only if the downstream packages you depend on have published wheels for the various different platforms you need to support. https://cibuildwheel.readthedocs.io/ https://cibuildwheel.readthedocs.io/ is a great tool for assisting with that - many projects run that in GitHub Actions as part of their release process. There's just one catch, in my experience: since GitHub don't offer Apple Silicon machines as part of GitHub Actions, it can be hard to build wheels there that work on those platforms. Maybe you can cross-compile them? I've not figured that out yet.
eyegor 3y ago
I vaguely remember nuitka or pyinstaller being able to handle this for me, but I used vms or qemu to build them "natively" for each target platform. Never found a way to solve cross compiling.

willcipriano 3y ago

Solved as in you don't have to do anything and it just works or solved as in there is a way to fix the problem but it requires some effort?

anonymouse008 3y ago
Ha! Yeah I want the magic wand. I have deep wounds from pre-chatGPT times of trying to manually figure out libraries. As in otooling each library, error, etc then recompiling into various places until things 'worked.' I'm appreciative of the experience, though now I think with LLMs the problem could be solved with some simple queries. So I guess you're right, it requires effort, but I was a 10x engineer in the wrong direction and never want to experience that again ;)
- willcipriano 3y ago
  What works for me is figuring out the platforms I need to support and creating a build pipeline that can handle those requirements. If you join onto a project I don't want you playing around with libraries on your local machine. If you are I consider that a failure of the build script/makefile/docker compose file/etc. First it's lost productivity, second you might get it wrong and have misleading results, causing more lost productivity. A lot of places do this nonsense "spend hours configuring your machine" method for projects. The only reason I can think of doing that is "job security" and who wants to keep a job with horrible tooling? But yeah, it takes effort to do.
  - anonymouse008 3y ago
    That's definitely working smarter. I'm coming around to understanding how all the tooling works - the lost weeks learning the basics happen less often now. In a way, that time was a chapter in 'job security.' Ran out of money, had to let everyone go, and picked up the pieces to fight another day (self-taught under threat of survival). By the grace of God I'm a reformed 'idea-guy' and am now a builder.

KeplerBoy 3y ago

esoteric as in a whole lot of essential libraries?

deleted 3y ago
[deleted]
stuaxo 3y ago
This needs solving. It makes it really hard to distribute certain python libraries.
- robertlagrant 3y ago
  Do wheels not solve this? I know not everyone ships wheels, but are they not the answer?

schemescape 3y ago

For Node, I’ve seen packages just bundle a copy of the shared library for each supported platform. I would assume you could do the same for Python.

aragilar 3y ago
That's what wheels do.

BiteCode_dev 3y ago

Nuitka deals pretty well with those in general: https://nuitka.net/ https://nuitka.net/

Jack5500 3y ago
Until you require pytorch ;)
- eyegor 3y ago
  What kind of madman tries to deploy pytorch? If it's a product, convert the finished model to onnx. Otherwise it's probably for a dev and you can tell them to have fun because installing all the gpu libs sucks no matter what. In theory mambaforge should make most of these scenarios less painful (mostly portable faster version of anaconda), but you'll still need the right cuda runtime, rocm, etc.
  - eternityforest 3y ago
    What makes this stuff so hard? I could probably find a Python tutorial to render a spinning cube in like 5 minutes, but as soon as you want to do ML you spend an hour just getting libraries to work? Is there ever going to be standardization, the way a WebGL shader just works everywhere?
- BiteCode_dev 3y ago
  True, I never think about the machine learning stack as something I want a stand alone executable for.

pjc50 3y ago

Python remains a mess for these use cases because it does not have packages. It has a very specific filesystem layout that it calls a package, but this is not a first class citizen; it has a packaging format called "wheel" that is now used for distribution, and that can contain native libraries. That currently mostly handles the distribution of compiled libraries for multiple platforms, but only within the traditional venv-style setup. Being able to ship interpreter+venv+launcher would go a long way towards providing a native-ish experience. A step beyond that would be being able to distribute a "virtual filesystem" containing a whole tree of python compiled as .pyc, appended to a native executable containing the interpreter and linked against the required native libraries already.

piyh 3y ago
Venvs not being able to be moved to other directories is such a PITA.
- plonk 3y ago
  They are though, at least on Windows, I’ve moved them before. The whole Lib/ tree was independent of the venv’s path, it’s only the “activate” script that hard-codes a full path. Since venvs on Linux symlink to the system’s Python executable, they should also work after moving.
  - ShamelessC 3y ago
    > it’s only the “activate” script that hard-codes a full path. So…they aren’t then.
    - plonk 3y ago
      They are in the sense that anything one would want to do with a venv, they can do with trivial effort. You don't need the activate script, you can just set PATH, PYTHONHOME and/or PYTHONPATH and be done with it.
  - dragonwriter 3y ago

oefrha 3y ago

PyInstaller bundles native dependencies just fine. Well, for Windows and macOS at least, never packaged as “native app” for Linux.

Frost1x 3y ago
Eh it depends, some python packages do things like pull data samples from my experience and it can be quite a mess detangling the package to give PyInstaller everything it needs to bundle stuff up. It also takes a lot of additional configuration for a lot of packages. As much as I hate how containers are often misused, this is one of the reasons people obsess over them from my experience. The ability to bundle all your crap regardless of how many layers of package management idiosyncrasies your application ultimately woul have to deal with pretty much transparently.

simonw 3y ago

Python wheels really can help here, but only if the downstream packages you depend on have published wheels for the various different platforms you need to support. https://cibuildwheel.readthedocs.io/ https://cibuildwheel.readthedocs.io/ is a great tool for assisting with that - many projects run that in GitHub Actions as part of their release process. There's just one catch, in my experience: since GitHub don't offer Apple Silicon machines as part of GitHub Actions, it can be hard to build wheels there that work on those platforms. Maybe you can cross-compile them? I've not figured that out yet.

eyegor 3y ago

I vaguely remember nuitka or pyinstaller being able to handle this for me, but I used vms or qemu to build them "natively" for each target platform. Never found a way to solve cross compiling.