6 ms·

IIRC a big issue with PyInstaller is that the built executables are really self-extracting archives, which have to write many files to disk before they can run.

by pansa2 5y ago

IIRC a big issue with PyInstaller is that the built executables are really self-extracting archives, which have to write many files to disk before they can run. Compared to a real compiled executable, it’s slow and inelegant. Unfortunately high-quality bundling into executables just isn’t a focus of Python (nor of any other high-level language). Personally, I’ve gone back to C++ for building command-line apps - as a developer I’d much rather be writing Python, but that’s no good if I can’t actually deliver software to users.

andi999 5y ago
I fully agree. As a close runner up on C++ for bundling into executables is C# in my opinion.
- pansa2 5y ago
  How well does C# work for this cross-platform? Are its executables very large? I considered using Go for command-line apps - but in my case they often have to interact with existing C++ code. That’s easy enough to do from Python, but I was advised that interop with other languages is an area in which Go is particularly weak.
  - pca 5y ago
    I think a standalone "Hello World", built with .NET Core, is around 60-70 MB large on both Linux and Windows. The startup times are alright - slower on Windows than on Linux in my experience because of the file system performance.
    - arboghast 5y ago
      Lower than this. If I use the AOT compile option with .NET Core 3.1 I get a 40MB binary which goes down to 10MB with the trim option. Just like Go which embed it’s runtime, .NET Core binary compiled with AOT embed the CLR but, unlike Go, it has options to trim out the pieces that aren’t necessary, which I really like.
      - pca 5y ago
        I did not know about PublishTrimmed and will try it out. Thank you.
        pjc50 5y ago
        I think this is one of those features that was a bit preview in 5 and you should use the just-released v6 for PublishTrimmed. There are also situations where it doesn't work out of the box (e.g. Winui3!)
    - formerly_proven 5y ago
      If you use legacy .NET (which is preinstalled on Windows) a GUI Hello World is about 10 kB.
    - chrisseaton 5y ago
      > I think a standalone "Hello World", built with .NET Core, is around 60-70 MB large on both Linux and Windows. Lol what? What goes into a 70 MB implementation of 'hello world'?
      - deleted 5y ago
        [deleted]
      - Kwpolska 5y ago
        It’s not a 70 MB implementation of Hello world. It’s a 70 MB compressed runtime with a standard library (or at least the most basic parts of it) and a couple KB "Hello world".
        chrisseaton 5y ago
        Why is the runtime so huge and why is so much of the standard library included? Java ‘hello world’ compiled into a static binary is just a couple of MB.
        pjc50 5y ago
        See PublishTrimmed: by default it includes all the standard library. "Trimming" is what almost all C-style linkers do by default, but C# doesn't because there are some reflection issues (e.g. you can reference types by strings and construct them at runtime, which would fail in a Trimmed environment unless you explicitly tell it not to trim those types).
        older 5y ago
        Can this couple MB static binary run without JRE?
        chrisseaton 5y ago
        Well it wouldn't be very 'static' if it couldn't, so yes of course.
  - dmingod666 5y ago
    gRPC is one way to go about it if you have control over both c++ and go. Go binaries work pretty consistently across Linux which is a positive.
abbbi 5y ago
yes, same for cxfreeze etc. It really comes with a cost. If you have mid ranged applications which might spawn several processes of the executable this becomes really problematic (cpu usage, memory usage).
ZuLuuuuuu 5y ago
I thought that it is the case only when you bundle into "one file"? If you don't pass the --onefile parameter, all files are in one folder without any archiving, am I wrong?
- pansa2 5y ago
  That’s right - but if you don’t use `-—onefile` you don’t really have a “stand-alone executable” at all.
  - scarygliders 5y ago
    Correct. I don't use --onefile at all. I go the old-skool way, and package my python Windows app(s) into an installer using Inno Setup.
  - mherrmann 5y ago
    Of course you have a stand-alone executable. At least if by "stand-alone" you mean a copy you can ship to users. Which is at least the only thing I care about. And tell me which popular (say) Windows software actually comes as a single .exe. They're all many files in multiple folders. And there's no problem with that. You have one installer, which users download, which then extracts the files. Your criticism points out a non-issue.
dmingod666 5y ago
You also need to make sure to compile the binaries in the OS version that is close to your target deployment version, I remember had RHEL 6.x Vs RHEL 7 issues due to libc or a similar dependency.
- Tsarbomb 5y ago
  Yes this. Differing glibc versions caused us headaches.
minusf 5y ago
check out https://pyoxidizer.readthedocs.io https://pyoxidizer.readthedocs.io
serial_dev 5y ago
> Unfortunately high-quality bundling into executables just isn’t a focus of Python (nor of any other high-level language) Wouldn't Dart's [1] and Go's executable support qualify as such (high levels languages and good executable support)? For example, Sass is also using now Dart [2]. [1] https://dart.dev/tools/dart-compile https://dart.dev/tools/dart-compile [2] https://sass-lang.com/dart-sass https://sass-lang.com/dart-sass
- moffkalast 5y ago
  Let me know once dart or go can run a wide ecosystem of libraries and middleware that apply to almost any situation. There's a reason people still use languages like C++, Python, and Java despite them being objectively bad in some respects. Without mainstream adoption you've got a chicken and the egg problem.
  - grey-area 5y ago
    Sure, here's your reminder that Go supports a wide ecosystem of libraries and middleware that apply to almost any situation.
    - moffkalast 5y ago
      I think you're missing my point here. Let's say you're given a pytorch model. You can either run it with native python or grab the golang bindings that are used by 5 people, and if you get stuck there, god help you. You'll be hitting obscure bugs in no time. Even if something officially supports multiple languages, the obscure ones will be usually get inferiorly implemented bindings just because they're not used as much and will consequently not be as tested and as patched as the main ones. It's unfortunate but that's life I guess.
      - chrisseaton 5y ago
        > I think you're missing my point here. Let's say you're given a PyTorch model. You can either run it with native python or grab the golang bindings that are used by 5 people, and if you get stuck there, god help you. You'll be hitting obscure bugs in no time. To use a Python-specific framework you're better off using Python. Well duh.
formerly_proven 5y ago
pyinstaller is kinda dumb here because it doesn't cache the extracted files; it extracts all of them (which is especially slow on Windows), runs the application from that temporary directory, then deletes it. At work I've built a simple cross-platform packaging system which avoids issues like this and also separates library components from the main application (the former is rarely changed, the latter is frequently changed). Despite the entire application being around ~25 MB, it starts in around 30-50 ms from a network share on Linux and 1-2 seconds on Windows. On Linux it's just a bash script eventually calling a system-provided python 3 interpreter, on Windows there's a C# .exe which does roughly the same but the package also includes a Python interpreter (which is just the official "Python for embedding into applications" ZIP) which the C# process loads via P/Invoke. The entire thing is less than 1000 lines of Python/bash/C# and works with every package under the sun and apart from the C# stubs, which basically never change, it's supremely easy to "cross compile" because compilation in this system is combining the wheels of all dependencies into a single zip, which also makes builds completely reproducible.
- LinuxBender 5y ago
  then deletes it Is that built into python or does the developer call that cleanup? I ask because I recently switched from youtube-dl to yt-dlp on a windows machine and it left multiple temporary directories full of .py files to be cleaned up by other tools. I did cntrl+c out of it at least once. Perhaps I interrupted the cleanup routine? Is the cleanup routine expected to trap kill signals or is that on the developer?
  - formerly_proven 5y ago
    The cleanup code is part of the stub/bootloader, so depending on how the process dies it doesn't run.
- gjvc 5y ago
  https://docs.python.org/3/library/zipapp.html https://docs.python.org/3/library/zipapp.html
pgcj_poster 5y ago
I would have thought most command-line users would be content to install with pip.
cycomanic 5y ago
I've not tried it myself, but heard people having good experience with using nuikita (https://nuitka.net/ https://nuitka.net/) for the purpose of building standalone python executables.
- dr_zoidberg 5y ago
  I've used it a couple of times. When it works, it's great, but there can be issues with bundling dependencies correctly. As such, I've resorted to using it just with the standard library, or a few 3rd party-modules that are nuitka-friendly.
ta988 5y ago
I love Go for quick and dirty CLI apps. They are a bit bigger than the C app but so much friendlier to develop on. Rust is better for more complex processing, but it is also less straightforward (at least for me).
- UnpossibleJim 5y ago
  I say this as a Rust fan and a (sometimes overzealous) promoter; it's just easier to wrap your head around programming in Go. They have made it so easy and enjoyable. It really is a pleasure for quick concepts to table, so to speak.
cmacleod4 5y ago
> Unfortunately high-quality bundling into executables just isn’t a focus of Python (nor of any other high-level language). You seem to be ignoring Tcl, which solved this problem about 20 years ago with Starkits (https://wiki.tcl-lang.org/page/Starkit https://wiki.tcl-lang.org/page/Starkit) and Starpacks (https://wiki.tcl-lang.org/page/Starpack https://wiki.tcl-lang.org/page/Starpack).
packetlost 5y ago
While not entirely self-contained, `pex` is pretty close. It relies on a Python interpreter in the environment somewhere, but in my experience it successfully packages a venv + code into a single executable "binary" with no extraction stage.

by pansa2 5y ago

andi999 5y ago

I fully agree. As a close runner up on C++ for bundling into executables is C# in my opinion.

pansa2 5y ago
How well does C# work for this cross-platform? Are its executables very large? I considered using Go for command-line apps - but in my case they often have to interact with existing C++ code. That’s easy enough to do from Python, but I was advised that interop with other languages is an area in which Go is particularly weak.
- pca 5y ago
  I think a standalone "Hello World", built with .NET Core, is around 60-70 MB large on both Linux and Windows. The startup times are alright - slower on Windows than on Linux in my experience because of the file system performance.
  - arboghast 5y ago
    Lower than this. If I use the AOT compile option with .NET Core 3.1 I get a 40MB binary which goes down to 10MB with the trim option. Just like Go which embed it’s runtime, .NET Core binary compiled with AOT embed the CLR but, unlike Go, it has options to trim out the pieces that aren’t necessary, which I really like.
    - pca 5y ago
      I did not know about PublishTrimmed and will try it out. Thank you.
      - pjc50 5y ago
        I think this is one of those features that was a bit preview in 5 and you should use the just-released v6 for PublishTrimmed. There are also situations where it doesn't work out of the box (e.g. Winui3!)
  - formerly_proven 5y ago
    If you use legacy .NET (which is preinstalled on Windows) a GUI Hello World is about 10 kB.
  - chrisseaton 5y ago
    > I think a standalone "Hello World", built with .NET Core, is around 60-70 MB large on both Linux and Windows. Lol what? What goes into a 70 MB implementation of 'hello world'?
    - deleted 5y ago
      [deleted]
    - Kwpolska 5y ago
      It’s not a 70 MB implementation of Hello world. It’s a 70 MB compressed runtime with a standard library (or at least the most basic parts of it) and a couple KB "Hello world".
      - chrisseaton 5y ago
        Why is the runtime so huge and why is so much of the standard library included? Java ‘hello world’ compiled into a static binary is just a couple of MB.
        pjc50 5y ago
        See PublishTrimmed: by default it includes all the standard library. "Trimming" is what almost all C-style linkers do by default, but C# doesn't because there are some reflection issues (e.g. you can reference types by strings and construct them at runtime, which would fail in a Trimmed environment unless you explicitly tell it not to trim those types).
        older 5y ago
        Can this couple MB static binary run without JRE?
        chrisseaton 5y ago
        Well it wouldn't be very 'static' if it couldn't, so yes of course.
- dmingod666 5y ago
  gRPC is one way to go about it if you have control over both c++ and go. Go binaries work pretty consistently across Linux which is a positive.

abbbi 5y ago

yes, same for cxfreeze etc. It really comes with a cost. If you have mid ranged applications which might spawn several processes of the executable this becomes really problematic (cpu usage, memory usage).

ZuLuuuuuu 5y ago

I thought that it is the case only when you bundle into "one file"? If you don't pass the --onefile parameter, all files are in one folder without any archiving, am I wrong?

pansa2 5y ago
That’s right - but if you don’t use `-—onefile` you don’t really have a “stand-alone executable” at all.
- scarygliders 5y ago
  Correct. I don't use --onefile at all. I go the old-skool way, and package my python Windows app(s) into an installer using Inno Setup.
- mherrmann 5y ago
  Of course you have a stand-alone executable. At least if by "stand-alone" you mean a copy you can ship to users. Which is at least the only thing I care about. And tell me which popular (say) Windows software actually comes as a single .exe. They're all many files in multiple folders. And there's no problem with that. You have one installer, which users download, which then extracts the files. Your criticism points out a non-issue.

dmingod666 5y ago

You also need to make sure to compile the binaries in the OS version that is close to your target deployment version, I remember had RHEL 6.x Vs RHEL 7 issues due to libc or a similar dependency.

Tsarbomb 5y ago
Yes this. Differing glibc versions caused us headaches.

minusf 5y ago

check out https://pyoxidizer.readthedocs.io https://pyoxidizer.readthedocs.io

serial_dev 5y ago

> Unfortunately high-quality bundling into executables just isn’t a focus of Python (nor of any other high-level language) Wouldn't Dart's [1] and Go's executable support qualify as such (high levels languages and good executable support)? For example, Sass is also using now Dart [2]. [1] https://dart.dev/tools/dart-compile https://dart.dev/tools/dart-compile [2] https://sass-lang.com/dart-sass https://sass-lang.com/dart-sass

moffkalast 5y ago
Let me know once dart or go can run a wide ecosystem of libraries and middleware that apply to almost any situation. There's a reason people still use languages like C++, Python, and Java despite them being objectively bad in some respects. Without mainstream adoption you've got a chicken and the egg problem.
- grey-area 5y ago
  Sure, here's your reminder that Go supports a wide ecosystem of libraries and middleware that apply to almost any situation.
  - moffkalast 5y ago
    I think you're missing my point here. Let's say you're given a pytorch model. You can either run it with native python or grab the golang bindings that are used by 5 people, and if you get stuck there, god help you. You'll be hitting obscure bugs in no time. Even if something officially supports multiple languages, the obscure ones will be usually get inferiorly implemented bindings just because they're not used as much and will consequently not be as tested and as patched as the main ones. It's unfortunate but that's life I guess.
    - chrisseaton 5y ago
      > I think you're missing my point here. Let's say you're given a PyTorch model. You can either run it with native python or grab the golang bindings that are used by 5 people, and if you get stuck there, god help you. You'll be hitting obscure bugs in no time. To use a Python-specific framework you're better off using Python. Well duh.

formerly_proven 5y ago

pyinstaller is kinda dumb here because it doesn't cache the extracted files; it extracts all of them (which is especially slow on Windows), runs the application from that temporary directory, then deletes it. At work I've built a simple cross-platform packaging system which avoids issues like this and also separates library components from the main application (the former is rarely changed, the latter is frequently changed). Despite the entire application being around ~25 MB, it starts in around 30-50 ms from a network share on Linux and 1-2 seconds on Windows. On Linux it's just a bash script eventually calling a system-provided python 3 interpreter, on Windows there's a C# .exe which does roughly the same but the package also includes a Python interpreter (which is just the official "Python for embedding into applications" ZIP) which the C# process loads via P/Invoke. The entire thing is less than 1000 lines of Python/bash/C# and works with every package under the sun and apart from the C# stubs, which basically never change, it's supremely easy to "cross compile" because compilation in this system is combining the wheels of all dependencies into a single zip, which also makes builds completely reproducible.

LinuxBender 5y ago
then deletes it Is that built into python or does the developer call that cleanup? I ask because I recently switched from youtube-dl to yt-dlp on a windows machine and it left multiple temporary directories full of .py files to be cleaned up by other tools. I did cntrl+c out of it at least once. Perhaps I interrupted the cleanup routine? Is the cleanup routine expected to trap kill signals or is that on the developer?
- formerly_proven 5y ago
  The cleanup code is part of the stub/bootloader, so depending on how the process dies it doesn't run.
gjvc 5y ago
https://docs.python.org/3/library/zipapp.html https://docs.python.org/3/library/zipapp.html

pgcj_poster 5y ago

I would have thought most command-line users would be content to install with pip.

cycomanic 5y ago

I've not tried it myself, but heard people having good experience with using nuikita (https://nuitka.net/ https://nuitka.net/) for the purpose of building standalone python executables.

dr_zoidberg 5y ago
I've used it a couple of times. When it works, it's great, but there can be issues with bundling dependencies correctly. As such, I've resorted to using it just with the standard library, or a few 3rd party-modules that are nuitka-friendly.

ta988 5y ago

I love Go for quick and dirty CLI apps. They are a bit bigger than the C app but so much friendlier to develop on. Rust is better for more complex processing, but it is also less straightforward (at least for me).

UnpossibleJim 5y ago
I say this as a Rust fan and a (sometimes overzealous) promoter; it's just easier to wrap your head around programming in Go. They have made it so easy and enjoyable. It really is a pleasure for quick concepts to table, so to speak.

cmacleod4 5y ago

> Unfortunately high-quality bundling into executables just isn’t a focus of Python (nor of any other high-level language). You seem to be ignoring Tcl, which solved this problem about 20 years ago with Starkits (https://wiki.tcl-lang.org/page/Starkit https://wiki.tcl-lang.org/page/Starkit) and Starpacks (https://wiki.tcl-lang.org/page/Starpack https://wiki.tcl-lang.org/page/Starpack).

packetlost 5y ago

While not entirely self-contained, `pex` is pretty close. It relies on a Python interpreter in the environment somewhere, but in my experience it successfully packages a venv + code into a single executable "binary" with no extraction stage.