7 ms·

This is truly a problem. Even more so if you host your application on a network directory. Loading all the small files takes ages. I really wish there would be

This is truly a problem. Even more so if you host your application on a network directory. Loading all the small files takes ages. I really wish there would be a good way to compile the whole application with all the modules into one package once you're ready to release. I really wish the creators of Python would have given such use-cases more consideration. Edit: I'm aware that there are solutions that put everything a program touches into a kind of executable archive. A single file several hundred Megabytes in size. I've tested it. It doesn't really pre-compile the modules. The startup time was exactly the same.

alexhutcheson 8y ago
Check out subpar: https://github.com/google/subpar https://github.com/google/subpar
sametmax 8y ago
Nuikta (http://nuitka.net/ http://nuitka.net/) already does that and much more: - it compiles your program and make it stand alone so you can distribute just the exe - it makes it start faster - it makes it run faster - it's fully independant of the system python. Actually your system doesn't even need a python at all I don't get why it's not used, it's very robust, compatible with 3.6 and on some of my script I get about x4 speed up just on start up alone.
- rhizome 8y ago
  Is it a perl2exe descendant, packing the interpreter into an executable wrapper?
  - sametmax 8y ago
    No it compiles python to C, then compile the C.
- std_throwaway 8y ago
  This is different from the package that I've tested (PyInstaller or py2exe). Is Nuikta compatible with numpy, pickle, etc? I remember that numpy was very problematic with compilers like pypy for a long time.
  - sametmax 8y ago
    In my experience it's easier and more reliable than PyInstaller or Py2exe to use, and cross plateform (but no cross compilation). It doesn't pack python files with an executable. It translates the Python code to C then compiles it. Nuikta supports numpy officially, you can even see it in change logs: http://nuitka.net/posts/nuitka-release-0521.html http://nuitka.net/posts/nuitka-release-0521.html I haven't tried pickle.
  - vram22 8y ago
    >This is different from the package that I've tested (PyInstaller or py2exe). In an ancestor comment you say: > A single file several hundred Megabytes in size. Are both points referring to PyInstaller? Asking because I've tried out PyInstaller with small CLI as well as GUI (wxPython) programs, and the resulting EXEs did not reach near that size, IIRC.
    - std_throwaway 8y ago
      It was around 150..300 MB if I remember right. Admittedly, I do import a lot of modules. Program startup for me is in the seconds, not milliseconds. I maybe could cut it down 50% or even 80% but then that would cost a few weeks or a few months. Having a quick and robust solution to be implemented in a single week to cut down startup times by that amount would be highly preferable.
      - vram22 8y ago
        Interesting. I see what you mean. That tip that someone else mentioned, IIRC, in this thread, to import modules inside functions that use them, could cut down startup time, but only if those functions are not called at startup, only later during the program's run. But it would not change the EXE size, I guess. I wish Python and all other interpreted languages came with a way to build EXEs from the start. It would be great for deployment.
- scrollaway 8y ago
  First time I hear about this, and I've looked for alternatives to cxfreeze and its cousins in the past. Any time I see something like this, I feel like I'm hearing about some homeopathic cancer cure. If Nuitka actually does what it says it does, it's solving a big recurrent problem for the Python community, so why is nobody talking about it?
  - carapace 8y ago
    I collect ideas, especially weird and powerful ideas. I've learned not to try to talk about it because of that question: "If foo is so great, why isn't everybody using it?" It's one of the single greatest frustrations of my life. I don't know. I've never made any progress on it. The best you can say is, "Well, that seems to be human nature." The world is full of "magic beans" and most people seem interested in banging their heads against the wall. (Did you know, you can make a 140hp engine that fits in the volume of two stacked pizza boxes and has only one moving part?) Anyhow, Nuitka is great, it does do all that. And the creator is a freakin' saint for putting up with the way he's been treated by the Python community, is my opinion.
    - sametmax 8y ago
      Saint is the word given he has be working really seriously at it for 8 years, alone, never complaining, and giving away everything without any recognition.
      - carapace 8y ago
        Bingo.
    - sametmax 8y ago
      Also, you should write a blig about those ideas. Or a place where we can share it, but in a non tin foil way / silver bullet way. Eg: after 7 years of having malaria, one tropical disease doctor explained to me that we have been able to cure malaria for years. Generalist doctors usually don't know it because they don't encounter the disease often enough to keep up to date. It's kind of hard pill to swallow given that i always though you had it for life.
- svenklemm 8y ago
  Does nuitka build a static executable or do you still need to supply shared libraries with the executable?
  - sametmax 8y ago
    It builds a static executable if you pass it the --standalone option.
    - jwilk 8y ago
      No, it doesn't. $ echo 'print "Hello world"' > hello.py $ nuitka --standalone hello.py $ ldd hello.dist/hello.exe linux-gate.so.1 (0xf7f94000) libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf7895000) libpython2.7.so.1.0 => /home/jwilk/hello.dist/libpython2.7.so.1.0 (0xf7508000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf732f000) /lib/ld-linux.so.2 (0xf7f96000) libz.so.1 => /home/jwilk/hello.dist/libz.so.1 (0xf7310000) libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf72f1000) libutil.so.1 => /home/jwilk/hello.dist/libutil.so.1 (0xf72ed000) libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf71eb000)
      - sametmax 8y ago
        Ok, it buids a static executable against the python runtime and extensions, including the c ones. It doesn't against libc, lubutil, etc. But aren't those almost always installed ? If it's a real problem on your machines, it's PR time !
- DonbunEf7 8y ago
  Nuitka has not been able to compile any Python code I've written myself. It's not used because it's incredibly limited.
  - std_throwaway 8y ago
    When was the last time you've tried it?
49bc 8y ago
also it’s not static linked, so you need to make sure all of the shared libraries exist on the host, requiring to install a whole bunch of trash.
no_wizard 8y ago
I know this isn’t everyone’s favorite but Cython has a way to convert your python code Into an executable with Python embedded and I bekievr it also Packs your imports Cython is a complicated beast but I feel like it just needs a more friendly wrapper for this to be more widespread. https://stackoverflow.com/questions/22507592/making-an-executable-in-cython https://stackoverflow.com/questions/22507592/making-an-execu... https://github.com/cython/cython/wiki/EmbeddingCython https://github.com/cython/cython/wiki/EmbeddingCython Why Cython isn’t in the stdlib (I think it could easily replace ctypes) is beyond me sometimes
deleted 8y ago
[deleted]
stefan_ 8y ago
I think design choices made in Python simply don't allow for comprehensive ahead of time compilation. For what it's worth, they have recently landed snapshots in Dart that do what you want: https://github.com/dart-lang/sdk/wiki/Snapshots https://github.com/dart-lang/sdk/wiki/Snapshots It's what Flutter uses on iOS since you can't run JITed code; AOT compile it and load it as just another shared library.
eesmith 8y ago
I worked on one Python application that had a startup time problem because it was on a network filesystem with slow metadata/stat times. It took several seconds to start Python. We were able to solve most of the problem by zipping up the Python standard library and the our application. That is, if you look at sys.path you'll see something like: >>> sys.path ['', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', ...] If you zip up the python3.6 directory into python36.zip then it will use that zip file as the source of the standard library, and use the zip directory structure instead of a bunch of stat calls to find the data. This should also include getting access to the pre-compiled byte code. You can also have Python byte-compile all of the .py files in a directory as part of your build/zip process. python -m compileall --help
- a-dub 8y ago
  cool!
- tomc1985 8y ago
  Don't forget find . -type f -name "*.py" -delete right afterwords. Also note calls to imp.load_source need to change to imp.load_compiled, and any .py files references directly in code need to be changed to .pyc (this is with 2.7, not sure about 3.x)
- std_throwaway 8y ago
  Thanks, I will try that!

alexhutcheson 8y ago

Check out subpar: https://github.com/google/subpar https://github.com/google/subpar

sametmax 8y ago

Nuikta (http://nuitka.net/ http://nuitka.net/) already does that and much more: - it compiles your program and make it stand alone so you can distribute just the exe - it makes it start faster - it makes it run faster - it's fully independant of the system python. Actually your system doesn't even need a python at all I don't get why it's not used, it's very robust, compatible with 3.6 and on some of my script I get about x4 speed up just on start up alone.

rhizome 8y ago
Is it a perl2exe descendant, packing the interpreter into an executable wrapper?
- sametmax 8y ago
  No it compiles python to C, then compile the C.
std_throwaway 8y ago
This is different from the package that I've tested (PyInstaller or py2exe). Is Nuikta compatible with numpy, pickle, etc? I remember that numpy was very problematic with compilers like pypy for a long time.
- sametmax 8y ago
  In my experience it's easier and more reliable than PyInstaller or Py2exe to use, and cross plateform (but no cross compilation). It doesn't pack python files with an executable. It translates the Python code to C then compiles it. Nuikta supports numpy officially, you can even see it in change logs: http://nuitka.net/posts/nuitka-release-0521.html http://nuitka.net/posts/nuitka-release-0521.html I haven't tried pickle.
- vram22 8y ago
  >This is different from the package that I've tested (PyInstaller or py2exe). In an ancestor comment you say: > A single file several hundred Megabytes in size. Are both points referring to PyInstaller? Asking because I've tried out PyInstaller with small CLI as well as GUI (wxPython) programs, and the resulting EXEs did not reach near that size, IIRC.
  - std_throwaway 8y ago
    It was around 150..300 MB if I remember right. Admittedly, I do import a lot of modules. Program startup for me is in the seconds, not milliseconds. I maybe could cut it down 50% or even 80% but then that would cost a few weeks or a few months. Having a quick and robust solution to be implemented in a single week to cut down startup times by that amount would be highly preferable.
    - vram22 8y ago
      Interesting. I see what you mean. That tip that someone else mentioned, IIRC, in this thread, to import modules inside functions that use them, could cut down startup time, but only if those functions are not called at startup, only later during the program's run. But it would not change the EXE size, I guess. I wish Python and all other interpreted languages came with a way to build EXEs from the start. It would be great for deployment.
scrollaway 8y ago
First time I hear about this, and I've looked for alternatives to cxfreeze and its cousins in the past. Any time I see something like this, I feel like I'm hearing about some homeopathic cancer cure. If Nuitka actually does what it says it does, it's solving a big recurrent problem for the Python community, so why is nobody talking about it?
- carapace 8y ago
  I collect ideas, especially weird and powerful ideas. I've learned not to try to talk about it because of that question: "If foo is so great, why isn't everybody using it?" It's one of the single greatest frustrations of my life. I don't know. I've never made any progress on it. The best you can say is, "Well, that seems to be human nature." The world is full of "magic beans" and most people seem interested in banging their heads against the wall. (Did you know, you can make a 140hp engine that fits in the volume of two stacked pizza boxes and has only one moving part?) Anyhow, Nuitka is great, it does do all that. And the creator is a freakin' saint for putting up with the way he's been treated by the Python community, is my opinion.
  - sametmax 8y ago
    Saint is the word given he has be working really seriously at it for 8 years, alone, never complaining, and giving away everything without any recognition.
    - carapace 8y ago
      Bingo.
  - sametmax 8y ago
    Also, you should write a blig about those ideas. Or a place where we can share it, but in a non tin foil way / silver bullet way. Eg: after 7 years of having malaria, one tropical disease doctor explained to me that we have been able to cure malaria for years. Generalist doctors usually don't know it because they don't encounter the disease often enough to keep up to date. It's kind of hard pill to swallow given that i always though you had it for life.
svenklemm 8y ago
Does nuitka build a static executable or do you still need to supply shared libraries with the executable?
- sametmax 8y ago
  It builds a static executable if you pass it the --standalone option.
  - jwilk 8y ago
    No, it doesn't. $ echo 'print "Hello world"' > hello.py $ nuitka --standalone hello.py $ ldd hello.dist/hello.exe linux-gate.so.1 (0xf7f94000) libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf7895000) libpython2.7.so.1.0 => /home/jwilk/hello.dist/libpython2.7.so.1.0 (0xf7508000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf732f000) /lib/ld-linux.so.2 (0xf7f96000) libz.so.1 => /home/jwilk/hello.dist/libz.so.1 (0xf7310000) libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf72f1000) libutil.so.1 => /home/jwilk/hello.dist/libutil.so.1 (0xf72ed000) libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xf71eb000)
    - sametmax 8y ago
      Ok, it buids a static executable against the python runtime and extensions, including the c ones. It doesn't against libc, lubutil, etc. But aren't those almost always installed ? If it's a real problem on your machines, it's PR time !
DonbunEf7 8y ago
Nuitka has not been able to compile any Python code I've written myself. It's not used because it's incredibly limited.
- std_throwaway 8y ago
  When was the last time you've tried it?

49bc 8y ago

also it’s not static linked, so you need to make sure all of the shared libraries exist on the host, requiring to install a whole bunch of trash.

no_wizard 8y ago

I know this isn’t everyone’s favorite but Cython has a way to convert your python code Into an executable with Python embedded and I bekievr it also Packs your imports Cython is a complicated beast but I feel like it just needs a more friendly wrapper for this to be more widespread. https://stackoverflow.com/questions/22507592/making-an-executable-in-cython https://stackoverflow.com/questions/22507592/making-an-execu... https://github.com/cython/cython/wiki/EmbeddingCython https://github.com/cython/cython/wiki/EmbeddingCython Why Cython isn’t in the stdlib (I think it could easily replace ctypes) is beyond me sometimes

deleted 8y ago

[deleted]

stefan_ 8y ago

I think design choices made in Python simply don't allow for comprehensive ahead of time compilation. For what it's worth, they have recently landed snapshots in Dart that do what you want: https://github.com/dart-lang/sdk/wiki/Snapshots https://github.com/dart-lang/sdk/wiki/Snapshots It's what Flutter uses on iOS since you can't run JITed code; AOT compile it and load it as just another shared library.

eesmith 8y ago

I worked on one Python application that had a startup time problem because it was on a network filesystem with slow metadata/stat times. It took several seconds to start Python. We were able to solve most of the problem by zipping up the Python standard library and the our application. That is, if you look at sys.path you'll see something like: >>> sys.path ['', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', ...] If you zip up the python3.6 directory into python36.zip then it will use that zip file as the source of the standard library, and use the zip directory structure instead of a bunch of stat calls to find the data. This should also include getting access to the pre-compiled byte code. You can also have Python byte-compile all of the .py files in a directory as part of your build/zip process. python -m compileall --help

a-dub 8y ago
cool!
tomc1985 8y ago
Don't forget find . -type f -name "*.py" -delete right afterwords. Also note calls to imp.load_source need to change to imp.load_compiled, and any .py files references directly in code need to be changed to .pyc (this is with 2.7, not sure about 3.x)
std_throwaway 8y ago
Thanks, I will try that!