35 ms·

PyOxidizer 0.8

133 points by dochtman 6y ago

nickcw 6y ago
If you are wondering what PyOxidizer is then here is a good explanation: https://pyoxidizer.readthedocs.io/en/v0.8.0/overview.html https://pyoxidizer.readthedocs.io/en/v0.8.0/overview.html From that page: From a very high level, PyOxidizer is a tool for packaging and distributing Python applications. The over-arching goal of PyOxidizer is to make this (often complex) problem space simple so application maintainers can focus on building quality applications instead of toiling with build systems and packaging tools. On a lower, more technical level, PyOxidizer has a command line tool - pyoxidizer - that is capable of building binaries (executables or libraries) that embed a fully-functional Python interpreter plus Python extensions and modules in a single binary. Binaries produced with PyOxidizer are highly portable and can work on nearly every system without any special requirements like containers, FUSE filesystems, or even temporary directory access. On Linux, PyOxidizer can produce executables that are fully statically linked and don’t even support dynamic loading. The Oxidizer part of the name comes from Rust: binaries built with PyOxidizer are compiled from Rust and Rust code is responsible for managing the embedded Python interpreter and all its operations. But the existence of Rust should be invisible to many users, much like the fact that CPython (the official Python distribution available from www.python.org) is implemented in C. Rust is simply a tool to achieve an end goal (albeit a rather effective and powerful tool).
- formerly_proven 6y ago
  So does this use a custom build of CPython linked against a different libc that's part of the package? That has been IME one of the major issues with distributing binary packages of Python apps on Linux, because code built using glibc tends to use recent versions of common symbols, so in practice you can't run them on a system with a glibc older than the build system's glibc, meaning you have to build on some ancient Linux where everything is a pain. Edit: Sounds like it: > PyOxidizer uses custom Python distributions that are built in such a way that they are highly portable across machines.
  - nickcw 6y ago
    The page states that a static build is possible so you'll get a binary you can distribute to any machine at that point.
    - quietbritishjim 6y ago
      Without any more clarification that doesn't quite answer the question because libc is such a special beast. Typically (but maybe not always) "static build" on Linux really means "all libraries are statically linked except libc which is dynamically linked". To make something very portable across Linux distributions, rather than statically linking libc you would dynamically link against an old version of it, as newer versions are always backwards compatible with older versions.
  - oefrha 6y ago
    > meaning you have to build on some ancient Linux where everything is a pain. Well, PyPA provides Docker images for manylinux﹡[1] and manylinux2014 isn’t that ancient, I found it mostly tolerable in my experience of building wheels for various computational packages of my own (depending on gmp, etc.). In fact even manylinux1 and manylinux2010 have been okay for me, but that of course depends on what libraries you need. The future of manylinux is defined in PEP 600[2], though I’m not sure about what has happened implementation-wise regarding the new scheme. [1] https://github.com/pypa/manylinux https://github.com/pypa/manylinux [2] https://www.python.org/dev/peps/pep-0600/ https://www.python.org/dev/peps/pep-0600/
  - nethunters 6y ago
    > Specifying --target x86_64-unknown-linux-musl will cause PyOxidizer to use a Python distribution built against musl libc as well as tell Rust to target musl on Linux. [1] https://pyoxidizer.readthedocs.io/en/stable/packaging_static_linking.html https://pyoxidizer.readthedocs.io/en/stable/packaging_static...
- deleted 6y ago
  [deleted]
artembugara 6y ago
How does it compare to Poetry? https://python-poetry.org/ https://python-poetry.org/
- wlkr 6y ago
  From what I gather, the focus of PyOxidizer is on enabling you to create executable Python binaries which embed a Python interpreter rather than providing Python packaging and dependency management.
- epage 6y ago
  Sounds like PyInstaller is the closer comparison https://www.pyinstaller.org/ https://www.pyinstaller.org/
- kissgyorgy 6y ago
  These two are totally different tools, not even comparable. With Poetry, you handle dependencies, just like with npm, cargo or pip itself. PyOxidizer can build a standalone binary which doesn't need an installed Python interpreter, so you need both.
  - artembugara 6y ago
    thx, I misunderstood what it does
xvilka 6y ago
I initially thought it's Python to Rust conversion tool.
- Barrin92 6y ago
  while not being a full conversion tool it actually also can function as a bridge between python and rust, from the docs "While solving packaging and distribution problems is the primary goal of PyOxidizer, a side-effect of solving that problem with Rust is that PyOxidizer can serve as a bridge between these two languages. PyOxidizer can be used to easily add a Python interpreter to any Rust project. But the opposite is also true: PyOxidizer can also be used to add Rust to Python. Using PyOxidizer, you could bootstrap a new Rust project which contains an embedded version of Python and your application. Initially, your project is a few lines of Rust that instantiates a Python interpreter and runs Python code. Over time, functionality could be (re)written in Rust and your previously Python-only project could leverage Rust and its diverse ecosystem. Since PyOxidizer abstracts the Python interpreter away, this could all be invisible to end-users: you could rewrite an application from Python to Rust and people may not even know because they never see a libpython, .py files, etc."
href 6y ago
I've been using PyOxidizer sind 0.7 for a personal CLI project and I really like it. Apart from being able to ship a single executable, it also speeds up the startup time of my CLI, as it doesn't need to scan loads of Python files during startup. Really great stuff!
- fastball 6y ago
  How big are your binaries?
  - kissgyorgy 6y ago
    I packaged with a very simple script with a couple of dependencies with 0.7 and it's ~84 Mb: https://github.com/kissgyorgy/cloudflare-dyndns https://github.com/kissgyorgy/cloudflare-dyndns
    - kangalioo 6y ago
      Ooof that's a lot. PyInstaller is considerably lower than that (~60 MB for a full desktop application including PyQt, NumPy and a plotting toolkit)
  - href 6y ago
    A simple CLI with only a few commands and not many dependencies is 56M here.
rusticpenn 6y ago
How does this compare to pyinstaller or nuitika?
- anentropic 6y ago
  Nuitka compiles Python to C, so it can make your code run faster, but you're no longer using the tried and tested CPython interpreter so you may run into Nuitka-specific issues (I haven't used it and don't know how common they are, I assume it's usable though) PyOxidizer and pyinstaller are rather bundlers that build a CPython interpreter and all your dependencies into a single binary. There is also https://github.com/pantsbuild/pex https://github.com/pantsbuild/pex which is a bit like jar files for Java, but it also bundles your code, dependencies and an interpreter into a single executable file.
  - dr_zoidberg 6y ago
    Nuitka compiles Python to C using the Python C-library (something Cython is also capable of), with a focus on portability. Nuitkas main goal is not speed (compared to Cython), but the ability to output an executable binary (plus a bunch of dynamic libraries) needed to run your program in a portable, "interpreter independent" way. From the "What is Nuitka" page[0]: > It translates the Python into a C program that then is linked against libpython to execute in the same way as CPython does, in a very compatible way. Once they managed full compatibility (I think it was in 0.4?) then they started to focus on optimizing and translating even more things to C (again, keeping compatibility at 100% -- so you shouldn't event run into Nuitka-specific issues). I've used it to deploy a few custom triage programs in the field for DFIR operations where we'd want to run a few specific things from portable drives without installing anything in the triaged computer. [0] http://nuitka.net/pages/overview.html http://nuitka.net/pages/overview.html
    - Rotareti 6y ago
      Nuitka is great! I use it to compile some of the command line tools which I have written. Command lines tools compiled with Nuitka have a much better startup time. Nuitka also has a single file binary target in the works. Compiled single file binaries for python. Wohoo!
alphachloride 6y ago
PyOxidizer generates standalone binaries for python. It requires the rust (and optionally C) toolchain to be installed first. I think requiring another language development environment set up to package code is a big turn-off. A friendlier approach is either to have a package in the same language so it interpolates with the ecosystem, or have a standalone binary which doesn't need dependencies. At this point, I look at it as a python distribution tool for rust developers.

133 points by dochtman 6y ago

If you are wondering what PyOxidizer is then here is a good explanation: https://pyoxidizer.readthedocs.io/en/v0.8.0/overview.html https://pyoxidizer.readthedocs.io/en/v0.8.0/overview.html From that page: From a very high level, PyOxidizer is a tool for packaging and distributing Python applications. The over-arching goal of PyOxidizer is to make this (often complex) problem space simple so application maintainers can focus on building quality applications instead of toiling with build systems and packaging tools. On a lower, more technical level, PyOxidizer has a command line tool - pyoxidizer - that is capable of building binaries (executables or libraries) that embed a fully-functional Python interpreter plus Python extensions and modules in a single binary. Binaries produced with PyOxidizer are highly portable and can work on nearly every system without any special requirements like containers, FUSE filesystems, or even temporary directory access. On Linux, PyOxidizer can produce executables that are fully statically linked and don’t even support dynamic loading. The Oxidizer part of the name comes from Rust: binaries built with PyOxidizer are compiled from Rust and Rust code is responsible for managing the embedded Python interpreter and all its operations. But the existence of Rust should be invisible to many users, much like the fact that CPython (the official Python distribution available from www.python.org) is implemented in C. Rust is simply a tool to achieve an end goal (albeit a rather effective and powerful tool).

formerly_proven 6y ago
So does this use a custom build of CPython linked against a different libc that's part of the package? That has been IME one of the major issues with distributing binary packages of Python apps on Linux, because code built using glibc tends to use recent versions of common symbols, so in practice you can't run them on a system with a glibc older than the build system's glibc, meaning you have to build on some ancient Linux where everything is a pain. Edit: Sounds like it: > PyOxidizer uses custom Python distributions that are built in such a way that they are highly portable across machines.
- nickcw 6y ago
  The page states that a static build is possible so you'll get a binary you can distribute to any machine at that point.
  - quietbritishjim 6y ago
    Without any more clarification that doesn't quite answer the question because libc is such a special beast. Typically (but maybe not always) "static build" on Linux really means "all libraries are statically linked except libc which is dynamically linked". To make something very portable across Linux distributions, rather than statically linking libc you would dynamically link against an old version of it, as newer versions are always backwards compatible with older versions.
- oefrha 6y ago
  > meaning you have to build on some ancient Linux where everything is a pain. Well, PyPA provides Docker images for manylinux﹡[1] and manylinux2014 isn’t that ancient, I found it mostly tolerable in my experience of building wheels for various computational packages of my own (depending on gmp, etc.). In fact even manylinux1 and manylinux2010 have been okay for me, but that of course depends on what libraries you need. The future of manylinux is defined in PEP 600[2], though I’m not sure about what has happened implementation-wise regarding the new scheme. [1] https://github.com/pypa/manylinux https://github.com/pypa/manylinux [2] https://www.python.org/dev/peps/pep-0600/ https://www.python.org/dev/peps/pep-0600/
- nethunters 6y ago
  > Specifying --target x86_64-unknown-linux-musl will cause PyOxidizer to use a Python distribution built against musl libc as well as tell Rust to target musl on Linux. [1] https://pyoxidizer.readthedocs.io/en/stable/packaging_static_linking.html https://pyoxidizer.readthedocs.io/en/stable/packaging_static...
deleted 6y ago
[deleted]

artembugara 6y ago

How does it compare to Poetry? https://python-poetry.org/ https://python-poetry.org/

wlkr 6y ago
From what I gather, the focus of PyOxidizer is on enabling you to create executable Python binaries which embed a Python interpreter rather than providing Python packaging and dependency management.
epage 6y ago
Sounds like PyInstaller is the closer comparison https://www.pyinstaller.org/ https://www.pyinstaller.org/
kissgyorgy 6y ago
These two are totally different tools, not even comparable. With Poetry, you handle dependencies, just like with npm, cargo or pip itself. PyOxidizer can build a standalone binary which doesn't need an installed Python interpreter, so you need both.
- artembugara 6y ago
  thx, I misunderstood what it does

xvilka 6y ago

I initially thought it's Python to Rust conversion tool.

Barrin92 6y ago
while not being a full conversion tool it actually also can function as a bridge between python and rust, from the docs "While solving packaging and distribution problems is the primary goal of PyOxidizer, a side-effect of solving that problem with Rust is that PyOxidizer can serve as a bridge between these two languages. PyOxidizer can be used to easily add a Python interpreter to any Rust project. But the opposite is also true: PyOxidizer can also be used to add Rust to Python. Using PyOxidizer, you could bootstrap a new Rust project which contains an embedded version of Python and your application. Initially, your project is a few lines of Rust that instantiates a Python interpreter and runs Python code. Over time, functionality could be (re)written in Rust and your previously Python-only project could leverage Rust and its diverse ecosystem. Since PyOxidizer abstracts the Python interpreter away, this could all be invisible to end-users: you could rewrite an application from Python to Rust and people may not even know because they never see a libpython, .py files, etc."

href 6y ago

I've been using PyOxidizer sind 0.7 for a personal CLI project and I really like it. Apart from being able to ship a single executable, it also speeds up the startup time of my CLI, as it doesn't need to scan loads of Python files during startup. Really great stuff!

fastball 6y ago
How big are your binaries?
- kissgyorgy 6y ago
  I packaged with a very simple script with a couple of dependencies with 0.7 and it's ~84 Mb: https://github.com/kissgyorgy/cloudflare-dyndns https://github.com/kissgyorgy/cloudflare-dyndns
  - kangalioo 6y ago
    Ooof that's a lot. PyInstaller is considerably lower than that (~60 MB for a full desktop application including PyQt, NumPy and a plotting toolkit)
- href 6y ago
  A simple CLI with only a few commands and not many dependencies is 56M here.

rusticpenn 6y ago

How does this compare to pyinstaller or nuitika?

anentropic 6y ago
Nuitka compiles Python to C, so it can make your code run faster, but you're no longer using the tried and tested CPython interpreter so you may run into Nuitka-specific issues (I haven't used it and don't know how common they are, I assume it's usable though) PyOxidizer and pyinstaller are rather bundlers that build a CPython interpreter and all your dependencies into a single binary. There is also https://github.com/pantsbuild/pex https://github.com/pantsbuild/pex which is a bit like jar files for Java, but it also bundles your code, dependencies and an interpreter into a single executable file.
- dr_zoidberg 6y ago
  Nuitka compiles Python to C using the Python C-library (something Cython is also capable of), with a focus on portability. Nuitkas main goal is not speed (compared to Cython), but the ability to output an executable binary (plus a bunch of dynamic libraries) needed to run your program in a portable, "interpreter independent" way. From the "What is Nuitka" page[0]: > It translates the Python into a C program that then is linked against libpython to execute in the same way as CPython does, in a very compatible way. Once they managed full compatibility (I think it was in 0.4?) then they started to focus on optimizing and translating even more things to C (again, keeping compatibility at 100% -- so you shouldn't event run into Nuitka-specific issues). I've used it to deploy a few custom triage programs in the field for DFIR operations where we'd want to run a few specific things from portable drives without installing anything in the triaged computer. [0] http://nuitka.net/pages/overview.html http://nuitka.net/pages/overview.html
  - Rotareti 6y ago
    Nuitka is great! I use it to compile some of the command line tools which I have written. Command lines tools compiled with Nuitka have a much better startup time. Nuitka also has a single file binary target in the works. Compiled single file binaries for python. Wohoo!

alphachloride 6y ago

PyOxidizer generates standalone binaries for python. It requires the rust (and optionally C) toolchain to be installed first. I think requiring another language development environment set up to package code is a big turn-off. A friendlier approach is either to have a package in the same language so it interpolates with the ecosystem, or have a standalone binary which doesn't need dependencies. At this point, I look at it as a python distribution tool for rust developers.