11 ms·

Today I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go.

Today I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. Or at least it's hell to debug when it goes wrong. I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit.

rtpg 6y ago
PyOxidizer might be of interest to you.
- haberman 6y ago
  Yes, that looks like exactly what I had in mind. Thanks for the reference.
- shoo 6y ago
  Around a decade ago I worked for a place that packaged and shipped python applications to windows desktops using a similar bundling tool, py2exe. It worked pretty fine. Pretty much just unzips into a directory: there's your self contained python app hiding inside an exe and some associated data files/dlls in the directory. We still used wheel archives etc for managing some python dependencies during our dev and build process, but it was never a complete solution: some python packages depend upon non python shared libraries so you need to install them using an operating system level package manager. Then when shipping the software to users you cannot assume the user has a working version of python, or even if they do you don't want to increase your support burden by having your application running inside some arbitrary python environment.
- pansa2 6y ago
  Thanks, I hadn’t heard of PyOxidizer before. Looks like a significant improvement over tools like PyInstaller and Shiv, which produce self-extracting executables that are slow and/or leave files hanging around after running.
  - BiteCode_dev 6y ago
    There are trade off to anything. E.G: - shiv leave files hanging around, but you can address those files directly. With PyOxidize, you will need to use https://pyoxidizer.readthedocs.io/en/stable/config_api.html#config-file-manifest https://pyoxidizer.readthedocs.io/en/stable/config_api.html#... a way to read a non Python files. Because most projects don't know this and just use open(), they won't work out of the box and you may need to monkey patch open(). - pyoxidizer requires a compiler, which is easy on linux, but a higher requirement on windows. Compared to shiv, which is just a pip install away. There is no perfect solution yet, because you have really often many things to balance: bringing in the python vm or not, allow compiled extensions or not, provide support for open() or not, etc.
m4rtink 6y ago
Its fun and games until you find out one of the things you bundled into your static binary has a remote code execution CVE that is being actively exploited in the wild. Good luck tracking down all the artifacts and making sure they are rebuild with the patched version - till the time next CVE hits and you can do it all again for every static binary bundling the affected component.
- marcinzm 6y ago
  I mean, containers are basically just that and people seem to like them just fine.
  - ethbro 6y ago
    People seem to like them just fine != People are patching CVEs
    - marcinzm 6y ago
      That's a question of process and tooling. Plenty of software, OSS and proprietary, which scans containers for vulnerabilities and sends out alerts. There's also dependabot that automatically creates PRs when dependencies get new versions. Sure people may not use the tools they have but then again people also don't upgrade their OS packages or maven packages. So for those people packaging everything doesn't hurt any more since they've got rampant CVEs already.
  - ClumsyPilot 6y ago
    Exactly, container is a way to package all these disorganised, chaotic snowflake languages into something you can actually run in a standard way. It full fills the function of an executable, but at a higher level of abstraction. We've come full circle again.
  - vmchale 6y ago
    Containers solve a different problem.
- closeparen 6y ago
u801e 6y ago
> Today I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. Most python packages I have seen will put their python source in the python site-packages directory. Where were the source files you were trying to load located on the file system?
est 6y ago
> Trying to load source files from all over the filesystem at runtime is hell > "static" binaries that package everything together into a single, self-contained unit. Ummm, except debugging "static" binaries without symbol file is hell. If there's a .py source somewhere on the disk, I can at least try edit the source file from the package and debug internal private variables as last resort.
- josefx 6y ago
  Python can execute zipped files and attaching a zip to an executable is trivial. With a decent zip tool you also should be able to edit the zipped source files directly.
Spiritus 6y ago
I’m curious. Can you explain the problems you were having? And how they would’ve been solved by a static binary. > Trying to load source files from all over the filesystem at runtime is hell. This sounds like a mess unrelated to Python. Why are you trying to load source files from "all over the filesystem" at runtime? > I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. Compiling source files from all over the filesystem would be equally annoying.
- IshKebab 6y ago
  Assuming you are genuinely asking... Python programs are almost always run from source, and they usually import third party libraries. Those libraries are installed by Pip... somewhere on your system. And then Python has to find it somehow at runtime. That relies on environment variables being set correctly, and works differently on different systems and is basically a huge mess even before you consider things like virtualenv and the fact that non-Linux systems usually have multiple copies of Python installed (and Python 2 and 3!). It's basically a huge mess. There's even an XKCD about it. Consider the alternative: Go compiles programs to a single statically linked executable with no dependencies. You literally just copy one file to your target machine and run it. It basically can't fail. That's part of the reason Go is so popular for server stuff. And it's not just because Python isn't compiled. Other scripting languages handle this much better. Even JavaScript - for all the hate node_modules gets for being enormous, at least it works reliably!
  - Spiritus 6y ago
    Yes, I was genuinely asking. I'm very familiar with how the Python ecosystem works (I've been using it professionally for the past 8 years or so), and I disagree with your assessment. At least not when comparing to other dynamic languages. > Those libraries are installed by Pip... somewhere on your system. They go into your virtualenv. > And then Python has to find it somehow at runtime. If you use a virtualenv, this works 99.9% of the time. > That relies on environment variables being set correctly What environment variables? Just use `./venv/bin/python` directly without environment variables. > a huge mess even before you consider things like virtualenv There's nothing to consider, always use a virtualenv. It's the same for thing for Node, except it implicitly handles it for you. > and the fact that non-Linux systems usually have multiple copies of Python installed (and Python 2 and 3!). How is that a problem? Just pick the interpreter you want to use when creating the virtualenv: virtualenv -p /usr/bin/python2 venv virtualenv -p python3 venv virtualenv -p python3.7 venv > Consider the alternative: Go compiles programs to a single statically linked executable with no dependencies. You literally just copy one file to your target machine and run it. It basically can't fail. That's part of the reason Go is so popular for server stuff. I agree that Go got it mostly right and it just works, except for that fact that it didn't even have a package manager for like 10 years so pinning dependencies was impossible unless you forked repositories. > Even JavaScript - for all the hate node_modules gets for being enormous, at least it works reliably! In what way is Python + virtualenv + requirements.txt less reliable?
jensgk 6y ago
> I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. Isn't that why Docker is so popular? If we had Python static binaries (or the ability to have several versions of the same Python library installed) would we use Docker so much?
- agumonkey 6y ago
  and docker is (was) written in go who kinda put more light on static, the circle is complete
gorgoiler 6y ago
In the past week my coding colleague (whose age is only just into double digits) and I have had to: * insert lines of code into asciidoctor Ruby to debug how the command line interface differs from the library interface; * read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!); * rummage through Python’s PY Sequence List to answer the question “what kind of sort does Python actually use?”. The problem with shipping binaries is it lowers the standard for being able to build from source. If a package is easier to ship as a binary because the source is hard to distribute reliably, we give up hope of end users being able to modify and build their own versions. This is obviously bad for freedom. It’s also hard for debugging and learning.
- executesorder66 6y ago
  You have a 10 year old coding colleague?
  - heavenlyblue 6y ago
    The answer to “which sorting algorithm python uses” is the first result in Google. The third is official python docs.
    - gorgoiler 6y ago
      The context was one where we were learning about how things work, rather than wanting to just know the answer per se. I always _try_ to show how one can answer problems from first principles before resorting to “go look it up on stack overflow”. It boosts ones sense of usefulness as a teacher if one can occasionally give more in depth answers as teachable moments. For one thing, demonstrating deeper understand is how you score high marks in exams.
      - heavenlyblue 6y ago
        I don't understand what you mean by "first principles". Open up Python's source code and check - this is first principles-first hand. As a matter of fact I am going to go as far as saying that reading source code is the _only_ way to actually be any good in software engineering because it's the only source of truth that never gets out of sync with the de-facto. Did you expect to find a blog post about it (you can find it on the first page of Google search I mentioned)? Or a YouTube video (didn't pay attention to that)? Books are reasonably good but usually get out of sync quite fast. And going back to first principles: the reason most of the literature may not mention what sorting algorithm Python uses is because a) it's irrelevant as a first principle knowledge b) it gets quickly out of sync.
oefrha 6y ago
zipapp? It’s not like a self-contained solution didn’t exist for a long time. (If you mean the Python installation also needs to be contained in that self-contained binary — sorry you chose a scripting language.) And yesterday I spent at least an hour fighting with cargo build. Without knowing exactly why we fought both our anecdotes are quite useless.
kkapelon 6y ago
>I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. You mean Docker containers? :-)
- drran 6y ago
  VMWare images or even hardware images: it's easier to buy new phone than to patch old one.
mturilin 6y ago
Deno (https://deno.land https://deno.land) solves this problem very well for JavaScript. I’d like so see more languages adopting “dependencies are URLs” approach. No more package hell.
- julianwachholz 6y ago
  Like GoLang did?
- aequitas 6y ago
  How does this solve the problem? As I can tell it replaces a centralized dependencies (requirements.txt) containing references to a known and trusted registry (pypi.org) with url's in all files across your project (and 3rd party library's files). Making maitenance only harder and security updates even more so. And making it harder to debug things as you now keep in mind that multiple versions of the same packages can be used in your source. Also, requirements.txt already support's urls if you want your sources from other places. For me, most of the pain with the Python packaging went away after I started using Pip-tools[0]. It's just a simple utility to add lockfile capabilities to Pip. Nothing new to learn, no new filosophies or paradigm's. No PEP waiting to be adopted by everyone. Just good old requirements.txt + Pip. [0] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
atoav 6y ago
The best thing would be if python had something like a cross between pip and python-poetry, but better integrated with the interpreter. Python has the most unpythonic package experience ever. As elegant as the language is, packaging is a convoluted nightmare. It still might work for some project, but certainly is no fit for the average python user.
- necrotic_comp 6y ago
  I started using the pipenv this weekend and it seems like it's pretty good for at least maintaining an environment within a project. I found it friendly at least. That being said, I'll admit I don't know much about what it means to be pythonic. What do you think the drawbacks are to pipenv-type approach and what would be more pythonic in your mind ?
  - atoav 6y ago
    What I mean by pythonic is the "batteries included" paradigma: if you want to — say — quickly script something up that does some time calculations you just go and import datetime. There is no need to research what datetime libraries are ok, download and install dependecies etc. You are able to go and (with the included batteries) solve your problem. I wish I could just as easily say: ahh this script I made should become a package that I can reuse on my server. But then the server has Python 3.6 while your script has 3.8, so you end up installing another python, painstakingly watch not to install over the old one etc. When installing modules you have to do the same or set up a venv/pipenv. All along the road there are possible ways to shoot yourself into the leg. Meanwhile in Rust you do a cargo new foobar, work on it, add dependecies to the cargo.toml, copy it to the server, build a binary with cargo build --release and copy the binary into the PATH. You spent zero brainpower on not breaking things and had capacity to think about other stuff. The closest thing we have in those terms in pythonland is poetry, which is a very good start. But this should be part of the language, not something one has to do extra.
    - necrotic_comp 6y ago
      Got it - that makes sense. I actually ended up with a smattering of the version problems when I was working with pipenv for the first time, and figured it was my fault in some way because I hadn't set up the "correct" python version on my system. That's somewhat disheartening to hear that it's a consistent problem. Thanks for the insight and clarification.
ashishb 6y ago
Even more so in a world where storage is cheap.
andarleen 6y ago
Just use a language with proper package management and OOP, such as PHP or NodeJS. Python is great for machine learning, data science, hobbies and one off scripts, and learning basic programming.
jillesvangurp 6y ago
Agreed. I'd go one step further and call that docker container. If it needs to run on a server, the preferred format these days is a docker container. I too spend a bit of time fighting pipenv lately and generally trying to figure out the various ways python tries to virtualize barfing all over the filesystem by default. IMHO python, node.js, ruby, and other scripting languages share a historical confusion of concerns related to development tooling and attempts to counter the unfortunate choice of installing dependencies globally that tend to leak to the server side in inappropriate ways. With Docker.
- ClumsyPilot 6y ago
  Except we solved the problem at a higher level of abstraction, creating a more complex system. If every language used LLVM (for example) we'd have almost the same effect. There are other things docker can do (like mapping volumes) but I would be much happier if we solved it at the level of the executable.
  - jillesvangurp 6y ago
    Your happiness and practical reality are two things. Docker is there right now and sort of the de-facto way of getting stuff on a server in basically all projects I've been on in the last five years. Programs are more than just their binaries. Most programs also expect non binary content like config files, data files, shell scripts, symbolic links, etc. Packaging that up in a sane way is basically what docker is good at.
mintplant 6y ago
Just use Poetry. I haven't fought with Python packaging in a long time.
- ghshephard 6y ago
  I see a lot of references to Poetry, but I've yet to see any uptake with the "professional" python developers that I know - which could be entirely to inertia, rather than "best tool for the job" - most of them see to be still in the pip/virtualenv/requirements.txt world for all of their development/docker/k8s deployments. I wonder if there have been any surveys on poetry, and whether it's trajectory (presumably exponential) suggests that it's going to become a dominant package management utility in the next 10 years or so?
nh2 6y ago
You may enjoy this StackOverflow answer on The Return of Static Linking, summarising advancements on the topic: https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead/56771892#56771892 https://stackoverflow.com/questions/3430400/linux-static-lin...
vmchale 6y ago
> The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. One need not depend on the filesystem for such things. And as bad as python packaging is, others get it better. One can have less-than-static-binaries without being quite that bad. > I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. Plenty of problems with that too!
blaisio 6y ago
I do prefer static self contained binaries, but I think it's important to note that Python's package management system is horrible, and making self contained binaries is not the only way to fix it.
developer2 6y ago
IMO: Python is a garbage language, and should never be used for professional applications. Try to write an app whose very first block is a try/except for KeyboardInterrupt, with or without an if __name__ == '__main__' block. Now run the program a couple dozen times, and press ctrl+c as fast as you can. You will wind up with a debug-level stack trace dump for an uncaught KeyboardInterrupt exception from the core libraries that is thrown before your first line of userland code is parsed or executed. Python runs so much internal setup code before your first line of code is even considered. It's literally impossible to write a Python program that doesn't dump debug garbage to its output, without writing a separate bash script that uses trap to proxy SIGINT and SIGTERM to the running process. Python is a fucking joke in my eyes; and this is before discussing the way async/await was implemented by gutting the core of the language in a way that makes zero fucking sense. Python is a total… fucking… joke. It has some nice features when all you care is about the syntax and ease of development, but the fact is it's a terribly broken language. Use C. Or C++. Or Rust. Or Go. Or Ruby, or literally no joke… even PHP. Python is literally THAT HORRIBLE once you understand its internals. It absolutely disgusts me to say, but PHP with its single-threaded model is put together better than Python. tldr; Do not use Python. It makes certain tasks seem easy, but all it truly does is make you look like an idiot developer. Pick literally any other language, as Python has screwed up too many basic concepts (related to POSIX and expectations from C) to be taken seriously.
- shadeslayer_ 6y ago
  comments like this make me want to ask for a downvote button.
  - developer2 6y ago
    So you are capable of writing a script in Python that always catches SIGINT and SIGTERM, exiting gracefully without an exception stack being thrown by core code? No? Right… no. Python is literally the only language I've used that cannot do this. Even PHP can do it. For real, PHP is overall a shit language (it has zero async/threading support), but it still handles signals according to POSIX (eg. it doesn't override signal handlers unless userland code does it, preserving default SIG_DFL behavior), unlike Python which breaks that simple contract. The "KeyboardInterrrupt" exception being thrown by core code before userland code is parsed and executed is such an extreme bastardization of POSIX standards that Python is simply, inexcusably, broken. I stand by my opinion that Python is not well-suited to applications written by a professional entity, intended to be shipped to 3rd-party clients. Python plays too loose with the fundamentals provided by the operating system to be a professional language.

rtpg 6y ago

PyOxidizer might be of interest to you.

haberman 6y ago
Yes, that looks like exactly what I had in mind. Thanks for the reference.
shoo 6y ago
Around a decade ago I worked for a place that packaged and shipped python applications to windows desktops using a similar bundling tool, py2exe. It worked pretty fine. Pretty much just unzips into a directory: there's your self contained python app hiding inside an exe and some associated data files/dlls in the directory. We still used wheel archives etc for managing some python dependencies during our dev and build process, but it was never a complete solution: some python packages depend upon non python shared libraries so you need to install them using an operating system level package manager. Then when shipping the software to users you cannot assume the user has a working version of python, or even if they do you don't want to increase your support burden by having your application running inside some arbitrary python environment.
pansa2 6y ago
Thanks, I hadn’t heard of PyOxidizer before. Looks like a significant improvement over tools like PyInstaller and Shiv, which produce self-extracting executables that are slow and/or leave files hanging around after running.
- BiteCode_dev 6y ago
  There are trade off to anything. E.G: - shiv leave files hanging around, but you can address those files directly. With PyOxidize, you will need to use https://pyoxidizer.readthedocs.io/en/stable/config_api.html#config-file-manifest https://pyoxidizer.readthedocs.io/en/stable/config_api.html#... a way to read a non Python files. Because most projects don't know this and just use open(), they won't work out of the box and you may need to monkey patch open(). - pyoxidizer requires a compiler, which is easy on linux, but a higher requirement on windows. Compared to shiv, which is just a pip install away. There is no perfect solution yet, because you have really often many things to balance: bringing in the python vm or not, allow compiled extensions or not, provide support for open() or not, etc.

m4rtink 6y ago

Its fun and games until you find out one of the things you bundled into your static binary has a remote code execution CVE that is being actively exploited in the wild. Good luck tracking down all the artifacts and making sure they are rebuild with the patched version - till the time next CVE hits and you can do it all again for every static binary bundling the affected component.

marcinzm 6y ago
I mean, containers are basically just that and people seem to like them just fine.
- ethbro 6y ago
  People seem to like them just fine != People are patching CVEs
  - marcinzm 6y ago
    That's a question of process and tooling. Plenty of software, OSS and proprietary, which scans containers for vulnerabilities and sends out alerts. There's also dependabot that automatically creates PRs when dependencies get new versions. Sure people may not use the tools they have but then again people also don't upgrade their OS packages or maven packages. So for those people packaging everything doesn't hurt any more since they've got rampant CVEs already.
- ClumsyPilot 6y ago
  Exactly, container is a way to package all these disorganised, chaotic snowflake languages into something you can actually run in a standard way. It full fills the function of an executable, but at a higher level of abstraction. We've come full circle again.
- vmchale 6y ago
  Containers solve a different problem.
closeparen 6y ago

u801e 6y ago

> Today I spent at least an hour fighting with Python packaging. The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. Most python packages I have seen will put their python source in the python site-packages directory. Where were the source files you were trying to load located on the file system?

est 6y ago

> Trying to load source files from all over the filesystem at runtime is hell > "static" binaries that package everything together into a single, self-contained unit. Ummm, except debugging "static" binaries without symbol file is hell. If there's a .py source somewhere on the disk, I can at least try edit the source file from the package and debug internal private variables as last resort.

josefx 6y ago
Python can execute zipped files and attaching a zip to an executable is trivial. With a decent zip tool you also should be able to edit the zipped source files directly.

Spiritus 6y ago

I’m curious. Can you explain the problems you were having? And how they would’ve been solved by a static binary. > Trying to load source files from all over the filesystem at runtime is hell. This sounds like a mess unrelated to Python. Why are you trying to load source files from "all over the filesystem" at runtime? > I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. Compiling source files from all over the filesystem would be equally annoying.

IshKebab 6y ago
Assuming you are genuinely asking... Python programs are almost always run from source, and they usually import third party libraries. Those libraries are installed by Pip... somewhere on your system. And then Python has to find it somehow at runtime. That relies on environment variables being set correctly, and works differently on different systems and is basically a huge mess even before you consider things like virtualenv and the fact that non-Linux systems usually have multiple copies of Python installed (and Python 2 and 3!). It's basically a huge mess. There's even an XKCD about it. Consider the alternative: Go compiles programs to a single statically linked executable with no dependencies. You literally just copy one file to your target machine and run it. It basically can't fail. That's part of the reason Go is so popular for server stuff. And it's not just because Python isn't compiled. Other scripting languages handle this much better. Even JavaScript - for all the hate node_modules gets for being enormous, at least it works reliably!
- Spiritus 6y ago
  Yes, I was genuinely asking. I'm very familiar with how the Python ecosystem works (I've been using it professionally for the past 8 years or so), and I disagree with your assessment. At least not when comparing to other dynamic languages. > Those libraries are installed by Pip... somewhere on your system. They go into your virtualenv. > And then Python has to find it somehow at runtime. If you use a virtualenv, this works 99.9% of the time. > That relies on environment variables being set correctly What environment variables? Just use `./venv/bin/python` directly without environment variables. > a huge mess even before you consider things like virtualenv There's nothing to consider, always use a virtualenv. It's the same for thing for Node, except it implicitly handles it for you. > and the fact that non-Linux systems usually have multiple copies of Python installed (and Python 2 and 3!). How is that a problem? Just pick the interpreter you want to use when creating the virtualenv: virtualenv -p /usr/bin/python2 venv virtualenv -p python3 venv virtualenv -p python3.7 venv > Consider the alternative: Go compiles programs to a single statically linked executable with no dependencies. You literally just copy one file to your target machine and run it. It basically can't fail. That's part of the reason Go is so popular for server stuff. I agree that Go got it mostly right and it just works, except for that fact that it didn't even have a package manager for like 10 years so pinning dependencies was impossible unless you forked repositories. > Even JavaScript - for all the hate node_modules gets for being enormous, at least it works reliably! In what way is Python + virtualenv + requirements.txt less reliable?

jensgk 6y ago

> I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. Isn't that why Docker is so popular? If we had Python static binaries (or the ability to have several versions of the same Python library installed) would we use Docker so much?

agumonkey 6y ago
and docker is (was) written in go who kinda put more light on static, the circle is complete

gorgoiler 6y ago

In the past week my coding colleague (whose age is only just into double digits) and I have had to: * insert lines of code into asciidoctor Ruby to debug how the command line interface differs from the library interface; * read through Raspberry Pi SenseHat code to figure out that the drivers won’t load via SSH (you have to plug in an HDMI cable!); * rummage through Python’s PY Sequence List to answer the question “what kind of sort does Python actually use?”. The problem with shipping binaries is it lowers the standard for being able to build from source. If a package is easier to ship as a binary because the source is hard to distribute reliably, we give up hope of end users being able to modify and build their own versions. This is obviously bad for freedom. It’s also hard for debugging and learning.

executesorder66 6y ago
You have a 10 year old coding colleague?
- heavenlyblue 6y ago
  The answer to “which sorting algorithm python uses” is the first result in Google. The third is official python docs.
  - gorgoiler 6y ago
    The context was one where we were learning about how things work, rather than wanting to just know the answer per se. I always _try_ to show how one can answer problems from first principles before resorting to “go look it up on stack overflow”. It boosts ones sense of usefulness as a teacher if one can occasionally give more in depth answers as teachable moments. For one thing, demonstrating deeper understand is how you score high marks in exams.
    - heavenlyblue 6y ago
      I don't understand what you mean by "first principles". Open up Python's source code and check - this is first principles-first hand. As a matter of fact I am going to go as far as saying that reading source code is the _only_ way to actually be any good in software engineering because it's the only source of truth that never gets out of sync with the de-facto. Did you expect to find a blog post about it (you can find it on the first page of Google search I mentioned)? Or a YouTube video (didn't pay attention to that)? Books are reasonably good but usually get out of sync quite fast. And going back to first principles: the reason most of the literature may not mention what sorting algorithm Python uses is because a) it's irrelevant as a first principle knowledge b) it gets quickly out of sync.

oefrha 6y ago

zipapp? It’s not like a self-contained solution didn’t exist for a long time. (If you mean the Python installation also needs to be contained in that self-contained binary — sorry you chose a scripting language.) And yesterday I spent at least an hour fighting with cargo build. Without knowing exactly why we fought both our anecdotes are quite useless.

kkapelon 6y ago

>I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. You mean Docker containers? :-)

drran 6y ago
VMWare images or even hardware images: it's easier to buy new phone than to patch old one.

mturilin 6y ago

Deno (https://deno.land https://deno.land) solves this problem very well for JavaScript. I’d like so see more languages adopting “dependencies are URLs” approach. No more package hell.

julianwachholz 6y ago
Like GoLang did?
aequitas 6y ago
How does this solve the problem? As I can tell it replaces a centralized dependencies (requirements.txt) containing references to a known and trusted registry (pypi.org) with url's in all files across your project (and 3rd party library's files). Making maitenance only harder and security updates even more so. And making it harder to debug things as you now keep in mind that multiple versions of the same packages can be used in your source. Also, requirements.txt already support's urls if you want your sources from other places. For me, most of the pain with the Python packaging went away after I started using Pip-tools[0]. It's just a simple utility to add lockfile capabilities to Pip. Nothing new to learn, no new filosophies or paradigm's. No PEP waiting to be adopted by everyone. Just good old requirements.txt + Pip. [0] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools

atoav 6y ago

The best thing would be if python had something like a cross between pip and python-poetry, but better integrated with the interpreter. Python has the most unpythonic package experience ever. As elegant as the language is, packaging is a convoluted nightmare. It still might work for some project, but certainly is no fit for the average python user.

necrotic_comp 6y ago
I started using the pipenv this weekend and it seems like it's pretty good for at least maintaining an environment within a project. I found it friendly at least. That being said, I'll admit I don't know much about what it means to be pythonic. What do you think the drawbacks are to pipenv-type approach and what would be more pythonic in your mind ?
- atoav 6y ago
  What I mean by pythonic is the "batteries included" paradigma: if you want to — say — quickly script something up that does some time calculations you just go and import datetime. There is no need to research what datetime libraries are ok, download and install dependecies etc. You are able to go and (with the included batteries) solve your problem. I wish I could just as easily say: ahh this script I made should become a package that I can reuse on my server. But then the server has Python 3.6 while your script has 3.8, so you end up installing another python, painstakingly watch not to install over the old one etc. When installing modules you have to do the same or set up a venv/pipenv. All along the road there are possible ways to shoot yourself into the leg. Meanwhile in Rust you do a cargo new foobar, work on it, add dependecies to the cargo.toml, copy it to the server, build a binary with cargo build --release and copy the binary into the PATH. You spent zero brainpower on not breaking things and had capacity to think about other stuff. The closest thing we have in those terms in pythonland is poetry, which is a very good start. But this should be part of the language, not something one has to do extra.
  - necrotic_comp 6y ago
    Got it - that makes sense. I actually ended up with a smattering of the version problems when I was working with pipenv for the first time, and figured it was my fault in some way because I hadn't set up the "correct" python version on my system. That's somewhat disheartening to hear that it's a consistent problem. Thanks for the insight and clarification.

ashishb 6y ago

Even more so in a world where storage is cheap.

andarleen 6y ago

Just use a language with proper package management and OOP, such as PHP or NodeJS. Python is great for machine learning, data science, hobbies and one off scripts, and learning basic programming.

jillesvangurp 6y ago

Agreed. I'd go one step further and call that docker container. If it needs to run on a server, the preferred format these days is a docker container. I too spend a bit of time fighting pipenv lately and generally trying to figure out the various ways python tries to virtualize barfing all over the filesystem by default. IMHO python, node.js, ruby, and other scripting languages share a historical confusion of concerns related to development tooling and attempts to counter the unfortunate choice of installing dependencies globally that tend to leak to the server side in inappropriate ways. With Docker.

ClumsyPilot 6y ago
Except we solved the problem at a higher level of abstraction, creating a more complex system. If every language used LLVM (for example) we'd have almost the same effect. There are other things docker can do (like mapping volumes) but I would be much happier if we solved it at the level of the executable.
- jillesvangurp 6y ago
  Your happiness and practical reality are two things. Docker is there right now and sort of the de-facto way of getting stuff on a server in basically all projects I've been on in the last five years. Programs are more than just their binaries. Most programs also expect non binary content like config files, data files, shell scripts, symbolic links, etc. Packaging that up in a sane way is basically what docker is good at.

mintplant 6y ago

Just use Poetry. I haven't fought with Python packaging in a long time.

ghshephard 6y ago
I see a lot of references to Poetry, but I've yet to see any uptake with the "professional" python developers that I know - which could be entirely to inertia, rather than "best tool for the job" - most of them see to be still in the pip/virtualenv/requirements.txt world for all of their development/docker/k8s deployments. I wonder if there have been any surveys on poetry, and whether it's trajectory (presumably exponential) suggests that it's going to become a dominant package management utility in the next 10 years or so?

nh2 6y ago

You may enjoy this StackOverflow answer on The Return of Static Linking, summarising advancements on the topic: https://stackoverflow.com/questions/3430400/linux-static-linking-is-dead/56771892#56771892 https://stackoverflow.com/questions/3430400/linux-static-lin...

vmchale 6y ago

> The more I think about it, the more I feel that self-contained static binaries are the way to go. Trying to load source files from all over the filesystem at runtime is hell. One need not depend on the filesystem for such things. And as bad as python packaging is, others get it better. One can have less-than-static-binaries without being quite that bad. > I would love to see a move towards "static" binaries that package everything together into a single, self-contained unit. Plenty of problems with that too!

blaisio 6y ago

I do prefer static self contained binaries, but I think it's important to note that Python's package management system is horrible, and making self contained binaries is not the only way to fix it.

developer2 6y ago

IMO: Python is a garbage language, and should never be used for professional applications. Try to write an app whose very first block is a try/except for KeyboardInterrupt, with or without an if __name__ == '__main__' block. Now run the program a couple dozen times, and press ctrl+c as fast as you can. You will wind up with a debug-level stack trace dump for an uncaught KeyboardInterrupt exception from the core libraries that is thrown before your first line of userland code is parsed or executed. Python runs so much internal setup code before your first line of code is even considered. It's literally impossible to write a Python program that doesn't dump debug garbage to its output, without writing a separate bash script that uses trap to proxy SIGINT and SIGTERM to the running process. Python is a fucking joke in my eyes; and this is before discussing the way async/await was implemented by gutting the core of the language in a way that makes zero fucking sense. Python is a total… fucking… joke. It has some nice features when all you care is about the syntax and ease of development, but the fact is it's a terribly broken language. Use C. Or C++. Or Rust. Or Go. Or Ruby, or literally no joke… even PHP. Python is literally THAT HORRIBLE once you understand its internals. It absolutely disgusts me to say, but PHP with its single-threaded model is put together better than Python. tldr; Do not use Python. It makes certain tasks seem easy, but all it truly does is make you look like an idiot developer. Pick literally any other language, as Python has screwed up too many basic concepts (related to POSIX and expectations from C) to be taken seriously.

shadeslayer_ 6y ago
comments like this make me want to ask for a downvote button.
- developer2 6y ago
  So you are capable of writing a script in Python that always catches SIGINT and SIGTERM, exiting gracefully without an exception stack being thrown by core code? No? Right… no. Python is literally the only language I've used that cannot do this. Even PHP can do it. For real, PHP is overall a shit language (it has zero async/threading support), but it still handles signals according to POSIX (eg. it doesn't override signal handlers unless userland code does it, preserving default SIG_DFL behavior), unlike Python which breaks that simple contract. The "KeyboardInterrrupt" exception being thrown by core code before userland code is parsed and executed is such an extreme bastardization of POSIX standards that Python is simply, inexcusably, broken. I stand by my opinion that Python is not well-suited to applications written by a professional entity, intended to be shipped to 3rd-party clients. Python plays too loose with the fundamentals provided by the operating system to be a professional language.