10 ms·

Python packaging is good now

160 points by r0muald 10y ago

draw_down 10y ago
I wish!
- dalke 10y ago
  Could you elaborate on the issues you have with the current system?
  - yxhuvud 10y ago
    Is there anything that even remotely comes close to bundler in power and ease of use? Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? Or suppose I have different environments a project can be run in that should load slightly different set of dependencies?
    - StavrosK 10y ago
      > Is there anything that even remotely comes close to bundler in power and ease of use? Yes, pip, since before bundler existed. > Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? pip install --upgrade package > suppose I have different environments a project can be run in that should load slightly different set of dependencies? venv activate myproject
      - actsasbuffoon 10y ago
        There's a bit that's still lacking in Pip. For instance, Pip only installs new packages into your venv, rather than trying to keep it synchronized. I've run into bugs where someone ripped a library out of requirements.txt because they thought it wasn't being used any longer. Turns out it was still being used, but the dev couldn't tell because the lib was still in their venv. Tools like Bundler will actually take the dependency away from you. Python's packaging system is great compared to C, but it's primitive compared to Ruby, Haskell, Clojure, and lots of others.
        chronial 10y ago
        https://github.com/nvie/pip-tools https://github.com/nvie/pip-tools solves this by providing pip-sync, which will uninstall removed packages.
    - Nullabillity 10y ago
      > Is there anything that even remotely comes close to bundler in power and ease of use? Maven and co. > Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? Change the version number in pom.xml/build.sbt/build.gradle. Done. > Or suppose I have different environments a project can be run in that should load slightly different set of dependencies? Define a new project that depends on your main project and the extra dependencies.
      - yxhuvud 10y ago
        Ah yes. The question was about the Python ecosystem. There are other good solutions for other platforms.
    - krakensden 10y ago
      Bundlers default-global behavior is a disaster. Last time I got hired at a ruby shop none of the scripts worked, most projects didn't have a lock file, etc. Because they just ran the things directly, using whatever from the global environment.
      - bhaak 10y ago
        If the gems are installed globally and no Gemfile.lock file is used, of what use is bundler anyway?
      - yxhuvud 10y ago
        While bundler certainly have warts, the disaster in that case is not using a lockfile. That completely defeat the purpose of the tool, to get repeatable installations.
  - sametmax 10y ago
    Having to explain the concept of virtualenv is still a pain compared to local install à la node. "How do I make an exe" with my PyQT app is still a very hard question to answer. Web deployment is still hard in Python because of lack of general hosting support if you don't own your server. The situation is way better, and I'm very glad of the work done, but it has still a long way to go.
    - mwcampbell 10y ago
      > Web deployment is still hard in Python because of lack of general hosting support if you don't own your server. Does conventional shared hosting actually matter anymore, for application developers as opposed to someone just running a WordPress site? Virtual private servers are just as cheap now.
      - sametmax 10y ago
        Yes it does, because a lot of people needing to deploy tools are not dev.
    - jsmthrowaway 10y ago
      > Web deployment is still hard in Python because of lack of general hosting support if you don't own your server. GAE? PythonAnywhere? Heroku? OpenShift? Docker-based stuff? https://wiki.python.org/moin/SpecializedCommercialHosts https://wiki.python.org/moin/SpecializedCommercialHosts maybe? https://wiki.python.org/moin/OtherCommercialHosts https://wiki.python.org/moin/OtherCommercialHosts if that wasn't enough?
      - sametmax 10y ago
        0.1% if the commercial offer for PHP, and they all involve using the command line or know a bit about coding. Again, it's better, but many people I know run PHP business with zero coding knowledge. They upload to ftp scripts they bought on the web, and make a living out of it. Some are kids trying stuff. The little things matter.
  - pvg 10y ago
    It's a good summary of the context, history and tools that have been cleaned up but sweeps an awful lot under the rug with "Then, for each project you want to do, make a new virtualenv [...]". What's that and why are you doing it? Oh, because the packaging doesn't work.
    - empthought 10y ago
      The packaging works fine. Do all Java apps running on a host use the same classpath pointing to /usr/lib directories full of JAR files? Of course not. That's all that virtualenv is doing; the equivalent of managing /lib directories for an invocation of the interpreter.
      - pvg 10y ago
        I don't follow how these are at all similar other than you saying they are 'equivalent'. Typical java apps get their dependencies from the same maven-managed local repo. The tool itself is baroque but conceptually this is much simpler than creating magical 'environments' - the repository of dependencies, what dependencies a particular app uses and what version of the runtime it's invoked with are all separate and orthogonal to each other. An even simpler way to think about it - imagine you just invented python and are designing the tooling around dependency management and packaging. Would you come up with 'environments'?
        empthought 10y ago
        > Typical java apps get their dependencies from the same maven-managed local repo. What? No, they don't. Typical Java apps are deployed on systems that don't even have Maven installed. Typical Java apps are executed with a classpath argument pointing to the libraries installed for the application. What are the directories containing those JARs (or an uberjar) except an "environment?" Hell, the Java classloader interacts with the classpath in pretty much the exact same way that the Python interpreter interacts with a virtualenv.
  - Anderkent 10y ago
    Deploying a python application is still a horrible experience. My best attempts so far involve doing `pip wheel` during the build, then discarding anything that's platform dependent and fetching their source dists instead, then bundling that together with a script that sets up a virtualenv and installs all these dependencies and the project being installed ... Compare to a jvm uberjar.
    - brianwawok 10y ago
      Except Python you can often run from git. Git pull do some pip done.
      - shadowmint 10y ago
        To be fair, you need a python runtime installed to run python, the same way you need a java runtime installed to run uberjars. ...but, the deployment is a single artifact. I don't think doing a git clone, followed by a pip install and waiting for all the dependencies to download... and compile... (or not, if you have the wrong system packages installed) really compares.
        brianwawok 10y ago
        Why not? If you are doing a blue/green type deploy.. you are basically building up a new VM to be your new environment. It can do all kinds of download and compile type stuff. Who cares if some is pip installing?
      - Eridrus 10y ago
        You can do the same things in Java land as long as you have used a build system written in the last decade such as Maven or Gradle, you can just as easily pull down a git repo and run mvn package
        brianwawok 10y ago
        Well that is a) A 30 sec - 10 minute process to build (vs 0 in python) b) Weird because Java has a fatjar concept. Python and Java are different. The easy path to deploy them doesn't need to be the same. You may need to make them the same for ops sanity.. but if you are just doing 1 or the other, meh
  - _rs 10y ago
    pip doesn't even use a solver to determine which packages to install, it just uses the first mention it finds.
INTPenis 10y ago
Personal perspective of someone who's only used python 4 years and mostly in an linux ops capacity is that the packaging has always been pleasant to use and the few times I've had to install a node/ruby app as an end user have been comparatively horrible.
cocotino 10y ago
Funny that the PyPI web interface is down right now.
0xmohit 10y ago
The real takeaway here though, is that although it’s still not perfect, other languages are no longer doing appreciably better. Not sure how accurate is this. At least not true in the context of OCaml/OPAM. As always, I’m sure none of this applies to Rust and Cargo is basically perfect, but that doesn’t matter, because nobody reading this is actually using Rust. So it seems that those using rust don't read HN: https://github.com/servo/servo https://github.com/servo/servo https://github.com/dropbox/rust-brotli https://github.com/dropbox/rust-brotli -- Bottom line: Bash everyone else arbitrarily and claim oneself to be the best!
- galacticpony 10y ago
  https://en.wikipedia.org/wiki/Hyperbole https://en.wikipedia.org/wiki/Hyperbole
kwgibson 10y ago
"From 2008 to 2012 or so, Python packaging was a total mess." Some things have improved, but fundamentally there really is not much difference in creating a package now vs. then. A certain xkcd comes to mind.
scarygliders 10y ago
Python packaging is great. What's a bother is - as that article mentions - end-user deployments. Now, I sort of have that down to a tee by using cx_freeze and Inno Setup, such that I can distribute my software to my end-users and all they have to do is click on a Setup.exe and my applications install just like any other Windows program. I can do a similar thing on the Mac, but it's still a pain. The main trouble with packaging Python applications up, is an apparent assumption (at least, that's the impression I have gotten over years of figuring out how to distribute my applications), that Python programs should all be open source - so for example, distutils assumes that the author wants his code to be readable by everyone who wishes to. I wanted to be able to produce full applications and somehow compile and package them as in for e example, a C/C++ application. The closest I can get to that currently, is cx_freeze, which isn't compilation, merely turning the Python code into bytecode. Nuitka is showing lots of promise, but as yet it still cannot compile my applications into something usable (my really big Qt4 GUI applications either won't compile or there are lots of problems cropping up even if they do manage to run). Here's hoping end-user deployment gets much easier than it is now, someday!
- yolesaber 10y ago
  Could you explicate on your end-user deployment workflow a bit? I'm building a game in python and would love to distribute .exes to Windows players
  - ju-st 10y ago
    py2exe works well for me. It's no Windows installer but packaged in a zip file is good enough (for my use case at least). The only annoyance is the need to install some Microsoft C++ redistributables on the client side.
    - scarygliders 10y ago
      See my workflow - once you get it down to a tee, it becomes a very smooth workflow and a breeze to distribute in one convenient setup.exe.
  - 10y ago
tux1968 10y ago
The article say that with any recent python you can get started with a one liner: Python Core started distributing the ensurepip module along with both Python 2.7 and 3.3 ... $ python -m ensurepip --user Tried here with python-2.7.11-3 and just got a traceback error. Turns out according to one of the links given by the article[1] that this feature was actually not backported to these older versions of Python, apparently to encourage people to upgrade to new Python. This is unfortunate. [1] https://www.python.org/dev/peps/pep-0453/#including-ensurepip-in-python-2-7-and-3-3 https://www.python.org/dev/peps/pep-0453/#including-ensurepi...
- Flimm 10y ago
  Which OS are you using? On Ubuntu, I get this useful output: $ python2 -m ensurepip --user ensurepip is disabled in Debian/Ubuntu for the system python. Python modules For the system python are usually handled by dpkg and apt-get. apt-get install python-<module name> Install the python-pip package to use pip itself. Using pip together with the system python might have unexpected results for any system installed module, so use it on your own risk, or make sure to only use it in virtual environments.
  - tux1968 10y ago
    Interesting. On Fedora here, and it actually includes version 3.4 of python with the command name "python3", and the above instructions worked just fine using it. However, now you have me worried that the article left out caveats and warnings about how these commands might lead to "unexpected results".
    - Macha 10y ago
      The worry is: 1. Install system python 2. Install pip from somewhere 3. pip install requests 4. apt-get install something with depends on python-requests Should the installation of apt's requests fail? Should it work but clobber your pip installed requests? What happens when the apt version is older than the pip version? Does this break whatever you used pip install requests for? There's two resolutions to this issue. 1. Don't use pip, always use apt for your python dependencies. The Ubuntu devs would really prefer you pick this option but not everything on pypi is on their repo and sometimes working consistently across platforms is more about than within platform consistency. 2. Use a different python environment than the system python. This could be done by using virtualenv, a different python version or a separate installation.
Flimm 10y ago
This is the first time I've heard of ensurepip. It looks cool, but I'll wait until it becomes common knowledge before declaring that "Python packaging is good now", but I definitely agree that it's improving. I'll have to update my StackOverflow answer now to mention this. http://stackoverflow.com/a/14753678/247696 http://stackoverflow.com/a/14753678/247696
fdik 10y ago
Is it? On Debian there is Debian Python, and Debian pip. Then there is easy_install to install own pip, and that's different. virtualenv comes on top of all. Of course, there is Debian's Python packages, too. On MacOS X with Macports, there is Apple's Python and Macports' Pythons, all with different pips. Of course, there is Macports' Python packages, too, for different Pythons.
- deleted 10y ago
  [deleted]
kevin_thibedeau 10y ago
> python setup.py sdist bdist_wheel Wheels suck. By using the bdist mechanism they make it impossible to build a single cross-platform, cross-Python package that just works. Wheels can't run 2to3 at installation so guess what? You have to make separate packages for Python 2 and 3. There should be an easy way to have a pure Python package that "just works" on Python 2 and 3, 32-bit and 64-bit platforms without having to build separate packages. Right now that can only be done with the third-party Setuptools using sdist and a traditional setup.py. Distutils is still a half implemented packaging system. I wouldn't say that Python packaging is a solved thing.
- zzzeek 10y ago
  wheels are only useful for Pypi uploads as far as they allow uploads of difficult builds, like Win32 compilations. wheels are unbelievably critical in that they allow super easy caching of pre-built packages on any given host. If you've noticed these days you can type "pip install numpy" at will and it seems to usually run in less than two seconds rather than 5 minutes, thank wheels. This is particularly a big deal if you work lot with CI.
- ak217 10y ago
  You can easily build a single cross-platform, cross-Python wheel that just works. You just need to add "[bdist_wheel] universal=1" to your setup.cfg and write your code in a cross-Python style enabled by six and future (https://pypi.python.org/pypi/future https://pypi.python.org/pypi/future).
  - zzzeek 10y ago
    if you have any C code, you can't. Even if your C code is optional, you shouldn't, at least one that you'd put on Pypi, because unix users with full toolchains will never see your source .tar.gz anymore from pip.
    - ak217 10y ago
      Fair to say, but then there isn't any technology out there to build "a single cross-platform, cross-Python wheel that just works" and links against C code. That's true of any packaging framework that links against native code...
vog 10y ago
While this is all nice, I have yet to find a package system that is as straight forward to use as NPM. Don't get me wrong, I love Python and I'm not so much a fan of NodeJS, as Python is so much better designed than JavaScript, even in time of ES6. But their NPM is really well designed. NPM alone gives NodeJS a competitve advantage over many other popular languages including Ruby (Gem) and Python (PIP). For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv. NodeJS just ships with NPM. Done. Any why do we need to create and manage a "virtualenv"? Why do you have to go through extra hassle just to avoid global installs? NPM just creates a local sub directory node_modules and automatically installs and removes from that. Global installs happen only if you really want it to (npm -g). But you usually don't. The only central place at NPM is the package cache so it doesn't download things twice if you have two separate projects with similar dependencies. In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup.
- jessaustin 10y ago
  It almost seems, based on TFA's risibly misguided "dumb idea" link to npmjs.com, that the author thinks python's story is better than node/npm's. It's called "separation of concerns", kids, and it's a good thing, that one part of a script doesn't have to worry about what a different part of the script is doing. The fact python never got it to work correctly is python's problem; node handled it from the beginning.
- swsieber 10y ago
  NPM is also a pain in the neck - I've had to wipe out the node_modules folder many times because it got in a bad state when adding dependencies. Other than that, I agree.
  - transcranial 10y ago
    In my experience, the only time this happens is with `npm uninstall`. If you only `npm install` stuff forever, node_modules seems pretty stable. Disk usage, however, is another matter.
zb 10y ago
> There should be tools that help you write and update a setup.py. Or a setup.python.json or something, so you don’t actually need to write code just to ship some metadata. There are some out there, e.g. OpenStack uses https://pypi.python.org/pypi/pbr https://pypi.python.org/pypi/pbr - it was pretty painful around the 1.0 transition when no two libraries seemed to be using compatible versions, but it hasn't caused me any trouble in quite a while now. I'm more concerned by the lack of a (non-greedy) dependency solver in pip.
beagle3 10y ago
No discussion of python packaging without mention of conda is complete. It's as good as "apt-get" for Python (and not just for Python, but ... mostly for Python). http://conda.pydata.org/miniconda.html http://conda.pydata.org/miniconda.html
- anc84 10y ago
  I believed that! Then I actually used it and got incompatible packages, packages only for 32 bit XOR 64 bit etc. It was not remotely as good as the package managers I am used to on several distros. :(
lucb1e 10y ago
I don't know what this article is talking about. What is wrong with apt-get install python-something to install the python package called something? Or if it's not in the repositories, you checkout the library to a subdirectory of your project and call `import something` just like you would when apt-getting it. So honest question, what's this easy_install stuff? Apt has been around since the nineties, just like python, and didn't change much as far as I know.
- chronial 10y ago
  apt is not cross-platform.
  - lucb1e 10y ago
    OS X, BSD, Android, Fedora, etc. users will just have to use their respective package managers, or use the second option if there is none (downloading the source to a subdirectory). Python code will not run without a platform-specific interpreter anyway.
    - pvg 10y ago
      Those package managers would then have to know a lot about the specific needs of each language runtime they support. They don't and shouldn't. The alternative - each runtime has its own canonical package manager means you get parallel package systems and duplicated effort which is unfortunate. On the other hand, they actually work unlike, say, 'apt for python packages'.
  - mhurron 10y ago
    apt also installs packages globally to the system and not every python package exists as a deb package.
    - dozzie 10y ago
      Are you from the crowd that thinks that DEB packages only grow on preselected trees and people can't build a DEB themselves for what they need?
kiwidrew 10y ago
My main peeve is that distutils2 died, and with it, the idea of putting package metadata (author, version, dependencies) in `setup.cfg` instead of `setup.py`. It's bad enough that I have a boilerplate `setup.py` that just does `import ConfigParser` and blasts the entire `[setup.py]` section into `setuptools.setup()` as `kwargs`. I copy and paste this boilerplate into every new project that I start. Oh and while we're ranting about Python packaging, let me rant about tox, which is so-very-useful but also so-very-dangerous: IF YOU (SLIGHTLY) MISCONFIGURE YOUR TOX.INI FILE, IT WILL DELETE YOUR ENTIRE PROJECT DIRECTORY! [1] I suppose I am getting too old and cranky for this stuff. :) [1] https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-of-my-code https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-o...
- galacticpony 10y ago
  Maybe I'm a bit too old-fashioned, but you should always assume that any program you run will delete your home directory with a probability > 0. If you're not prepared for that situation at all times, you are using computers wrong.
- tomprince 10y ago
  distutils2 may have died. The idea of declarative package metadata hasn't (although it may not have advanced as quickly as you might like). There is a PEP[1] (which is sadly not yet implemented[2]) which a) starts to move some metadata into declarative format b) is designed to allow implementation of alternative build systems that have more declarative metadata. [1] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0518/ [2] https://github.com/pypa/pip/issues/3691 https://github.com/pypa/pip/issues/3691
andrewvijay 10y ago
That's a very good type face to read in mobile!
deleted 10y ago
[deleted]
a3n 10y ago
It may well be. But one problem is all the references to all the various old ways that a first time deployer has to wade through. It took me quite a while to figure out what's current and what should be ignored, as well as what's old but still needs to be done, on a small project that I did to learn exactly this.
hardmath123 10y ago
A while back, I wrote some of my thoughts on Python packaging. I got excited seeing this title, but reading the article, I remain unconvinced. The whole thing is still way too confusing, and it's still not nearly as intuitive (or teachable) as "npm install ___".
- ak217 10y ago
  Most of the things you said in your writeup were no longer true in February 2015, and I think there are no concrete things that you wrote that are true anymore, thanks to the hard work of Python packaging infrastructure maintainers (https://github.com/pypa https://github.com/pypa). I find the griping a bit over the top. The point of the OP is that new exciting improvements have arrived in the Python packaging infrastructure. Yes, there were many ugly overwrought things in setuptools' past. Yes, npm has several great ideas. npm is also much younger than setuptools/PyPI, and I can cite lots of horror stories from its pre-io.js days too. But the volunteers who got us this far deserve to be recognized for the progress they've made. I think the biggest legitimate concern right now is how unintuitive it is to use pip in the presence of several Python runtimes/versions.
  - hardmath123 10y ago
    Yeah, I'm sure things are better. And your typical developer will probably be just fine (I mean, we know they are because Python is so popular!). But for a beginner who is just trying to figure out how to share his or her code, it's still very confusing. And yes, part of that is definitely the fact that all of the mistakes of the past and their messy workarounds still show up in Google searches, muddying the waters; this obviously gives npm an edge. But even so, I feel that npm just does a better job when it comes to being learnable: people pick it up right away with very little trouble, and you can have your first "public-facing" module up on day one with very little effort. Indeed, in a way, the fact that it's so easy is what's giving it the most grief right now! So there are definitely lessons to be learned there, for everyone.
- galacticpony 10y ago
  "pip install ___" Everything else is pretty much obsolete. The problem is in how it doesn't really work reliably, especially with C extensions, which are much more common in Python-Land than with anything Javascript/NodeJS-related. You're also comparing with something that, by virtue of its age, has far less technical debt. Also, NPM is a "commercial product", PyPI is a community effort. Maybe the comparison to something like Anaconda would be more apt. The author acknowledges most problems anyway, he just comes to the wrong conclusion: It's not "good" now, it's still terrible, but everything else is more-or-less as bad or worse.
a3n 10y ago
$ python -m ensurepip --user $ python -m pip install --user --upgrade pip $ python -m pip install --user --upgrade virtualenv > ... > From here on out, now the world is your oyster; you can pip install to your heart’s content, and you probably won’t even need to compile any C for most packages. These instructions don’t depend on Python version, either: as long as it’s up-to-date, the same steps work on Python 2, Python 3, Nit: I suppose installing virtualenv works in 2 and 3, but hasn't venv in 3 made that unnecessary?
- rspeer 10y ago
  virtualenv has an advantage over venv, which is that people have spent years and years writing instructions for virtualenv, and not for the cases that venv handles slightly differently. For example, with virtualenv you can choose to use a mix of globally- and locally-installed packages. With venv you can't. (You get global or local, but not both.)
  - yladiz 10y ago
    In that regard, venv is better. It's better to have a single place where you install your packages rather than a few different places, especially if you're working on a team. It would be hell to have a mix, not knowing exactly which goes where or why, and then explaining to another team member your setup in a coherent manner. It's much better to just know that your packages are either global or local and no in-betweens.
    - justanotherbody 10y ago
      > It's much better to just know that your packages are either global or local and no in-betweens. The database driver and scipy are installed globally. Everything else is local to the environment. Not hell at all, especially compared to compiling numpy/scipy every time you want to spin up a venv that uses a single feature. Examples other than DB drivers and scipy exist. Compiling software isn't really that hard, but it's an unnecessary hurdle in many cases - especially when you've already bought into an OS that does binary distributions specifically so you don't have to compile anything. I'd be comfortable agreeing that any pure-python package should be venv only, but not that every package should be.
slantedview 10y ago
No matter what small tooling changes come about, the Python packaging ecosystem is fundamentally flawed, and will always be more complicated and less useful than others such as Maven (yes, Maven). The primary reason is that setup.py, etc., have you download/manage one set of dependencies for each project you're working on rather than simply housing all dependencies in a common folder, separated by version, as Maven does. It's pointlessly redundant. But the real pain comes when you start to install things. Since in Python land, dependencies for all tools are typically stored in the same folder, you very easily and quickly hit version conflicts since the dependencies are not really segregated by version. This leads to brutal bugs that creep on you at runtime when you realize, too late, that some tool you installed has dependencies that conflict with another. What's worse, complex Python projects like OpenStack may have numerous dependencies which all in turn have differing and conflicting dependency versions of their own. Trying to keep them all in sync is a ridiculous exercise in time wasting and pointlessness, and problems caused by version mismatches are common. The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run. Again, pointlessly redundant and a pain to manage. Is there a better way? Obviously. Use a single common repo with dependencies structurally segregated by version (ie: different folders), as is the way of doing things in Java-land. I know that Java-land stuff is bad and we're all supposed to hate it. But I've always failed to understand why developers in other language ecosystems continue to create tools in a vacuum, as if roadmaps for solving these problems don't already exist elsewhere. Look beyond your bubble. Hopefully one day the Python folks will catch on.
- zzzcpan 10y ago
  All of the CPAN-inspired packaging ecosystems are fundamentally broken. Including Maven. > The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run. Brilliant solution is to recognize that environments need to be packaged and managed too. No sarcasm. That's what nix does. And it makes package management a solved problem (ignoring current implementation-specific issues of course). So, yes, there is a better way and it's nix [1] and guix [2] and the ideas behind them. Check them out. [1] https://nixos.org/nix/ https://nixos.org/nix/ [2] http://www.gnu.org/software/guix/ http://www.gnu.org/software/guix/
predakanga 10y ago
In the past 12 hours, I started a new Python project. I can't agree with the sentiment - the most I can say is that Python packaging isn't terrible now. My intention was to create a Python 3 package using best-practices, a nice division of labour by process to avoid threading problems, etc. So far I've run into the following issues regarding packaging: Getting started - there's a glut of documentation, much of it contradictory, much of it out-of-date. Simply google "Python packaging" and you'll have to pick between 4 sources of info on the first page, all of which look official. The tools - this one was a new one for me; I normally use virtualenvs for development, but vaguely remembered it being bundled in Python 3. Turns out it's not virtualenv but "pyvenv". Beyond the confusion of the name change, there's also a tool called "pyenv" muddying the waters. The specifics - I started with https://packaging.python.org/ https://packaging.python.org/ and was struck by how ambiguous the instructions were. Doing something as simple as declaring your package's version has 7 different options: https://packaging.python.org/single_source_version/ https://packaging.python.org/single_source_version/. In the end, I got the most help from reading the guide's sample project... I only have to wonder about the licensing implications of that. The infrastructure - because I was creating a project with multiple executables, I needed a way to start one from the other. Despite knowing that setuptools is responsible for generating the executables, I quickly found that there's just no way to find those executable paths programmatically. Library choice - I needed an inotify library. I searched for "inotify", "notify" and "watch" on Pypi but had little way of distinguishing the results - the "weight" column is useless as compared to a rating system or even download count. Thankfully, I remembered about awesome-python.com which bridges the gap. All told, it took me several hours to bootstrap the project, not even including things like testing. I'd say Python packaging has a way to go before I'd call it "good".
- joshmarlow 10y ago
  I tend to use [0] for the boilerplate for getting started in writing a Python package. It was mostly used for Python 2.7, but I've written a few local Python 3.5 packages with it and everything seems to be fine (ie, "it worked on my machine - what could go wrong?"). I then use `nosetests -v` to run the tests. [0] - github.com/ambitioninc/ambition-python-template
harlowja 10y ago
While I agree with the fact that it isn't terrible as it once was; I personally can't say pip is really starting to look any better until the following is solved: https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 ''Pip needs a dependency resolver'' Without such a thing you really don't quite have a way to ensure what you are requesting to be installed actually is a valid set of things.
pbreit 10y ago
I seem to feel more comfortable with redundant copies of libraries than virtual environments.
gsmethells 10y ago
I'd take pip over npm any day of the week. My experiences with Node have been horrible. Npm just grows okay but do not try to uninstall.
libeclipse 10y ago
For a long time, PyPi intermittently reported extremely exaggerated download statistics and sometimes never even reported any statistics. Then they got bored of trying to fix it and removed it altogether, saying they want to concentrate on warehouse. I believe currently the only way to get any kind of statistics is to perform a query on Google's Big. Any attempt to talk about this issue results in the thread being locked and offended replies, and while I understand what it's like from the developer's point of view, the current state of things is completely unreasonable.
urschrei 10y ago
The last three (cross-platform, Python 2 and Python 3) packages I published to PyPI have been Ctypes/Cython wrappers around Rust binaries. Yes, I'm a sample size of 1, but whatever, never let reality get in the way of your rhetorical blogpost, I suppose.
je42 10y ago
NPM did it right. PIP got it wrong. Requiring the same version of a library per application is really a bad idea. It creates a lot of work to align your dependencies.

160 points by r0muald 10y ago

draw_down 10y ago

I wish!

dalke 10y ago
Could you elaborate on the issues you have with the current system?
- yxhuvud 10y ago
  Is there anything that even remotely comes close to bundler in power and ease of use? Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? Or suppose I have different environments a project can be run in that should load slightly different set of dependencies?
  - StavrosK 10y ago
    > Is there anything that even remotely comes close to bundler in power and ease of use? Yes, pip, since before bundler existed. > Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? pip install --upgrade package > suppose I have different environments a project can be run in that should load slightly different set of dependencies? venv activate myproject
    - actsasbuffoon 10y ago
      There's a bit that's still lacking in Pip. For instance, Pip only installs new packages into your venv, rather than trying to keep it synchronized. I've run into bugs where someone ripped a library out of requirements.txt because they thought it wasn't being used any longer. Turns out it was still being used, but the dev couldn't tell because the lib was still in their venv. Tools like Bundler will actually take the dependency away from you. Python's packaging system is great compared to C, but it's primitive compared to Ruby, Haskell, Clojure, and lots of others.
      - chronial 10y ago
        https://github.com/nvie/pip-tools https://github.com/nvie/pip-tools solves this by providing pip-sync, which will uninstall removed packages.
  - Nullabillity 10y ago
    > Is there anything that even remotely comes close to bundler in power and ease of use? Maven and co. > Suppose for example that I want to upgrade a package and all packages that it depends on, how would that be done? Change the version number in pom.xml/build.sbt/build.gradle. Done. > Or suppose I have different environments a project can be run in that should load slightly different set of dependencies? Define a new project that depends on your main project and the extra dependencies.
    - yxhuvud 10y ago
      Ah yes. The question was about the Python ecosystem. There are other good solutions for other platforms.
  - krakensden 10y ago
    Bundlers default-global behavior is a disaster. Last time I got hired at a ruby shop none of the scripts worked, most projects didn't have a lock file, etc. Because they just ran the things directly, using whatever from the global environment.
    - bhaak 10y ago
      If the gems are installed globally and no Gemfile.lock file is used, of what use is bundler anyway?
    - yxhuvud 10y ago
      While bundler certainly have warts, the disaster in that case is not using a lockfile. That completely defeat the purpose of the tool, to get repeatable installations.
- sametmax 10y ago
  Having to explain the concept of virtualenv is still a pain compared to local install à la node. "How do I make an exe" with my PyQT app is still a very hard question to answer. Web deployment is still hard in Python because of lack of general hosting support if you don't own your server. The situation is way better, and I'm very glad of the work done, but it has still a long way to go.
  - mwcampbell 10y ago
    > Web deployment is still hard in Python because of lack of general hosting support if you don't own your server. Does conventional shared hosting actually matter anymore, for application developers as opposed to someone just running a WordPress site? Virtual private servers are just as cheap now.
    - sametmax 10y ago
      Yes it does, because a lot of people needing to deploy tools are not dev.
  - jsmthrowaway 10y ago
    > Web deployment is still hard in Python because of lack of general hosting support if you don't own your server. GAE? PythonAnywhere? Heroku? OpenShift? Docker-based stuff? https://wiki.python.org/moin/SpecializedCommercialHosts https://wiki.python.org/moin/SpecializedCommercialHosts maybe? https://wiki.python.org/moin/OtherCommercialHosts https://wiki.python.org/moin/OtherCommercialHosts if that wasn't enough?
    - sametmax 10y ago
      0.1% if the commercial offer for PHP, and they all involve using the command line or know a bit about coding. Again, it's better, but many people I know run PHP business with zero coding knowledge. They upload to ftp scripts they bought on the web, and make a living out of it. Some are kids trying stuff. The little things matter.
- pvg 10y ago
  It's a good summary of the context, history and tools that have been cleaned up but sweeps an awful lot under the rug with "Then, for each project you want to do, make a new virtualenv [...]". What's that and why are you doing it? Oh, because the packaging doesn't work.
  - empthought 10y ago
    The packaging works fine. Do all Java apps running on a host use the same classpath pointing to /usr/lib directories full of JAR files? Of course not. That's all that virtualenv is doing; the equivalent of managing /lib directories for an invocation of the interpreter.
    - pvg 10y ago
      I don't follow how these are at all similar other than you saying they are 'equivalent'. Typical java apps get their dependencies from the same maven-managed local repo. The tool itself is baroque but conceptually this is much simpler than creating magical 'environments' - the repository of dependencies, what dependencies a particular app uses and what version of the runtime it's invoked with are all separate and orthogonal to each other. An even simpler way to think about it - imagine you just invented python and are designing the tooling around dependency management and packaging. Would you come up with 'environments'?
      - empthought 10y ago
        > Typical java apps get their dependencies from the same maven-managed local repo. What? No, they don't. Typical Java apps are deployed on systems that don't even have Maven installed. Typical Java apps are executed with a classpath argument pointing to the libraries installed for the application. What are the directories containing those JARs (or an uberjar) except an "environment?" Hell, the Java classloader interacts with the classpath in pretty much the exact same way that the Python interpreter interacts with a virtualenv.
- Anderkent 10y ago
  Deploying a python application is still a horrible experience. My best attempts so far involve doing `pip wheel` during the build, then discarding anything that's platform dependent and fetching their source dists instead, then bundling that together with a script that sets up a virtualenv and installs all these dependencies and the project being installed ... Compare to a jvm uberjar.
  - brianwawok 10y ago
    Except Python you can often run from git. Git pull do some pip done.
    - shadowmint 10y ago
      To be fair, you need a python runtime installed to run python, the same way you need a java runtime installed to run uberjars. ...but, the deployment is a single artifact. I don't think doing a git clone, followed by a pip install and waiting for all the dependencies to download... and compile... (or not, if you have the wrong system packages installed) really compares.
      - brianwawok 10y ago
        Why not? If you are doing a blue/green type deploy.. you are basically building up a new VM to be your new environment. It can do all kinds of download and compile type stuff. Who cares if some is pip installing?
    - Eridrus 10y ago
      You can do the same things in Java land as long as you have used a build system written in the last decade such as Maven or Gradle, you can just as easily pull down a git repo and run mvn package
      - brianwawok 10y ago
        Well that is a) A 30 sec - 10 minute process to build (vs 0 in python) b) Weird because Java has a fatjar concept. Python and Java are different. The easy path to deploy them doesn't need to be the same. You may need to make them the same for ops sanity.. but if you are just doing 1 or the other, meh
- _rs 10y ago
  pip doesn't even use a solver to determine which packages to install, it just uses the first mention it finds.

INTPenis 10y ago

Personal perspective of someone who's only used python 4 years and mostly in an linux ops capacity is that the packaging has always been pleasant to use and the few times I've had to install a node/ruby app as an end user have been comparatively horrible.

cocotino 10y ago

Funny that the PyPI web interface is down right now.

0xmohit 10y ago

The real takeaway here though, is that although it’s still not perfect, other languages are no longer doing appreciably better. Not sure how accurate is this. At least not true in the context of OCaml/OPAM. As always, I’m sure none of this applies to Rust and Cargo is basically perfect, but that doesn’t matter, because nobody reading this is actually using Rust. So it seems that those using rust don't read HN: https://github.com/servo/servo https://github.com/servo/servo https://github.com/dropbox/rust-brotli https://github.com/dropbox/rust-brotli -- Bottom line: Bash everyone else arbitrarily and claim oneself to be the best!

galacticpony 10y ago
https://en.wikipedia.org/wiki/Hyperbole https://en.wikipedia.org/wiki/Hyperbole

kwgibson 10y ago

"From 2008 to 2012 or so, Python packaging was a total mess." Some things have improved, but fundamentally there really is not much difference in creating a package now vs. then. A certain xkcd comes to mind.

scarygliders 10y ago

Python packaging is great. What's a bother is - as that article mentions - end-user deployments. Now, I sort of have that down to a tee by using cx_freeze and Inno Setup, such that I can distribute my software to my end-users and all they have to do is click on a Setup.exe and my applications install just like any other Windows program. I can do a similar thing on the Mac, but it's still a pain. The main trouble with packaging Python applications up, is an apparent assumption (at least, that's the impression I have gotten over years of figuring out how to distribute my applications), that Python programs should all be open source - so for example, distutils assumes that the author wants his code to be readable by everyone who wishes to. I wanted to be able to produce full applications and somehow compile and package them as in for e example, a C/C++ application. The closest I can get to that currently, is cx_freeze, which isn't compilation, merely turning the Python code into bytecode. Nuitka is showing lots of promise, but as yet it still cannot compile my applications into something usable (my really big Qt4 GUI applications either won't compile or there are lots of problems cropping up even if they do manage to run). Here's hoping end-user deployment gets much easier than it is now, someday!

yolesaber 10y ago
Could you explicate on your end-user deployment workflow a bit? I'm building a game in python and would love to distribute .exes to Windows players
- ju-st 10y ago
  py2exe works well for me. It's no Windows installer but packaged in a zip file is good enough (for my use case at least). The only annoyance is the need to install some Microsoft C++ redistributables on the client side.
  - scarygliders 10y ago
    See my workflow - once you get it down to a tee, it becomes a very smooth workflow and a breeze to distribute in one convenient setup.exe.
- 10y ago

tux1968 10y ago

The article say that with any recent python you can get started with a one liner: Python Core started distributing the ensurepip module along with both Python 2.7 and 3.3 ... $ python -m ensurepip --user Tried here with python-2.7.11-3 and just got a traceback error. Turns out according to one of the links given by the article[1] that this feature was actually not backported to these older versions of Python, apparently to encourage people to upgrade to new Python. This is unfortunate. [1] https://www.python.org/dev/peps/pep-0453/#including-ensurepip-in-python-2-7-and-3-3 https://www.python.org/dev/peps/pep-0453/#including-ensurepi...

Flimm 10y ago
Which OS are you using? On Ubuntu, I get this useful output: $ python2 -m ensurepip --user ensurepip is disabled in Debian/Ubuntu for the system python. Python modules For the system python are usually handled by dpkg and apt-get. apt-get install python-<module name> Install the python-pip package to use pip itself. Using pip together with the system python might have unexpected results for any system installed module, so use it on your own risk, or make sure to only use it in virtual environments.
- tux1968 10y ago
  Interesting. On Fedora here, and it actually includes version 3.4 of python with the command name "python3", and the above instructions worked just fine using it. However, now you have me worried that the article left out caveats and warnings about how these commands might lead to "unexpected results".
  - Macha 10y ago
    The worry is: 1. Install system python 2. Install pip from somewhere 3. pip install requests 4. apt-get install something with depends on python-requests Should the installation of apt's requests fail? Should it work but clobber your pip installed requests? What happens when the apt version is older than the pip version? Does this break whatever you used pip install requests for? There's two resolutions to this issue. 1. Don't use pip, always use apt for your python dependencies. The Ubuntu devs would really prefer you pick this option but not everything on pypi is on their repo and sometimes working consistently across platforms is more about than within platform consistency. 2. Use a different python environment than the system python. This could be done by using virtualenv, a different python version or a separate installation.

Flimm 10y ago

This is the first time I've heard of ensurepip. It looks cool, but I'll wait until it becomes common knowledge before declaring that "Python packaging is good now", but I definitely agree that it's improving. I'll have to update my StackOverflow answer now to mention this. http://stackoverflow.com/a/14753678/247696 http://stackoverflow.com/a/14753678/247696

fdik 10y ago

Is it? On Debian there is Debian Python, and Debian pip. Then there is easy_install to install own pip, and that's different. virtualenv comes on top of all. Of course, there is Debian's Python packages, too. On MacOS X with Macports, there is Apple's Python and Macports' Pythons, all with different pips. Of course, there is Macports' Python packages, too, for different Pythons.

deleted 10y ago
[deleted]

kevin_thibedeau 10y ago

> python setup.py sdist bdist_wheel Wheels suck. By using the bdist mechanism they make it impossible to build a single cross-platform, cross-Python package that just works. Wheels can't run 2to3 at installation so guess what? You have to make separate packages for Python 2 and 3. There should be an easy way to have a pure Python package that "just works" on Python 2 and 3, 32-bit and 64-bit platforms without having to build separate packages. Right now that can only be done with the third-party Setuptools using sdist and a traditional setup.py. Distutils is still a half implemented packaging system. I wouldn't say that Python packaging is a solved thing.

zzzeek 10y ago
wheels are only useful for Pypi uploads as far as they allow uploads of difficult builds, like Win32 compilations. wheels are unbelievably critical in that they allow super easy caching of pre-built packages on any given host. If you've noticed these days you can type "pip install numpy" at will and it seems to usually run in less than two seconds rather than 5 minutes, thank wheels. This is particularly a big deal if you work lot with CI.
ak217 10y ago
You can easily build a single cross-platform, cross-Python wheel that just works. You just need to add "[bdist_wheel] universal=1" to your setup.cfg and write your code in a cross-Python style enabled by six and future (https://pypi.python.org/pypi/future https://pypi.python.org/pypi/future).
- zzzeek 10y ago
  if you have any C code, you can't. Even if your C code is optional, you shouldn't, at least one that you'd put on Pypi, because unix users with full toolchains will never see your source .tar.gz anymore from pip.
  - ak217 10y ago
    Fair to say, but then there isn't any technology out there to build "a single cross-platform, cross-Python wheel that just works" and links against C code. That's true of any packaging framework that links against native code...

vog 10y ago

While this is all nice, I have yet to find a package system that is as straight forward to use as NPM. Don't get me wrong, I love Python and I'm not so much a fan of NodeJS, as Python is so much better designed than JavaScript, even in time of ES6. But their NPM is really well designed. NPM alone gives NodeJS a competitve advantage over many other popular languages including Ruby (Gem) and Python (PIP). For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv. NodeJS just ships with NPM. Done. Any why do we need to create and manage a "virtualenv"? Why do you have to go through extra hassle just to avoid global installs? NPM just creates a local sub directory node_modules and automatically installs and removes from that. Global installs happen only if you really want it to (npm -g). But you usually don't. The only central place at NPM is the package cache so it doesn't download things twice if you have two separate projects with similar dependencies. In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup.

jessaustin 10y ago
It almost seems, based on TFA's risibly misguided "dumb idea" link to npmjs.com, that the author thinks python's story is better than node/npm's. It's called "separation of concerns", kids, and it's a good thing, that one part of a script doesn't have to worry about what a different part of the script is doing. The fact python never got it to work correctly is python's problem; node handled it from the beginning.
swsieber 10y ago
NPM is also a pain in the neck - I've had to wipe out the node_modules folder many times because it got in a bad state when adding dependencies. Other than that, I agree.
- transcranial 10y ago
  In my experience, the only time this happens is with `npm uninstall`. If you only `npm install` stuff forever, node_modules seems pretty stable. Disk usage, however, is another matter.

zb 10y ago

> There should be tools that help you write and update a setup.py. Or a setup.python.json or something, so you don’t actually need to write code just to ship some metadata. There are some out there, e.g. OpenStack uses https://pypi.python.org/pypi/pbr https://pypi.python.org/pypi/pbr - it was pretty painful around the 1.0 transition when no two libraries seemed to be using compatible versions, but it hasn't caused me any trouble in quite a while now. I'm more concerned by the lack of a (non-greedy) dependency solver in pip.

beagle3 10y ago

No discussion of python packaging without mention of conda is complete. It's as good as "apt-get" for Python (and not just for Python, but ... mostly for Python). http://conda.pydata.org/miniconda.html http://conda.pydata.org/miniconda.html

anc84 10y ago
I believed that! Then I actually used it and got incompatible packages, packages only for 32 bit XOR 64 bit etc. It was not remotely as good as the package managers I am used to on several distros. :(

lucb1e 10y ago

I don't know what this article is talking about. What is wrong with apt-get install python-something to install the python package called something? Or if it's not in the repositories, you checkout the library to a subdirectory of your project and call `import something` just like you would when apt-getting it. So honest question, what's this easy_install stuff? Apt has been around since the nineties, just like python, and didn't change much as far as I know.

chronial 10y ago
apt is not cross-platform.
- lucb1e 10y ago
  OS X, BSD, Android, Fedora, etc. users will just have to use their respective package managers, or use the second option if there is none (downloading the source to a subdirectory). Python code will not run without a platform-specific interpreter anyway.
  - pvg 10y ago
    Those package managers would then have to know a lot about the specific needs of each language runtime they support. They don't and shouldn't. The alternative - each runtime has its own canonical package manager means you get parallel package systems and duplicated effort which is unfortunate. On the other hand, they actually work unlike, say, 'apt for python packages'.
- mhurron 10y ago
  apt also installs packages globally to the system and not every python package exists as a deb package.
  - dozzie 10y ago
    Are you from the crowd that thinks that DEB packages only grow on preselected trees and people can't build a DEB themselves for what they need?

kiwidrew 10y ago

My main peeve is that distutils2 died, and with it, the idea of putting package metadata (author, version, dependencies) in `setup.cfg` instead of `setup.py`. It's bad enough that I have a boilerplate `setup.py` that just does `import ConfigParser` and blasts the entire `[setup.py]` section into `setuptools.setup()` as `kwargs`. I copy and paste this boilerplate into every new project that I start. Oh and while we're ranting about Python packaging, let me rant about tox, which is so-very-useful but also so-very-dangerous: IF YOU (SLIGHTLY) MISCONFIGURE YOUR TOX.INI FILE, IT WILL DELETE YOUR ENTIRE PROJECT DIRECTORY! [1] I suppose I am getting too old and cranky for this stuff. :) [1] https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-of-my-code https://bitbucket.org/hpk42/tox/issues/352/tox-deleted-all-o...

galacticpony 10y ago
Maybe I'm a bit too old-fashioned, but you should always assume that any program you run will delete your home directory with a probability > 0. If you're not prepared for that situation at all times, you are using computers wrong.
tomprince 10y ago
distutils2 may have died. The idea of declarative package metadata hasn't (although it may not have advanced as quickly as you might like). There is a PEP[1] (which is sadly not yet implemented[2]) which a) starts to move some metadata into declarative format b) is designed to allow implementation of alternative build systems that have more declarative metadata. [1] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0518/ [2] https://github.com/pypa/pip/issues/3691 https://github.com/pypa/pip/issues/3691

andrewvijay 10y ago

That's a very good type face to read in mobile!

deleted 10y ago

[deleted]

a3n 10y ago

It may well be. But one problem is all the references to all the various old ways that a first time deployer has to wade through. It took me quite a while to figure out what's current and what should be ignored, as well as what's old but still needs to be done, on a small project that I did to learn exactly this.

hardmath123 10y ago

A while back, I wrote some of my thoughts on Python packaging. I got excited seeing this title, but reading the article, I remain unconvinced. The whole thing is still way too confusing, and it's still not nearly as intuitive (or teachable) as "npm install ___".

ak217 10y ago
Most of the things you said in your writeup were no longer true in February 2015, and I think there are no concrete things that you wrote that are true anymore, thanks to the hard work of Python packaging infrastructure maintainers (https://github.com/pypa https://github.com/pypa). I find the griping a bit over the top. The point of the OP is that new exciting improvements have arrived in the Python packaging infrastructure. Yes, there were many ugly overwrought things in setuptools' past. Yes, npm has several great ideas. npm is also much younger than setuptools/PyPI, and I can cite lots of horror stories from its pre-io.js days too. But the volunteers who got us this far deserve to be recognized for the progress they've made. I think the biggest legitimate concern right now is how unintuitive it is to use pip in the presence of several Python runtimes/versions.
- hardmath123 10y ago
  Yeah, I'm sure things are better. And your typical developer will probably be just fine (I mean, we know they are because Python is so popular!). But for a beginner who is just trying to figure out how to share his or her code, it's still very confusing. And yes, part of that is definitely the fact that all of the mistakes of the past and their messy workarounds still show up in Google searches, muddying the waters; this obviously gives npm an edge. But even so, I feel that npm just does a better job when it comes to being learnable: people pick it up right away with very little trouble, and you can have your first "public-facing" module up on day one with very little effort. Indeed, in a way, the fact that it's so easy is what's giving it the most grief right now! So there are definitely lessons to be learned there, for everyone.
galacticpony 10y ago
"pip install ___" Everything else is pretty much obsolete. The problem is in how it doesn't really work reliably, especially with C extensions, which are much more common in Python-Land than with anything Javascript/NodeJS-related. You're also comparing with something that, by virtue of its age, has far less technical debt. Also, NPM is a "commercial product", PyPI is a community effort. Maybe the comparison to something like Anaconda would be more apt. The author acknowledges most problems anyway, he just comes to the wrong conclusion: It's not "good" now, it's still terrible, but everything else is more-or-less as bad or worse.

a3n 10y ago

$ python -m ensurepip --user $ python -m pip install --user --upgrade pip $ python -m pip install --user --upgrade virtualenv > ... > From here on out, now the world is your oyster; you can pip install to your heart’s content, and you probably won’t even need to compile any C for most packages. These instructions don’t depend on Python version, either: as long as it’s up-to-date, the same steps work on Python 2, Python 3, Nit: I suppose installing virtualenv works in 2 and 3, but hasn't venv in 3 made that unnecessary?

rspeer 10y ago
virtualenv has an advantage over venv, which is that people have spent years and years writing instructions for virtualenv, and not for the cases that venv handles slightly differently. For example, with virtualenv you can choose to use a mix of globally- and locally-installed packages. With venv you can't. (You get global or local, but not both.)
- yladiz 10y ago
  In that regard, venv is better. It's better to have a single place where you install your packages rather than a few different places, especially if you're working on a team. It would be hell to have a mix, not knowing exactly which goes where or why, and then explaining to another team member your setup in a coherent manner. It's much better to just know that your packages are either global or local and no in-betweens.
  - justanotherbody 10y ago
    > It's much better to just know that your packages are either global or local and no in-betweens. The database driver and scipy are installed globally. Everything else is local to the environment. Not hell at all, especially compared to compiling numpy/scipy every time you want to spin up a venv that uses a single feature. Examples other than DB drivers and scipy exist. Compiling software isn't really that hard, but it's an unnecessary hurdle in many cases - especially when you've already bought into an OS that does binary distributions specifically so you don't have to compile anything. I'd be comfortable agreeing that any pure-python package should be venv only, but not that every package should be.

slantedview 10y ago

No matter what small tooling changes come about, the Python packaging ecosystem is fundamentally flawed, and will always be more complicated and less useful than others such as Maven (yes, Maven). The primary reason is that setup.py, etc., have you download/manage one set of dependencies for each project you're working on rather than simply housing all dependencies in a common folder, separated by version, as Maven does. It's pointlessly redundant. But the real pain comes when you start to install things. Since in Python land, dependencies for all tools are typically stored in the same folder, you very easily and quickly hit version conflicts since the dependencies are not really segregated by version. This leads to brutal bugs that creep on you at runtime when you realize, too late, that some tool you installed has dependencies that conflict with another. What's worse, complex Python projects like OpenStack may have numerous dependencies which all in turn have differing and conflicting dependency versions of their own. Trying to keep them all in sync is a ridiculous exercise in time wasting and pointlessness, and problems caused by version mismatches are common. The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run. Again, pointlessly redundant and a pain to manage. Is there a better way? Obviously. Use a single common repo with dependencies structurally segregated by version (ie: different folders), as is the way of doing things in Java-land. I know that Java-land stuff is bad and we're all supposed to hate it. But I've always failed to understand why developers in other language ecosystems continue to create tools in a vacuum, as if roadmaps for solving these problems don't already exist elsewhere. Look beyond your bubble. Hopefully one day the Python folks will catch on.

zzzcpan 10y ago
All of the CPAN-inspired packaging ecosystems are fundamentally broken. Including Maven. > The brilliant solution (sarcasm here), of course, is to create virtual environments with separate copies of every dependency for every tool we run. Brilliant solution is to recognize that environments need to be packaged and managed too. No sarcasm. That's what nix does. And it makes package management a solved problem (ignoring current implementation-specific issues of course). So, yes, there is a better way and it's nix [1] and guix [2] and the ideas behind them. Check them out. [1] https://nixos.org/nix/ https://nixos.org/nix/ [2] http://www.gnu.org/software/guix/ http://www.gnu.org/software/guix/

predakanga 10y ago

In the past 12 hours, I started a new Python project. I can't agree with the sentiment - the most I can say is that Python packaging isn't terrible now. My intention was to create a Python 3 package using best-practices, a nice division of labour by process to avoid threading problems, etc. So far I've run into the following issues regarding packaging: Getting started - there's a glut of documentation, much of it contradictory, much of it out-of-date. Simply google "Python packaging" and you'll have to pick between 4 sources of info on the first page, all of which look official. The tools - this one was a new one for me; I normally use virtualenvs for development, but vaguely remembered it being bundled in Python 3. Turns out it's not virtualenv but "pyvenv". Beyond the confusion of the name change, there's also a tool called "pyenv" muddying the waters. The specifics - I started with https://packaging.python.org/ https://packaging.python.org/ and was struck by how ambiguous the instructions were. Doing something as simple as declaring your package's version has 7 different options: https://packaging.python.org/single_source_version/ https://packaging.python.org/single_source_version/. In the end, I got the most help from reading the guide's sample project... I only have to wonder about the licensing implications of that. The infrastructure - because I was creating a project with multiple executables, I needed a way to start one from the other. Despite knowing that setuptools is responsible for generating the executables, I quickly found that there's just no way to find those executable paths programmatically. Library choice - I needed an inotify library. I searched for "inotify", "notify" and "watch" on Pypi but had little way of distinguishing the results - the "weight" column is useless as compared to a rating system or even download count. Thankfully, I remembered about awesome-python.com which bridges the gap. All told, it took me several hours to bootstrap the project, not even including things like testing. I'd say Python packaging has a way to go before I'd call it "good".

joshmarlow 10y ago
I tend to use [0] for the boilerplate for getting started in writing a Python package. It was mostly used for Python 2.7, but I've written a few local Python 3.5 packages with it and everything seems to be fine (ie, "it worked on my machine - what could go wrong?"). I then use `nosetests -v` to run the tests. [0] - github.com/ambitioninc/ambition-python-template

harlowja 10y ago

While I agree with the fact that it isn't terrible as it once was; I personally can't say pip is really starting to look any better until the following is solved: https://github.com/pypa/pip/issues/988 https://github.com/pypa/pip/issues/988 ''Pip needs a dependency resolver'' Without such a thing you really don't quite have a way to ensure what you are requesting to be installed actually is a valid set of things.

pbreit 10y ago

I seem to feel more comfortable with redundant copies of libraries than virtual environments.

gsmethells 10y ago

I'd take pip over npm any day of the week. My experiences with Node have been horrible. Npm just grows okay but do not try to uninstall.

libeclipse 10y ago

For a long time, PyPi intermittently reported extremely exaggerated download statistics and sometimes never even reported any statistics. Then they got bored of trying to fix it and removed it altogether, saying they want to concentrate on warehouse. I believe currently the only way to get any kind of statistics is to perform a query on Google's Big. Any attempt to talk about this issue results in the thread being locked and offended replies, and while I understand what it's like from the developer's point of view, the current state of things is completely unreasonable.

urschrei 10y ago

The last three (cross-platform, Python 2 and Python 3) packages I published to PyPI have been Ctypes/Cython wrappers around Rust binaries. Yes, I'm a sample size of 1, but whatever, never let reality get in the way of your rhetorical blogpost, I suppose.

je42 10y ago

NPM did it right. PIP got it wrong. Requiring the same version of a library per application is really a bad idea. It creates a lot of work to align your dependencies.