12 ms·

You shouldn't invoke setup.py directly

139 points by itamarst 5y ago

earthboundkid 5y ago
How long until packaging makes Python the new PHP?
- PhoenixReborn 5y ago
  Poetry helps alleviate some of the issues mentioned here, and lets you separate dev vs prod dependencies. As far as the "new PHP" goes, it'd be great if Python ever powered almost 80% of websites: https://kinsta.com/blog/is-php-dead/ https://kinsta.com/blog/is-php-dead/
- icedchai 5y ago
  Probably never. PHP is at least generally simple to deploy.
- 1_player 5y ago
  Docker has been the saving grace for Python. I once used to build deb and rpms of a Python project of mine and I'd rather stick a finger in my eye than go through that pain again.
alexmcc81 5y ago
Python is such a joy to write and such a nightmare to set up, package and distribute. The Python Foundation needs to adopt or create standard and supported ways of doing things. I spend so much time dealing with issues that should have been solved a long time ago. Special shout out to the PyInstaller devs for providing so much value while constantly fighting to deal with upstream changes.
- danjac 5y ago
  > The Python Foundation needs to adopt or create standard and supported ways of doing things The Python community has adopted or created a standard way of doing things for the (checks calendar) fourth or fifth time now.
  - alexmcc81 5y ago
    I used to recommend articles with titles like "modern Python package structure and tooling" to graduates. I stopped because the tools (poetry, pipenv etc) were changing so much. I try to fit as much of our workflows into hand made templates and PyCharm now.
    - sseagull 5y ago
      There was a post here a while ago about packaging or virtual environments. Within a few minutes, there were quite a few responses about better ways to do it. Unfortunately, each response had a different recommendation. I wish I could find that thread again.
      - dendrite9 5y ago
        Maybe this one? https://news.ycombinator.com/item?id=26733423 https://news.ycombinator.com/item?id=26733423 We have been talking about how to support customers who want to use python. At least for me it isn't very appealing, seems like it could take a lot of attention to keep up with what's expected.
  - judge2020 5y ago
    Obligatory https://xkcd.com/927/ https://xkcd.com/927/
    - garaetjjte 5y ago
      Also https://xkcd.com/1987/ https://xkcd.com/1987/
      - egberts1 5y ago
        ^THIS^
  - colpabar 5y ago
    Right, but the community is different than the foundation. I would like an authoritative "this is how you do it" doc, from the foundation, about setup/packaging/dist, and I want it to work. And if said doc advises using some third party tool, then the foundation needs to fund or take over maintenance of that tool so that it continues to work and the "standard" way of doing it doesn't change every year. It seems crazy to me that we don't have this yet.
- montroser 5y ago
  Agreed. There has been so much churn and fragmentation with build tools over the last few years. So much complexity for what is not such a complicated problem to solve. Hopefully this is the "has to get worse before it can get better" part.
  - gaborbernat 5y ago
    > So much complexity for what is not such a complicated problem to solve. As someone who actually worked weeks on their spare time, I beg to differ. This is a very complicated problem to solve. And everyone has their own opinion on how things should work, which is goverened by their narrow use case. But you see a "standard" packaging tool needs to be the opposite of narrow use case. The main reason Anacond Inc exists is because they wanted to solve this for data science. Even with them being a relatively big corporation their "solution" is not loved universally, but works ok most of the time.
  - thrashh 5y ago
    I disagree. I actually think it’s a complicated problem because there’s so many issues that have to be solved all at once. What has been happening is that someone writes a build tool to solve a specific problem that they have but they neglect to solve all the other ones. Why would they? They’re not being funded. That’s what caused (and still causes) a lot of the churn in JS too.
- stavros 5y ago
  Nowadays I use Poetry and PyInstaller, which make things easier. For some of my projects, I produce a "compiled"/bundled executable that takes all the pain out of distribution. Here's my config, maybe it will be useful to you: https://gitlab.com/stavros/harbormaster/-/blob/master/.gitlab-ci.yml#L29 https://gitlab.com/stavros/harbormaster/-/blob/master/.gitla...
  - k1rcher 5y ago
    I’ve noticed Poetry being used more and more for projects I setup locally, and I must say, it does seem to be very, very good. Minimal overhead, easy to adopt, etc. I’ll need to setup a proper workflow with it for my current projects. Being able to abstract dev/prod environments in particular are really handy.
    - stavros 5y ago
      Agreed, I really like it. It does everything and does it well, zero complaints so far.
      - bigbillheck 5y ago
        I managed to find an annoying situation with poetry in which if you specify a range of versions for a dependency, and one of your other dependencies specifies a slightly different range of versions for that dependency, it will refuse to proceed. (I was expecting it to have chosen something in the intersection of the ranges, but it did not)
        stavros 5y ago
        Hm, yeah, that definitely sounds like a resolver bug. Did you report it? I'm curious to see if it was fixed.
        bigbillheck 5y ago
        I did not report it because I found somebody else had already reported it and it was a WONTFIX (for reasons I don't immediately recall).
- game_the0ry 5y ago
  > Python is such a joy to write and such a nightmare to set up, package and distribute. Agreed. Though I would not call it a "nightmare," the python ecosystem and runtime are cumbersome and annoying to work with when compared to some other languages that have put in a lot of thought into DX. Still, it could be worse.
  - a_cool_username 5y ago
    Once you hit the sweet spot of developing for cross-platform (even just Linux, MacOS, and Windows) and supporting normal average-people users and have (even optional!) C dependencies, Python's packaging situation quickly deteriorates into "nightmare" territory.
    - alexmcc81 5y ago
      This is exactly my problem. I have to support Windows (a locked down corporate version) and Linux.
- satyanash 5y ago
  > Python is such a joy to write Try out Ruby: so many Python syntax decisions make no sense after you see the Ruby way.
  - qudat 5y ago
    Im convinced the only people that like ruby are those that want to delve into metaprogramming. It might be fun for the person that constructs it but it is a nightmare as an outsider looking into a project. I know this is more rails but the fact that autoloading is even a thing speaks to the wild nature of ruby.
  - gorgoiler 5y ago
    I used to love Ruby. But it’s too exciting. Python is boring, in a good way. Like mowing the lawn. Or buying potatoes. I no longer want excitement in the process, only in the results.
  - stavros 5y ago
    I did, but so many Ruby syntax decisions made no sense to me.
  - mrweasel 5y ago
    This might just be me not understanding Ruby well enough, but managing packages and dependecies feel even worse in Ruby.
    - enchiridion 5y ago
      Conda has become vastly better since I used it a few years ago. Worth a try again
      - mrweasel 5y ago
        That’s a pretty good idea, also a tool many Python developers already use, making the move between the two languages easier.
    - riffraff 5y ago
- BiteCode_dev 5y ago
  setting up is still unsolved, but: - for librairy packaging and general dep maangement, poetry is the solution - for packaging a web project and ship it on the server, shiv is the solution - for distribution, half the solution is nuitka for compiling the program. Creating the installer is now the hard part
  - alexmcc81 5y ago
    I haven't looked at Nuitka in a long time. Last time I did Windows support was not great and compilation took forever. I'll try it again.
- qbasic_forever 5y ago
  I almost wonder if we need something like deno but for the python language--a reboot that uses the same language and syntax, but jettisons the cruft of packaging, etc. and starts over with a better developer experience. Perhaps just reference and import packages directly by URLs and forget all the pain and struggle of curating system-wide and virtual environments.
- shakezula 5y ago
  I've stopped recommending Python for learning programming, and instead have started recommending Go. Python's environment difficulties make it exponentially harder for beginners to focus on actual programming concepts. Things like pyenv or virtualenvs or any of those things are blackboxes to beginners. Go is simple, it's tightly coupled to version control so you can teach git at the same time in a meaningful way, and it's low level enough that CS concepts come up frequently, while being powerful enough that it will scale with a beginner's knowledge.
  - ebb_earl_co 5y ago
    Interesting. Do you have a beginner’s course or tutorial that you would suggest? Something like the Rust book is wha came to mind
    - shakezula 5y ago
      I would start with https://gobyexample.com/ https://gobyexample.com/ and https://learnxinyminutes.com/docs/go/ https://learnxinyminutes.com/docs/go/ Go By Example will take a beginner from Hello World to writing a server in a short time, and you can naturally splice in other skills.
  - slownews45 5y ago
    My wife started to learn to program - the setup overhead in python (going via mac -> git -> windows / linux) was crazy.
    - spicybright 5y ago
      Can you not use your system's default python and open idle? Why involve git?
      - macintux 5y ago
        On Mac, at least as of Catalina, the system default Python is 2.something.
- robomartin 5y ago
  > Python is such a joy to write and such a nightmare to set up, package and distribute. Here's my simple-minded metric on when this problem can be called "solved": Get a VPS from GoDaddy. Deploy your Django app as easily as you can deploy a PHP app, say, Wordpress. I have written about this before. I love Django. And yet I think that they have done a huge disservice with the development server and DB configuration out of the box. Even for a simple application, going from developing on your desktop to deploying the same application on, as an example, GoDaddy, is in a range between nightmare and impossible. I have personally given up multiple times and just said "Fuck it! Just use WP" even when I really, truly wanted to stay in the Python/Django ecosystem. Try it. Develop a simple photo album application. Get a VPS from GoDaddy and deploy it. No Heroku et. al., are not solutions. They are indicative of the problem.
  - riekus 5y ago
    Sorry, but GoDaddy is a disgrace, and your comment makes no sense at all, you are comparing WP with Django. And instead of recommending VPS as a solution, you recommend the worst party offering these services? GoDaddy should burn in hell for their shady shit.
    - selfhoster11 5y ago
      GoDaddy is shady and I would not recommend it ever, but they are a good representative of the target market - something people pick despite the shadiness and inflexibility, because they require zero technical competency to get something up and running.
    - robomartin 5y ago
      GoDaddy is a disgrace? It's shady shit? C'mon, get some perspective. I have been using GoDaddy for, well, I forget how long, maybe two decades. I have run websites for multiple companies from their servers. I can't remember a single issue. Not one. Even hosting and managing email. Frankly, I don't know what you are talking about. I have also used Linode, AWS, Dreamhost, Rackspace and a bunch of others for different kinds of work. Not sure where the hatred for GoDaddy comes from. I also know a bunch of people using GoDaddy servers. I think people repeat shit just because they think they sound "cool" and yet have no clue what they are talking about. Also, GoDaddy support, on the rare occasions they are needed, has always been top notch. So, yeah, no clue what you are talking about.
- deleted 5y ago
  [deleted]
- fifilura 5y ago
  > Python is such a joy to write and such a nightmare to set up, package and distribute. Python is such a joy to write and such a nightmare to maintain, set up, package and distribute. Fixed that for you.
3np 5y ago
Is this really an issue? While there are a lot of unmaintained packages doing it (which I assume will still be working for some years?), I don't think I've seen a recent package doing it in years. In the off-chance you end up depending on that legacy code, repackaging that one or two dependencies shouldn't be that big of a deal.
- pganssle 5y ago
  I think that it is true that I should have written this article or something like it years ago, but there's still a long tail of people who don't even know that invoking setup.py is bad, or that the goal is to get rid of all setup.py commands. Probably the biggest stragglers will be build scripts, and as recently as mid-2020, the answer to "how do I build distributable artifacts without using setup.py" was still "you can do it but there's no good definitive answer": https://stackoverflow.com/q/58753970/467366 https://stackoverflow.com/q/58753970/467366 There's also been a lot of mixed messaging in this space, so I think there's a decent population of people who think that the way to stop invoking setup.py is to stop using setuptools entirely, mixing up "don't invoke setup.py" with "use a backend that doesn't have a setup.py". But you are right to say that this has gotten a lot better over time and the word is slowly getting out.
  - 3np 5y ago
    To be honest, as more of a casual user of Python I wasn't aware of the situation at all so I found your post very informative! I did always find the setup.py approach misguided for several reasons, so it's more of a casual observation I've done that I thankfully rarely encounter it these days.
bjt2n3904 5y ago
What made python beat out Ruby? There was one way to do things. It was simple, and it worked. Then python 3 wanted to become JavaScript. Async, Unicode, floating point division. Move fast and break things. No one way to do things, it's however you want! I had to explain python deployments to a novice the other day. What a nightmare. Python 3.5 to 3.8 just kept drastically changing things. My first experience wasn't pleasant, and for the first time I needed virtual environments and custom apt repositories to install multiple versions of python. But it finally seems things have settled a bit. Here's hoping the Python cultural revolution settles down, and stops trying to be a trendy language, and instead a dependable one.
- _joel 5y ago
  I normally use https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv when I need to test multiple versions.
- joeberon 5y ago
  I don't see how any of that is relevant to the article, but the reason python beat out ruby was almost entirely due to ruby being really really slow.
  - samhw 5y ago
    > the reason python beat out ruby was almost entirely due to ruby being really really slow Relative to Python, that language which notoriously makes C look like a sloth on training wheels...
    - joeberon 5y ago
      Python is much faster than MRI Ruby used to be back in the day
    - elcomet 5y ago
      So you might imagine what ruby felt like ...
    - spicybright 5y ago
joeberon 5y ago
Python distribution is honestly horrific. The worst thing is that I actually see people defending it or saying it isn't that bad if you simply learn it "properly" but I have no idea what they mean when they say this.
- a_humean 5y ago
  It just means they have never worked in another langauge that does this any better.
  - mrweasel 5y ago
    It could be better, but it doesn’t feel worse than .Net or Go, and it’s much better than Javascript/Node. Personally I’m also confused by Ruby. I guess Java is sort of okay, but which languages do people feel have excellent package management?
    - a_humean 5y ago
      Rust's cargo is the high water mark today (and is a substantial reason for Rust's popularity). Ruby's rubygems, node's npm, and the elm package manager all get different things right and wrong, but at least they are a common approach that is somewhat sane. Python is total anarchy. I think most people would disagree with you about .Net, Go, and Javascript today vs python.
    - joeberon 5y ago
      I would guess that most people feel like .Net, Go, and Javascript have very easy distribution and package management
    - recursive 5y ago
      .net is just nuget. It seems pretty straightforward. The last time I looked for a "blessed" solution in python, it was some third part solution surrounded in some kind of internet/twitter drama.
    - OJFord 5y ago
dilawar 5y ago
Once you get into distributing python package, it is such a nightmare. Relative imports were also not easy to understand. And C/C++ extensions in your package makes things even worse. CMake for extension, setup.py for python of you hope to build cross-platform. Pybind11 helped a bit but for each python version you have to compile the wheel again, even with manylinux docker image. Imagine learning docker so you can release a wheel! The other language I often use, C++, is also no better. I am drawn to Rust and Nim lately (not entirely because of these issues).
CapmCrackaWaka 5y ago
As someone who actively contributes to Python packages using the setuptools cli interface, this is jarring. I had no idea. I guess I'll move my build process to some other tools. I started contributing to Python about a year ago. At that time, all of the results when you google "how to build a python package" refer to the setuptools CLI with setup.py. I put it in my workflow and bam, I've been using it ever since without incident.
- webknjaz 5y ago
  Save this tutorial to your bookmarks, then: https://packaging.python.org/tutorials/packaging-projects/#configuring-metadata https://packaging.python.org/tutorials/packaging-projects/#c...
- gaborbernat 5y ago
  Sadly there's no easy way to mark all those Google searches out of date... One of the big goals of this project is to spread the knowledge though, so you reading it means success.
  - monkeybutton 5y ago
    I find myself frequently using the date filter in Google when searching programming topics. Especially if if I'm looking up an error message, almost anything older than a year is noise.
- cdrt 5y ago
  You don't need to completely change if you don't want to. setuptools will most likely continue supporting the old way for a long time since there is so much Python code out there that will probably never update to the new standards. Also, it takes only minimal changes to bring a setuptools-based project into the present. You just need to add a `pyproject.toml` file to your project with the following: [build-system] requires = ["setuptools >= 40.9.0", "wheel"] build-backend = "setuptools.build_meta" Then when it's time to distribute the project, instead of `setup.py sdist` and `setup.py bdist_wheel`, you can just use the official build[1] tool like so: pyproject-build --sdist --wheel And that's all there is to it. Once that's in place and if you're feeling frisky, you can try taking advantage of new setuptools features that can allow you to eliminate `setup.py` entirely and specify everything in `setup.cfg`[2] so that all your project's information is static and no code needs to be run when installing it, assuming your project doesn't contain any compiled code. As and added benefit, by putting pyproject.toml in your projects now, if you wanted to switch over to another tool like poetry or flit later on, the act of packaging the project doesn't have to change as long as you update the pyproject.toml file with the new build system. The same `pyproject-build` command will work for all three systems. [1] https://pypi.org/project/build/ https://pypi.org/project/build/ [2] https://setuptools.pypa.io/en/latest/userguide/declarative_config.html https://setuptools.pypa.io/en/latest/userguide/declarative_c...
est 5y ago
Or simply copy everything into your project under the "lib" path, change your PYTHONPATH and done. (aka "vendoring")
carapace 5y ago
I think Python dependency/build/install/test stuff would be helped greatly by a library or module that did for it what pathlib did for file & path manipulation, eh?
carapace 5y ago
Py3 feels to me like a massive case of "screw backwards compatibility" for very little gain. I feel like Python the language peaked somewhere between 2.4 and 2.7. Python was much slower than most other languages, but in return you got great readability ("runnable pseudocode") and rapid prototyping. Now, with all the additions to the language (more syntax and features, static typing), those advantages are weakening but you're still not getting the advantages of faster compiled languages. It seems to me that Go or Nim (or D or Zig or ...) are preferable for readability, and that Rust is more worth one's time if you don't mind the complexity.
- musicale 5y ago
  In an alternate universe, Python 3 would have - not broken backward compatibility (an astonishing value-destroying move with huge externalities that are still being felt today) - provided a standard and improved packaging system
- 1_player 5y ago
  I agree 100% with you. I built my career on top of Python, and 2.7 is the last version I've used. I don't like that Python 3 is adding any feature under the sun for whatever reason. Have you seen their sad excuse of pattern match proposal? And packaging is still a bloody mess. There's worse languages, it definitely gets the job done, but I'd rather stick with better languages.
sleepysysadmin 5y ago
I've never looked into setuptools. I just 'python3 setup.py' for all my python projects. What benefit does this provide?
eesmith 5y ago
> you can remove your setup.py file and replace it with setuptools' own declarative configuration format Not if you have a Python/C extension. Not if your build step includes code generation (other than Cython). Not if you have special per-file compilation args. Also, in the declarative configuration format, how do I enable/disable compile-time flags like "compile with OpenMP" ? Yes, I have these, and it really irks me reading all of these "you don't need setup.py" when as far as I can tell, I can't.
- bobbyi 5y ago
  > Not if your build step includes code generation (other than Cython). Is it possible for Cython? Their guide shows using setup.py: https://cython.readthedocs.io/en/latest/src/tutorial/cython_tutorial.html#cython-hello-world https://cython.readthedocs.io/en/latest/src/tutorial/cython_...
  - eesmith 5y ago
    https://setuptools.pypa.io/en/latest/userguide/distribution.html#distributing-extensions-compiled-with-cython https://setuptools.pypa.io/en/latest/userguide/distribution.... > setuptools will detect at build time whether Cython is installed or not. If Cython is not found setuptools will ignore pyx files. > To ensure Cython is available, include Cython in the build-requires section of your pyproject.toml: [build-system] requires=[..., "cython"] > Built with pip 10 or later, that declaration is sufficient to include Cython in the build. For broader compatibility, declare the dependency in your setup-requires of setup.cfg:
- pganssle 5y ago
  Sorry, I could have been more clear there, and I've updated the post. I have the "curse of knowledge" and took is as assumed that you can only do this if your particular build configuration fits in setup.cfg. For me, the fact that you have the option to use a setup.py is one of the biggest selling points of setuptools. For other build backends, it's common to have a great experience when you are on the "happy path" and then have no options when you have any of a million exceedingly rare deviations. So if I had a complicated build like you have (and I do have some mildly complicated builds where this is the case), I'd look at that bullet point and think, "Ah, that doesn't apply to this, but it's good to know for my simpler projects". That said, it would be really nice if at least the C-extension part of this worked with the declarative format. My OSS time has been severely curtailed lately and I'm spread way too thin, so I probably won't be able to execute on this, but my long term vision for setuptools has always been that most common build configurations should be achievable using declarative config only, and everything else has setup.py as an escape hatch. This is similar to the way almost all Rust projects are configured with Cargo.toml, but in rare cases you can use build.rs to execute some Rust code as part of the build.
bobbyi 5y ago
The only case where I still invoke setup.py directly is "python setup.py test". I already knew it's deprecated, but don't understand what I'm supposed to do instead. The article recommends tox, but how did tox get installed? I can currently create and activate a new empty virtualenv, "pip install -U pip", clone and cd into my project, and run "python setup.py test". My test dependencies are automatically setup and my preferred runner (pytest) is used. If I change those in the future, the same commands will keep working because I am configuring those in setup.cfg, etc. If I am now expected to manually "pip install tox" before testing my project and change that in the future if we start using something else, isn't that a step backward to the situation pyproject.toml et al were supposed to solve with needing to manually install specific stuff before using a project?
- orf 5y ago
  Use poetry. “poetry run pytest”, or “poetry shell” then do whatever.
- pganssle 5y ago
  This is at worst a lateral move — you need something installed that handles your test invocation for you. It could be `make`, `tox`, `nox`, `poetry` or whatever. For a number of reasons (detailed in the article), that thing shouldn't be `setuptools`. Even if the setuptools maintainers wanted to continue supporting this use case, setuptools is uniquely unsuited to working with this, because of the way its hooks don't fire until all its dependencies have been imported already. With `tox` you just need to make sure you have `tox` installed, and you can even specify a minimum version in the `tox.ini` file. Additionally, tox doesn't need to be installed in your Python environment (in fact, it's an implementation detail that it's Python at all — if it were instead a Go executable, it would work the same). You just need the tool available to be invoked. This means you can install it with pipx, in its own virtual environment, through your system package manager, whatever is most convenient for you.
loloquwowndueo 5y ago
“ I realize that this article is quite long, and unfortunately this is the short version. Much nuance and history is lost here, even though the post is already long enough that it is destined for the limbo of a bookmarks folder called "to read later". Despite the fact that no one will read more than the next 3 paragraphs…” - pro tip - if you want to make the article shorter and easier to read don’t spend an entire paragraph blabbering about how long it is and how nobody reads anymore. I bet if the author had aimed to be succinct, the article would be half the current size and much quicker to read and understand.
- hermitsings 5y ago
  I've done just that: put it in bookmarks for read later.
scrollaway 5y ago
I'm surprised the article, for being so thorough, doesn't have a single mention of Poetry. Which in terms or "works for most people, for most use cases", really has that down. And of course uses the same PEP standards as the other tools mentioned.
- webknjaz 5y ago
  It also has a lot of downsides which is why I avoid it, for example. The purpose of the article is not to point at other tools but to show how to use setuptools better.

139 points by itamarst 5y ago

earthboundkid 5y ago

How long until packaging makes Python the new PHP?

PhoenixReborn 5y ago
Poetry helps alleviate some of the issues mentioned here, and lets you separate dev vs prod dependencies. As far as the "new PHP" goes, it'd be great if Python ever powered almost 80% of websites: https://kinsta.com/blog/is-php-dead/ https://kinsta.com/blog/is-php-dead/
icedchai 5y ago
Probably never. PHP is at least generally simple to deploy.
1_player 5y ago
Docker has been the saving grace for Python. I once used to build deb and rpms of a Python project of mine and I'd rather stick a finger in my eye than go through that pain again.

alexmcc81 5y ago

Python is such a joy to write and such a nightmare to set up, package and distribute. The Python Foundation needs to adopt or create standard and supported ways of doing things. I spend so much time dealing with issues that should have been solved a long time ago. Special shout out to the PyInstaller devs for providing so much value while constantly fighting to deal with upstream changes.

danjac 5y ago
> The Python Foundation needs to adopt or create standard and supported ways of doing things The Python community has adopted or created a standard way of doing things for the (checks calendar) fourth or fifth time now.
- alexmcc81 5y ago
  I used to recommend articles with titles like "modern Python package structure and tooling" to graduates. I stopped because the tools (poetry, pipenv etc) were changing so much. I try to fit as much of our workflows into hand made templates and PyCharm now.
  - sseagull 5y ago
    There was a post here a while ago about packaging or virtual environments. Within a few minutes, there were quite a few responses about better ways to do it. Unfortunately, each response had a different recommendation. I wish I could find that thread again.
    - dendrite9 5y ago
      Maybe this one? https://news.ycombinator.com/item?id=26733423 https://news.ycombinator.com/item?id=26733423 We have been talking about how to support customers who want to use python. At least for me it isn't very appealing, seems like it could take a lot of attention to keep up with what's expected.
- judge2020 5y ago
  Obligatory https://xkcd.com/927/ https://xkcd.com/927/
  - garaetjjte 5y ago
    Also https://xkcd.com/1987/ https://xkcd.com/1987/
    - egberts1 5y ago
      ^THIS^
- colpabar 5y ago
  Right, but the community is different than the foundation. I would like an authoritative "this is how you do it" doc, from the foundation, about setup/packaging/dist, and I want it to work. And if said doc advises using some third party tool, then the foundation needs to fund or take over maintenance of that tool so that it continues to work and the "standard" way of doing it doesn't change every year. It seems crazy to me that we don't have this yet.
montroser 5y ago
Agreed. There has been so much churn and fragmentation with build tools over the last few years. So much complexity for what is not such a complicated problem to solve. Hopefully this is the "has to get worse before it can get better" part.
- gaborbernat 5y ago
  > So much complexity for what is not such a complicated problem to solve. As someone who actually worked weeks on their spare time, I beg to differ. This is a very complicated problem to solve. And everyone has their own opinion on how things should work, which is goverened by their narrow use case. But you see a "standard" packaging tool needs to be the opposite of narrow use case. The main reason Anacond Inc exists is because they wanted to solve this for data science. Even with them being a relatively big corporation their "solution" is not loved universally, but works ok most of the time.
- thrashh 5y ago
  I disagree. I actually think it’s a complicated problem because there’s so many issues that have to be solved all at once. What has been happening is that someone writes a build tool to solve a specific problem that they have but they neglect to solve all the other ones. Why would they? They’re not being funded. That’s what caused (and still causes) a lot of the churn in JS too.
stavros 5y ago
Nowadays I use Poetry and PyInstaller, which make things easier. For some of my projects, I produce a "compiled"/bundled executable that takes all the pain out of distribution. Here's my config, maybe it will be useful to you: https://gitlab.com/stavros/harbormaster/-/blob/master/.gitlab-ci.yml#L29 https://gitlab.com/stavros/harbormaster/-/blob/master/.gitla...
- k1rcher 5y ago
  I’ve noticed Poetry being used more and more for projects I setup locally, and I must say, it does seem to be very, very good. Minimal overhead, easy to adopt, etc. I’ll need to setup a proper workflow with it for my current projects. Being able to abstract dev/prod environments in particular are really handy.
  - stavros 5y ago
    Agreed, I really like it. It does everything and does it well, zero complaints so far.
    - bigbillheck 5y ago
      I managed to find an annoying situation with poetry in which if you specify a range of versions for a dependency, and one of your other dependencies specifies a slightly different range of versions for that dependency, it will refuse to proceed. (I was expecting it to have chosen something in the intersection of the ranges, but it did not)
      - stavros 5y ago
        Hm, yeah, that definitely sounds like a resolver bug. Did you report it? I'm curious to see if it was fixed.
        bigbillheck 5y ago
        I did not report it because I found somebody else had already reported it and it was a WONTFIX (for reasons I don't immediately recall).
game_the0ry 5y ago
> Python is such a joy to write and such a nightmare to set up, package and distribute. Agreed. Though I would not call it a "nightmare," the python ecosystem and runtime are cumbersome and annoying to work with when compared to some other languages that have put in a lot of thought into DX. Still, it could be worse.
- a_cool_username 5y ago
  Once you hit the sweet spot of developing for cross-platform (even just Linux, MacOS, and Windows) and supporting normal average-people users and have (even optional!) C dependencies, Python's packaging situation quickly deteriorates into "nightmare" territory.
  - alexmcc81 5y ago
    This is exactly my problem. I have to support Windows (a locked down corporate version) and Linux.
satyanash 5y ago
> Python is such a joy to write Try out Ruby: so many Python syntax decisions make no sense after you see the Ruby way.
- qudat 5y ago
  Im convinced the only people that like ruby are those that want to delve into metaprogramming. It might be fun for the person that constructs it but it is a nightmare as an outsider looking into a project. I know this is more rails but the fact that autoloading is even a thing speaks to the wild nature of ruby.
- gorgoiler 5y ago
  I used to love Ruby. But it’s too exciting. Python is boring, in a good way. Like mowing the lawn. Or buying potatoes. I no longer want excitement in the process, only in the results.
- stavros 5y ago
  I did, but so many Ruby syntax decisions made no sense to me.
- mrweasel 5y ago
  This might just be me not understanding Ruby well enough, but managing packages and dependecies feel even worse in Ruby.
  - enchiridion 5y ago
    Conda has become vastly better since I used it a few years ago. Worth a try again
    - mrweasel 5y ago
      That’s a pretty good idea, also a tool many Python developers already use, making the move between the two languages easier.
  - riffraff 5y ago
BiteCode_dev 5y ago
setting up is still unsolved, but: - for librairy packaging and general dep maangement, poetry is the solution - for packaging a web project and ship it on the server, shiv is the solution - for distribution, half the solution is nuitka for compiling the program. Creating the installer is now the hard part
- alexmcc81 5y ago
  I haven't looked at Nuitka in a long time. Last time I did Windows support was not great and compilation took forever. I'll try it again.
qbasic_forever 5y ago
I almost wonder if we need something like deno but for the python language--a reboot that uses the same language and syntax, but jettisons the cruft of packaging, etc. and starts over with a better developer experience. Perhaps just reference and import packages directly by URLs and forget all the pain and struggle of curating system-wide and virtual environments.
shakezula 5y ago
I've stopped recommending Python for learning programming, and instead have started recommending Go. Python's environment difficulties make it exponentially harder for beginners to focus on actual programming concepts. Things like pyenv or virtualenvs or any of those things are blackboxes to beginners. Go is simple, it's tightly coupled to version control so you can teach git at the same time in a meaningful way, and it's low level enough that CS concepts come up frequently, while being powerful enough that it will scale with a beginner's knowledge.
- ebb_earl_co 5y ago
  Interesting. Do you have a beginner’s course or tutorial that you would suggest? Something like the Rust book is wha came to mind
  - shakezula 5y ago
    I would start with https://gobyexample.com/ https://gobyexample.com/ and https://learnxinyminutes.com/docs/go/ https://learnxinyminutes.com/docs/go/ Go By Example will take a beginner from Hello World to writing a server in a short time, and you can naturally splice in other skills.
- slownews45 5y ago
  My wife started to learn to program - the setup overhead in python (going via mac -> git -> windows / linux) was crazy.
  - spicybright 5y ago
    Can you not use your system's default python and open idle? Why involve git?
    - macintux 5y ago
      On Mac, at least as of Catalina, the system default Python is 2.something.
robomartin 5y ago
> Python is such a joy to write and such a nightmare to set up, package and distribute. Here's my simple-minded metric on when this problem can be called "solved": Get a VPS from GoDaddy. Deploy your Django app as easily as you can deploy a PHP app, say, Wordpress. I have written about this before. I love Django. And yet I think that they have done a huge disservice with the development server and DB configuration out of the box. Even for a simple application, going from developing on your desktop to deploying the same application on, as an example, GoDaddy, is in a range between nightmare and impossible. I have personally given up multiple times and just said "Fuck it! Just use WP" even when I really, truly wanted to stay in the Python/Django ecosystem. Try it. Develop a simple photo album application. Get a VPS from GoDaddy and deploy it. No Heroku et. al., are not solutions. They are indicative of the problem.
- riekus 5y ago
  Sorry, but GoDaddy is a disgrace, and your comment makes no sense at all, you are comparing WP with Django. And instead of recommending VPS as a solution, you recommend the worst party offering these services? GoDaddy should burn in hell for their shady shit.
  - selfhoster11 5y ago
    GoDaddy is shady and I would not recommend it ever, but they are a good representative of the target market - something people pick despite the shadiness and inflexibility, because they require zero technical competency to get something up and running.
  - robomartin 5y ago
    GoDaddy is a disgrace? It's shady shit? C'mon, get some perspective. I have been using GoDaddy for, well, I forget how long, maybe two decades. I have run websites for multiple companies from their servers. I can't remember a single issue. Not one. Even hosting and managing email. Frankly, I don't know what you are talking about. I have also used Linode, AWS, Dreamhost, Rackspace and a bunch of others for different kinds of work. Not sure where the hatred for GoDaddy comes from. I also know a bunch of people using GoDaddy servers. I think people repeat shit just because they think they sound "cool" and yet have no clue what they are talking about. Also, GoDaddy support, on the rare occasions they are needed, has always been top notch. So, yeah, no clue what you are talking about.
deleted 5y ago
[deleted]
fifilura 5y ago
> Python is such a joy to write and such a nightmare to set up, package and distribute. Python is such a joy to write and such a nightmare to maintain, set up, package and distribute. Fixed that for you.

3np 5y ago

Is this really an issue? While there are a lot of unmaintained packages doing it (which I assume will still be working for some years?), I don't think I've seen a recent package doing it in years. In the off-chance you end up depending on that legacy code, repackaging that one or two dependencies shouldn't be that big of a deal.

pganssle 5y ago
I think that it is true that I should have written this article or something like it years ago, but there's still a long tail of people who don't even know that invoking setup.py is bad, or that the goal is to get rid of all setup.py commands. Probably the biggest stragglers will be build scripts, and as recently as mid-2020, the answer to "how do I build distributable artifacts without using setup.py" was still "you can do it but there's no good definitive answer": https://stackoverflow.com/q/58753970/467366 https://stackoverflow.com/q/58753970/467366 There's also been a lot of mixed messaging in this space, so I think there's a decent population of people who think that the way to stop invoking setup.py is to stop using setuptools entirely, mixing up "don't invoke setup.py" with "use a backend that doesn't have a setup.py". But you are right to say that this has gotten a lot better over time and the word is slowly getting out.
- 3np 5y ago
  To be honest, as more of a casual user of Python I wasn't aware of the situation at all so I found your post very informative! I did always find the setup.py approach misguided for several reasons, so it's more of a casual observation I've done that I thankfully rarely encounter it these days.

bjt2n3904 5y ago

What made python beat out Ruby? There was one way to do things. It was simple, and it worked. Then python 3 wanted to become JavaScript. Async, Unicode, floating point division. Move fast and break things. No one way to do things, it's however you want! I had to explain python deployments to a novice the other day. What a nightmare. Python 3.5 to 3.8 just kept drastically changing things. My first experience wasn't pleasant, and for the first time I needed virtual environments and custom apt repositories to install multiple versions of python. But it finally seems things have settled a bit. Here's hoping the Python cultural revolution settles down, and stops trying to be a trendy language, and instead a dependable one.

_joel 5y ago
I normally use https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv when I need to test multiple versions.
joeberon 5y ago
I don't see how any of that is relevant to the article, but the reason python beat out ruby was almost entirely due to ruby being really really slow.
- samhw 5y ago
  > the reason python beat out ruby was almost entirely due to ruby being really really slow Relative to Python, that language which notoriously makes C look like a sloth on training wheels...
  - joeberon 5y ago
    Python is much faster than MRI Ruby used to be back in the day
  - elcomet 5y ago
    So you might imagine what ruby felt like ...
  - spicybright 5y ago

joeberon 5y ago

Python distribution is honestly horrific. The worst thing is that I actually see people defending it or saying it isn't that bad if you simply learn it "properly" but I have no idea what they mean when they say this.

a_humean 5y ago
It just means they have never worked in another langauge that does this any better.
- mrweasel 5y ago
  It could be better, but it doesn’t feel worse than .Net or Go, and it’s much better than Javascript/Node. Personally I’m also confused by Ruby. I guess Java is sort of okay, but which languages do people feel have excellent package management?
  - a_humean 5y ago
    Rust's cargo is the high water mark today (and is a substantial reason for Rust's popularity). Ruby's rubygems, node's npm, and the elm package manager all get different things right and wrong, but at least they are a common approach that is somewhat sane. Python is total anarchy. I think most people would disagree with you about .Net, Go, and Javascript today vs python.
  - joeberon 5y ago
    I would guess that most people feel like .Net, Go, and Javascript have very easy distribution and package management
  - recursive 5y ago
    .net is just nuget. It seems pretty straightforward. The last time I looked for a "blessed" solution in python, it was some third part solution surrounded in some kind of internet/twitter drama.
  - OJFord 5y ago

dilawar 5y ago

Once you get into distributing python package, it is such a nightmare. Relative imports were also not easy to understand. And C/C++ extensions in your package makes things even worse. CMake for extension, setup.py for python of you hope to build cross-platform. Pybind11 helped a bit but for each python version you have to compile the wheel again, even with manylinux docker image. Imagine learning docker so you can release a wheel! The other language I often use, C++, is also no better. I am drawn to Rust and Nim lately (not entirely because of these issues).

CapmCrackaWaka 5y ago

As someone who actively contributes to Python packages using the setuptools cli interface, this is jarring. I had no idea. I guess I'll move my build process to some other tools. I started contributing to Python about a year ago. At that time, all of the results when you google "how to build a python package" refer to the setuptools CLI with setup.py. I put it in my workflow and bam, I've been using it ever since without incident.

webknjaz 5y ago
Save this tutorial to your bookmarks, then: https://packaging.python.org/tutorials/packaging-projects/#configuring-metadata https://packaging.python.org/tutorials/packaging-projects/#c...
gaborbernat 5y ago
Sadly there's no easy way to mark all those Google searches out of date... One of the big goals of this project is to spread the knowledge though, so you reading it means success.
- monkeybutton 5y ago
  I find myself frequently using the date filter in Google when searching programming topics. Especially if if I'm looking up an error message, almost anything older than a year is noise.
cdrt 5y ago
You don't need to completely change if you don't want to. setuptools will most likely continue supporting the old way for a long time since there is so much Python code out there that will probably never update to the new standards. Also, it takes only minimal changes to bring a setuptools-based project into the present. You just need to add a `pyproject.toml` file to your project with the following: [build-system] requires = ["setuptools >= 40.9.0", "wheel"] build-backend = "setuptools.build_meta" Then when it's time to distribute the project, instead of `setup.py sdist` and `setup.py bdist_wheel`, you can just use the official build[1] tool like so: pyproject-build --sdist --wheel And that's all there is to it. Once that's in place and if you're feeling frisky, you can try taking advantage of new setuptools features that can allow you to eliminate `setup.py` entirely and specify everything in `setup.cfg`[2] so that all your project's information is static and no code needs to be run when installing it, assuming your project doesn't contain any compiled code. As and added benefit, by putting pyproject.toml in your projects now, if you wanted to switch over to another tool like poetry or flit later on, the act of packaging the project doesn't have to change as long as you update the pyproject.toml file with the new build system. The same `pyproject-build` command will work for all three systems. [1] https://pypi.org/project/build/ https://pypi.org/project/build/ [2] https://setuptools.pypa.io/en/latest/userguide/declarative_config.html https://setuptools.pypa.io/en/latest/userguide/declarative_c...

est 5y ago

Or simply copy everything into your project under the "lib" path, change your PYTHONPATH and done. (aka "vendoring")

carapace 5y ago

I think Python dependency/build/install/test stuff would be helped greatly by a library or module that did for it what pathlib did for file & path manipulation, eh?

carapace 5y ago

Py3 feels to me like a massive case of "screw backwards compatibility" for very little gain. I feel like Python the language peaked somewhere between 2.4 and 2.7. Python was much slower than most other languages, but in return you got great readability ("runnable pseudocode") and rapid prototyping. Now, with all the additions to the language (more syntax and features, static typing), those advantages are weakening but you're still not getting the advantages of faster compiled languages. It seems to me that Go or Nim (or D or Zig or ...) are preferable for readability, and that Rust is more worth one's time if you don't mind the complexity.

musicale 5y ago
In an alternate universe, Python 3 would have - not broken backward compatibility (an astonishing value-destroying move with huge externalities that are still being felt today) - provided a standard and improved packaging system
1_player 5y ago
I agree 100% with you. I built my career on top of Python, and 2.7 is the last version I've used. I don't like that Python 3 is adding any feature under the sun for whatever reason. Have you seen their sad excuse of pattern match proposal? And packaging is still a bloody mess. There's worse languages, it definitely gets the job done, but I'd rather stick with better languages.

sleepysysadmin 5y ago

I've never looked into setuptools. I just 'python3 setup.py' for all my python projects. What benefit does this provide?

eesmith 5y ago

> you can remove your setup.py file and replace it with setuptools' own declarative configuration format Not if you have a Python/C extension. Not if your build step includes code generation (other than Cython). Not if you have special per-file compilation args. Also, in the declarative configuration format, how do I enable/disable compile-time flags like "compile with OpenMP" ? Yes, I have these, and it really irks me reading all of these "you don't need setup.py" when as far as I can tell, I can't.

bobbyi 5y ago
> Not if your build step includes code generation (other than Cython). Is it possible for Cython? Their guide shows using setup.py: https://cython.readthedocs.io/en/latest/src/tutorial/cython_tutorial.html#cython-hello-world https://cython.readthedocs.io/en/latest/src/tutorial/cython_...
- eesmith 5y ago
  https://setuptools.pypa.io/en/latest/userguide/distribution.html#distributing-extensions-compiled-with-cython https://setuptools.pypa.io/en/latest/userguide/distribution.... > setuptools will detect at build time whether Cython is installed or not. If Cython is not found setuptools will ignore pyx files. > To ensure Cython is available, include Cython in the build-requires section of your pyproject.toml: [build-system] requires=[..., "cython"] > Built with pip 10 or later, that declaration is sufficient to include Cython in the build. For broader compatibility, declare the dependency in your setup-requires of setup.cfg:
pganssle 5y ago
Sorry, I could have been more clear there, and I've updated the post. I have the "curse of knowledge" and took is as assumed that you can only do this if your particular build configuration fits in setup.cfg. For me, the fact that you have the option to use a setup.py is one of the biggest selling points of setuptools. For other build backends, it's common to have a great experience when you are on the "happy path" and then have no options when you have any of a million exceedingly rare deviations. So if I had a complicated build like you have (and I do have some mildly complicated builds where this is the case), I'd look at that bullet point and think, "Ah, that doesn't apply to this, but it's good to know for my simpler projects". That said, it would be really nice if at least the C-extension part of this worked with the declarative format. My OSS time has been severely curtailed lately and I'm spread way too thin, so I probably won't be able to execute on this, but my long term vision for setuptools has always been that most common build configurations should be achievable using declarative config only, and everything else has setup.py as an escape hatch. This is similar to the way almost all Rust projects are configured with Cargo.toml, but in rare cases you can use build.rs to execute some Rust code as part of the build.

bobbyi 5y ago

The only case where I still invoke setup.py directly is "python setup.py test". I already knew it's deprecated, but don't understand what I'm supposed to do instead. The article recommends tox, but how did tox get installed? I can currently create and activate a new empty virtualenv, "pip install -U pip", clone and cd into my project, and run "python setup.py test". My test dependencies are automatically setup and my preferred runner (pytest) is used. If I change those in the future, the same commands will keep working because I am configuring those in setup.cfg, etc. If I am now expected to manually "pip install tox" before testing my project and change that in the future if we start using something else, isn't that a step backward to the situation pyproject.toml et al were supposed to solve with needing to manually install specific stuff before using a project?

orf 5y ago
Use poetry. “poetry run pytest”, or “poetry shell” then do whatever.
pganssle 5y ago
This is at worst a lateral move — you need something installed that handles your test invocation for you. It could be `make`, `tox`, `nox`, `poetry` or whatever. For a number of reasons (detailed in the article), that thing shouldn't be `setuptools`. Even if the setuptools maintainers wanted to continue supporting this use case, setuptools is uniquely unsuited to working with this, because of the way its hooks don't fire until all its dependencies have been imported already. With `tox` you just need to make sure you have `tox` installed, and you can even specify a minimum version in the `tox.ini` file. Additionally, tox doesn't need to be installed in your Python environment (in fact, it's an implementation detail that it's Python at all — if it were instead a Go executable, it would work the same). You just need the tool available to be invoked. This means you can install it with pipx, in its own virtual environment, through your system package manager, whatever is most convenient for you.

loloquwowndueo 5y ago

“ I realize that this article is quite long, and unfortunately this is the short version. Much nuance and history is lost here, even though the post is already long enough that it is destined for the limbo of a bookmarks folder called "to read later". Despite the fact that no one will read more than the next 3 paragraphs…” - pro tip - if you want to make the article shorter and easier to read don’t spend an entire paragraph blabbering about how long it is and how nobody reads anymore. I bet if the author had aimed to be succinct, the article would be half the current size and much quicker to read and understand.

hermitsings 5y ago
I've done just that: put it in bookmarks for read later.

scrollaway 5y ago

I'm surprised the article, for being so thorough, doesn't have a single mention of Poetry. Which in terms or "works for most people, for most use cases", really has that down. And of course uses the same PEP standards as the other tools mentioned.

webknjaz 5y ago
It also has a lot of downsides which is why I avoid it, for example. The purpose of the article is not to point at other tools but to show how to use setuptools better.