9 ms·

Python is such a joy to write and such a nightmare to set up, package and distribute. The Python Foundation needs to adopt or create standard and supported way

Python is such a joy to write and such a nightmare to set up, package and distribute. The Python Foundation needs to adopt or create standard and supported ways of doing things. I spend so much time dealing with issues that should have been solved a long time ago. Special shout out to the PyInstaller devs for providing so much value while constantly fighting to deal with upstream changes.

danjac 5y ago
> The Python Foundation needs to adopt or create standard and supported ways of doing things The Python community has adopted or created a standard way of doing things for the (checks calendar) fourth or fifth time now.
- alexmcc81 5y ago
  I used to recommend articles with titles like "modern Python package structure and tooling" to graduates. I stopped because the tools (poetry, pipenv etc) were changing so much. I try to fit as much of our workflows into hand made templates and PyCharm now.
  - sseagull 5y ago
    There was a post here a while ago about packaging or virtual environments. Within a few minutes, there were quite a few responses about better ways to do it. Unfortunately, each response had a different recommendation. I wish I could find that thread again.
    - dendrite9 5y ago
      Maybe this one? https://news.ycombinator.com/item?id=26733423 https://news.ycombinator.com/item?id=26733423 We have been talking about how to support customers who want to use python. At least for me it isn't very appealing, seems like it could take a lot of attention to keep up with what's expected.
- judge2020 5y ago
  Obligatory https://xkcd.com/927/ https://xkcd.com/927/
  - garaetjjte 5y ago
    Also https://xkcd.com/1987/ https://xkcd.com/1987/
    - egberts1 5y ago
      ^THIS^
- colpabar 5y ago
  Right, but the community is different than the foundation. I would like an authoritative "this is how you do it" doc, from the foundation, about setup/packaging/dist, and I want it to work. And if said doc advises using some third party tool, then the foundation needs to fund or take over maintenance of that tool so that it continues to work and the "standard" way of doing it doesn't change every year. It seems crazy to me that we don't have this yet.
  - gaborbernat 5y ago
    It's less crazy than you think. The foundation historically never had developer employees (today has just 1 - a CPython core developer that started 3 months ago). The only way taking over a project and making it de facto standard would be to have (IMHO at least) 5 full time employees working on it. That's a big investment the foundation doesn't have and no corporation commited to support that (for at least 3-4 years). Also, there's the huge backlash the PSF would have to deal with from people who inevitable don't like the choosen standard.
  - qbasic_forever 5y ago
    There have been like three or four iterations of the "here's the official way to package and distribute python code" document. Here's the latest: https://packaging.python.org/ https://packaging.python.org/
    - eesmith 5y ago
      My use case - distributing a Python package containing Python/C extensions - is described as "Incomplete / Last Reviewed 2013-12-08".
      - elevation 5y ago
        I maintain a library implemented mostly in python with some simple cpython extension modules for performance. Since you can't run unit tests against an extension module before you've built it, my Makefile invokes setup.py just to get things compiled: python setup.py build_ext --inplace I don't see a valid replacement for this invocation, but fortunately, it's the only one. When I include the library from a project that uses it, the same `pip install' command that brings the library in also performs all the compilation described by my setup.py. All I'd need to move away from deprecated syntax entirely is a way to build the module when I'm testing it within it's own source directory. Until then I won't be changing anything.
- selfhoster11 5y ago
  The difference between the community adopting a "standard", and something becoming the standard/being included into the langage by a direct decree is enormous. Especially with Python's philosophy.
montroser 5y ago
Agreed. There has been so much churn and fragmentation with build tools over the last few years. So much complexity for what is not such a complicated problem to solve. Hopefully this is the "has to get worse before it can get better" part.
- gaborbernat 5y ago
  > So much complexity for what is not such a complicated problem to solve. As someone who actually worked weeks on their spare time, I beg to differ. This is a very complicated problem to solve. And everyone has their own opinion on how things should work, which is goverened by their narrow use case. But you see a "standard" packaging tool needs to be the opposite of narrow use case. The main reason Anacond Inc exists is because they wanted to solve this for data science. Even with them being a relatively big corporation their "solution" is not loved universally, but works ok most of the time.
- thrashh 5y ago
  I disagree. I actually think it’s a complicated problem because there’s so many issues that have to be solved all at once. What has been happening is that someone writes a build tool to solve a specific problem that they have but they neglect to solve all the other ones. Why would they? They’re not being funded. That’s what caused (and still causes) a lot of the churn in JS too.
stavros 5y ago
Nowadays I use Poetry and PyInstaller, which make things easier. For some of my projects, I produce a "compiled"/bundled executable that takes all the pain out of distribution. Here's my config, maybe it will be useful to you: https://gitlab.com/stavros/harbormaster/-/blob/master/.gitlab-ci.yml#L29 https://gitlab.com/stavros/harbormaster/-/blob/master/.gitla...
- k1rcher 5y ago
  I’ve noticed Poetry being used more and more for projects I setup locally, and I must say, it does seem to be very, very good. Minimal overhead, easy to adopt, etc. I’ll need to setup a proper workflow with it for my current projects. Being able to abstract dev/prod environments in particular are really handy.
  - stavros 5y ago
    Agreed, I really like it. It does everything and does it well, zero complaints so far.
    - bigbillheck 5y ago
      I managed to find an annoying situation with poetry in which if you specify a range of versions for a dependency, and one of your other dependencies specifies a slightly different range of versions for that dependency, it will refuse to proceed. (I was expecting it to have chosen something in the intersection of the ranges, but it did not)
      - stavros 5y ago
        Hm, yeah, that definitely sounds like a resolver bug. Did you report it? I'm curious to see if it was fixed.
        bigbillheck 5y ago
        I did not report it because I found somebody else had already reported it and it was a WONTFIX (for reasons I don't immediately recall).
game_the0ry 5y ago
> Python is such a joy to write and such a nightmare to set up, package and distribute. Agreed. Though I would not call it a "nightmare," the python ecosystem and runtime are cumbersome and annoying to work with when compared to some other languages that have put in a lot of thought into DX. Still, it could be worse.
- a_cool_username 5y ago
  Once you hit the sweet spot of developing for cross-platform (even just Linux, MacOS, and Windows) and supporting normal average-people users and have (even optional!) C dependencies, Python's packaging situation quickly deteriorates into "nightmare" territory.
  - alexmcc81 5y ago
    This is exactly my problem. I have to support Windows (a locked down corporate version) and Linux.
satyanash 5y ago
> Python is such a joy to write Try out Ruby: so many Python syntax decisions make no sense after you see the Ruby way.
- qudat 5y ago
  Im convinced the only people that like ruby are those that want to delve into metaprogramming. It might be fun for the person that constructs it but it is a nightmare as an outsider looking into a project. I know this is more rails but the fact that autoloading is even a thing speaks to the wild nature of ruby.
- gorgoiler 5y ago
  I used to love Ruby. But it’s too exciting. Python is boring, in a good way. Like mowing the lawn. Or buying potatoes. I no longer want excitement in the process, only in the results.
- stavros 5y ago
  I did, but so many Ruby syntax decisions made no sense to me.
- mrweasel 5y ago
  This might just be me not understanding Ruby well enough, but managing packages and dependecies feel even worse in Ruby.
  - enchiridion 5y ago
    Conda has become vastly better since I used it a few years ago. Worth a try again
    - mrweasel 5y ago
      That’s a pretty good idea, also a tool many Python developers already use, making the move between the two languages easier.
  - riffraff 5y ago
BiteCode_dev 5y ago
setting up is still unsolved, but: - for librairy packaging and general dep maangement, poetry is the solution - for packaging a web project and ship it on the server, shiv is the solution - for distribution, half the solution is nuitka for compiling the program. Creating the installer is now the hard part
- alexmcc81 5y ago
  I haven't looked at Nuitka in a long time. Last time I did Windows support was not great and compilation took forever. I'll try it again.
qbasic_forever 5y ago
I almost wonder if we need something like deno but for the python language--a reboot that uses the same language and syntax, but jettisons the cruft of packaging, etc. and starts over with a better developer experience. Perhaps just reference and import packages directly by URLs and forget all the pain and struggle of curating system-wide and virtual environments.
shakezula 5y ago
I've stopped recommending Python for learning programming, and instead have started recommending Go. Python's environment difficulties make it exponentially harder for beginners to focus on actual programming concepts. Things like pyenv or virtualenvs or any of those things are blackboxes to beginners. Go is simple, it's tightly coupled to version control so you can teach git at the same time in a meaningful way, and it's low level enough that CS concepts come up frequently, while being powerful enough that it will scale with a beginner's knowledge.
- ebb_earl_co 5y ago
  Interesting. Do you have a beginner’s course or tutorial that you would suggest? Something like the Rust book is wha came to mind
  - shakezula 5y ago
    I would start with https://gobyexample.com/ https://gobyexample.com/ and https://learnxinyminutes.com/docs/go/ https://learnxinyminutes.com/docs/go/ Go By Example will take a beginner from Hello World to writing a server in a short time, and you can naturally splice in other skills.
- slownews45 5y ago
  My wife started to learn to program - the setup overhead in python (going via mac -> git -> windows / linux) was crazy.
  - spicybright 5y ago
    Can you not use your system's default python and open idle? Why involve git?
    - macintux 5y ago
      On Mac, at least as of Catalina, the system default Python is 2.something.
robomartin 5y ago
> Python is such a joy to write and such a nightmare to set up, package and distribute. Here's my simple-minded metric on when this problem can be called "solved": Get a VPS from GoDaddy. Deploy your Django app as easily as you can deploy a PHP app, say, Wordpress. I have written about this before. I love Django. And yet I think that they have done a huge disservice with the development server and DB configuration out of the box. Even for a simple application, going from developing on your desktop to deploying the same application on, as an example, GoDaddy, is in a range between nightmare and impossible. I have personally given up multiple times and just said "Fuck it! Just use WP" even when I really, truly wanted to stay in the Python/Django ecosystem. Try it. Develop a simple photo album application. Get a VPS from GoDaddy and deploy it. No Heroku et. al., are not solutions. They are indicative of the problem.
- riekus 5y ago
  Sorry, but GoDaddy is a disgrace, and your comment makes no sense at all, you are comparing WP with Django. And instead of recommending VPS as a solution, you recommend the worst party offering these services? GoDaddy should burn in hell for their shady shit.
  - selfhoster11 5y ago
    GoDaddy is shady and I would not recommend it ever, but they are a good representative of the target market - something people pick despite the shadiness and inflexibility, because they require zero technical competency to get something up and running.
  - robomartin 5y ago
    GoDaddy is a disgrace? It's shady shit? C'mon, get some perspective. I have been using GoDaddy for, well, I forget how long, maybe two decades. I have run websites for multiple companies from their servers. I can't remember a single issue. Not one. Even hosting and managing email. Frankly, I don't know what you are talking about. I have also used Linode, AWS, Dreamhost, Rackspace and a bunch of others for different kinds of work. Not sure where the hatred for GoDaddy comes from. I also know a bunch of people using GoDaddy servers. I think people repeat shit just because they think they sound "cool" and yet have no clue what they are talking about. Also, GoDaddy support, on the rare occasions they are needed, has always been top notch. So, yeah, no clue what you are talking about.
deleted 5y ago
[deleted]
fifilura 5y ago
> Python is such a joy to write and such a nightmare to set up, package and distribute. Python is such a joy to write and such a nightmare to maintain, set up, package and distribute. Fixed that for you.

danjac 5y ago

> The Python Foundation needs to adopt or create standard and supported ways of doing things The Python community has adopted or created a standard way of doing things for the (checks calendar) fourth or fifth time now.

alexmcc81 5y ago
I used to recommend articles with titles like "modern Python package structure and tooling" to graduates. I stopped because the tools (poetry, pipenv etc) were changing so much. I try to fit as much of our workflows into hand made templates and PyCharm now.
- sseagull 5y ago
  There was a post here a while ago about packaging or virtual environments. Within a few minutes, there were quite a few responses about better ways to do it. Unfortunately, each response had a different recommendation. I wish I could find that thread again.
  - dendrite9 5y ago
    Maybe this one? https://news.ycombinator.com/item?id=26733423 https://news.ycombinator.com/item?id=26733423 We have been talking about how to support customers who want to use python. At least for me it isn't very appealing, seems like it could take a lot of attention to keep up with what's expected.
judge2020 5y ago
Obligatory https://xkcd.com/927/ https://xkcd.com/927/
- garaetjjte 5y ago
  Also https://xkcd.com/1987/ https://xkcd.com/1987/
  - egberts1 5y ago
    ^THIS^
colpabar 5y ago
Right, but the community is different than the foundation. I would like an authoritative "this is how you do it" doc, from the foundation, about setup/packaging/dist, and I want it to work. And if said doc advises using some third party tool, then the foundation needs to fund or take over maintenance of that tool so that it continues to work and the "standard" way of doing it doesn't change every year. It seems crazy to me that we don't have this yet.
- gaborbernat 5y ago
  It's less crazy than you think. The foundation historically never had developer employees (today has just 1 - a CPython core developer that started 3 months ago). The only way taking over a project and making it de facto standard would be to have (IMHO at least) 5 full time employees working on it. That's a big investment the foundation doesn't have and no corporation commited to support that (for at least 3-4 years). Also, there's the huge backlash the PSF would have to deal with from people who inevitable don't like the choosen standard.
- qbasic_forever 5y ago
  There have been like three or four iterations of the "here's the official way to package and distribute python code" document. Here's the latest: https://packaging.python.org/ https://packaging.python.org/
  - eesmith 5y ago
    My use case - distributing a Python package containing Python/C extensions - is described as "Incomplete / Last Reviewed 2013-12-08".
    - elevation 5y ago
      I maintain a library implemented mostly in python with some simple cpython extension modules for performance. Since you can't run unit tests against an extension module before you've built it, my Makefile invokes setup.py just to get things compiled: python setup.py build_ext --inplace I don't see a valid replacement for this invocation, but fortunately, it's the only one. When I include the library from a project that uses it, the same `pip install' command that brings the library in also performs all the compilation described by my setup.py. All I'd need to move away from deprecated syntax entirely is a way to build the module when I'm testing it within it's own source directory. Until then I won't be changing anything.
selfhoster11 5y ago
The difference between the community adopting a "standard", and something becoming the standard/being included into the langage by a direct decree is enormous. Especially with Python's philosophy.

montroser 5y ago

Agreed. There has been so much churn and fragmentation with build tools over the last few years. So much complexity for what is not such a complicated problem to solve. Hopefully this is the "has to get worse before it can get better" part.

gaborbernat 5y ago
> So much complexity for what is not such a complicated problem to solve. As someone who actually worked weeks on their spare time, I beg to differ. This is a very complicated problem to solve. And everyone has their own opinion on how things should work, which is goverened by their narrow use case. But you see a "standard" packaging tool needs to be the opposite of narrow use case. The main reason Anacond Inc exists is because they wanted to solve this for data science. Even with them being a relatively big corporation their "solution" is not loved universally, but works ok most of the time.
thrashh 5y ago
I disagree. I actually think it’s a complicated problem because there’s so many issues that have to be solved all at once. What has been happening is that someone writes a build tool to solve a specific problem that they have but they neglect to solve all the other ones. Why would they? They’re not being funded. That’s what caused (and still causes) a lot of the churn in JS too.

stavros 5y ago

Nowadays I use Poetry and PyInstaller, which make things easier. For some of my projects, I produce a "compiled"/bundled executable that takes all the pain out of distribution. Here's my config, maybe it will be useful to you: https://gitlab.com/stavros/harbormaster/-/blob/master/.gitlab-ci.yml#L29 https://gitlab.com/stavros/harbormaster/-/blob/master/.gitla...

k1rcher 5y ago
I’ve noticed Poetry being used more and more for projects I setup locally, and I must say, it does seem to be very, very good. Minimal overhead, easy to adopt, etc. I’ll need to setup a proper workflow with it for my current projects. Being able to abstract dev/prod environments in particular are really handy.
- stavros 5y ago
  Agreed, I really like it. It does everything and does it well, zero complaints so far.
  - bigbillheck 5y ago
    I managed to find an annoying situation with poetry in which if you specify a range of versions for a dependency, and one of your other dependencies specifies a slightly different range of versions for that dependency, it will refuse to proceed. (I was expecting it to have chosen something in the intersection of the ranges, but it did not)
    - stavros 5y ago
      Hm, yeah, that definitely sounds like a resolver bug. Did you report it? I'm curious to see if it was fixed.
      - bigbillheck 5y ago
        I did not report it because I found somebody else had already reported it and it was a WONTFIX (for reasons I don't immediately recall).

game_the0ry 5y ago

> Python is such a joy to write and such a nightmare to set up, package and distribute. Agreed. Though I would not call it a "nightmare," the python ecosystem and runtime are cumbersome and annoying to work with when compared to some other languages that have put in a lot of thought into DX. Still, it could be worse.

a_cool_username 5y ago
Once you hit the sweet spot of developing for cross-platform (even just Linux, MacOS, and Windows) and supporting normal average-people users and have (even optional!) C dependencies, Python's packaging situation quickly deteriorates into "nightmare" territory.
- alexmcc81 5y ago
  This is exactly my problem. I have to support Windows (a locked down corporate version) and Linux.

satyanash 5y ago

> Python is such a joy to write Try out Ruby: so many Python syntax decisions make no sense after you see the Ruby way.

qudat 5y ago
Im convinced the only people that like ruby are those that want to delve into metaprogramming. It might be fun for the person that constructs it but it is a nightmare as an outsider looking into a project. I know this is more rails but the fact that autoloading is even a thing speaks to the wild nature of ruby.
gorgoiler 5y ago
I used to love Ruby. But it’s too exciting. Python is boring, in a good way. Like mowing the lawn. Or buying potatoes. I no longer want excitement in the process, only in the results.
stavros 5y ago
I did, but so many Ruby syntax decisions made no sense to me.
mrweasel 5y ago
This might just be me not understanding Ruby well enough, but managing packages and dependecies feel even worse in Ruby.
- enchiridion 5y ago
  Conda has become vastly better since I used it a few years ago. Worth a try again
  - mrweasel 5y ago
    That’s a pretty good idea, also a tool many Python developers already use, making the move between the two languages easier.
- riffraff 5y ago

BiteCode_dev 5y ago

setting up is still unsolved, but: - for librairy packaging and general dep maangement, poetry is the solution - for packaging a web project and ship it on the server, shiv is the solution - for distribution, half the solution is nuitka for compiling the program. Creating the installer is now the hard part

alexmcc81 5y ago
I haven't looked at Nuitka in a long time. Last time I did Windows support was not great and compilation took forever. I'll try it again.

qbasic_forever 5y ago

I almost wonder if we need something like deno but for the python language--a reboot that uses the same language and syntax, but jettisons the cruft of packaging, etc. and starts over with a better developer experience. Perhaps just reference and import packages directly by URLs and forget all the pain and struggle of curating system-wide and virtual environments.

shakezula 5y ago

I've stopped recommending Python for learning programming, and instead have started recommending Go. Python's environment difficulties make it exponentially harder for beginners to focus on actual programming concepts. Things like pyenv or virtualenvs or any of those things are blackboxes to beginners. Go is simple, it's tightly coupled to version control so you can teach git at the same time in a meaningful way, and it's low level enough that CS concepts come up frequently, while being powerful enough that it will scale with a beginner's knowledge.

ebb_earl_co 5y ago
Interesting. Do you have a beginner’s course or tutorial that you would suggest? Something like the Rust book is wha came to mind
- shakezula 5y ago
  I would start with https://gobyexample.com/ https://gobyexample.com/ and https://learnxinyminutes.com/docs/go/ https://learnxinyminutes.com/docs/go/ Go By Example will take a beginner from Hello World to writing a server in a short time, and you can naturally splice in other skills.
slownews45 5y ago
My wife started to learn to program - the setup overhead in python (going via mac -> git -> windows / linux) was crazy.
- spicybright 5y ago
  Can you not use your system's default python and open idle? Why involve git?
  - macintux 5y ago
    On Mac, at least as of Catalina, the system default Python is 2.something.

robomartin 5y ago

> Python is such a joy to write and such a nightmare to set up, package and distribute. Here's my simple-minded metric on when this problem can be called "solved": Get a VPS from GoDaddy. Deploy your Django app as easily as you can deploy a PHP app, say, Wordpress. I have written about this before. I love Django. And yet I think that they have done a huge disservice with the development server and DB configuration out of the box. Even for a simple application, going from developing on your desktop to deploying the same application on, as an example, GoDaddy, is in a range between nightmare and impossible. I have personally given up multiple times and just said "Fuck it! Just use WP" even when I really, truly wanted to stay in the Python/Django ecosystem. Try it. Develop a simple photo album application. Get a VPS from GoDaddy and deploy it. No Heroku et. al., are not solutions. They are indicative of the problem.

riekus 5y ago
Sorry, but GoDaddy is a disgrace, and your comment makes no sense at all, you are comparing WP with Django. And instead of recommending VPS as a solution, you recommend the worst party offering these services? GoDaddy should burn in hell for their shady shit.
- selfhoster11 5y ago
  GoDaddy is shady and I would not recommend it ever, but they are a good representative of the target market - something people pick despite the shadiness and inflexibility, because they require zero technical competency to get something up and running.
- robomartin 5y ago
  GoDaddy is a disgrace? It's shady shit? C'mon, get some perspective. I have been using GoDaddy for, well, I forget how long, maybe two decades. I have run websites for multiple companies from their servers. I can't remember a single issue. Not one. Even hosting and managing email. Frankly, I don't know what you are talking about. I have also used Linode, AWS, Dreamhost, Rackspace and a bunch of others for different kinds of work. Not sure where the hatred for GoDaddy comes from. I also know a bunch of people using GoDaddy servers. I think people repeat shit just because they think they sound "cool" and yet have no clue what they are talking about. Also, GoDaddy support, on the rare occasions they are needed, has always been top notch. So, yeah, no clue what you are talking about.

deleted 5y ago

[deleted]

fifilura 5y ago

> Python is such a joy to write and such a nightmare to set up, package and distribute. Python is such a joy to write and such a nightmare to maintain, set up, package and distribute. Fixed that for you.