16 ms·
Python has a lot of problems that really slow down development, but they are all fixable. The biggest issue, in my opinion, is in dependency management. Python
by blaisio 7y ago
Python has a lot of problems that really slow down development, but they are all fixable.
The biggest issue, in my opinion, is in dependency management. Python has a horrible dependency management system, from top-to-bottom.
Why do I need to make a "virtual environment" to have separate dependencies, and then source it my shell?
Why do I need to manually add version numbers to a file?
Why isn't there any builtin way to automatically define a lock file (currently, most Python projects just don't even specify indirect dependency versions, many Python developers probably don't even realize this is an issue!!!!!)?
Why can't I parallelize dependency installation?
Why isn't there a builtin way to create a redistributable executable with all my dependencies?
Why do I need to have fresh copies of my dependencies, even if they are the same versions, in each virtual environment?
There is so much chaos, I've seen very few projects that actually have reproducible builds. Most people just cross their fingers and hope dependencies don't change, and they just "deal with" the horrible kludge that is a virtual environment.
We need official support for a modern package management system, from the Python org itself. Third party solutions don't cut it, because they just end up being incompatible with each other.
Example: if the Python interpreter knew just a little bit about dependencies, it could pull in the correct version from a global cache - no need to reinstall the same module over and over again, just use the shared copy. Imagine how many CPU cycles would be saved. No more need for special wrapper tools like "tox".
- brainless 7y agoI understand what you are saying but "Python has a lot of problems" is not really a list of "only" dependency management issues. Once your project is setup, dependency management is what you do once in two weeks perhaps. Rest is just writing code.
- pietroglyph 7y agoHaving such a basic part of a programming language be awful is inexcusable. It's not just that it takes a lot of time; even if it took no extra time, you're still wasting extra space on your computer, risking breakage on external updates, and compromising security because you can't even tell what code you're running.
- jnwatson 7y ago"Inexcusable". You know this started in 1989 and its major competition was Perl and TCL, right? The same issues exist in C, C++, Java and nobody seems to be complaining about those at the same volume.
- futureastronaut 7y agoPeople here are just very dramatic. I think I need to not read (or write) HN comments for a long time, there is some serious distorted reality about.
- chii 7y agothis issue outlined above doesn't exist in java, as long as you use maven.
- deleted 7y ago[deleted]
- astonex 7y ago>Most people just cross their fingers and hope dependencies don't change Is there anything wrong with pip freeze > requirements.txt and then pip install -r requirements.txt ? This would install the exact versions
- zys5945 7y agoI think he is referring to indirect dependencies
- thaumasiotes 7y ago>>> Why isn't there any builtin way to automatically define a lock file pip isn't actually part of Python proper.
- jnwatson 7y agoIt isn't part of the Python executable, but it is part of the standard distribution.
- icebraining 7y agoNowadays, a regular installation of CPython lets you run "python -m pip". It's quite part of it.
- thaumasiotes 7y agoYou can do that with any library. You can issue Django commands by running `python -m django`; that doesn't change the fact that Django is a completely separate project from Python.
- voodoochicken 7y agoYes, those are caught when using pip freeze.
- zys5945 7y ago
- deleted 7y ago[deleted]
- fnord77 7y agoOne thing people overlook with interpreted languages is the environmental impact in terms of extra electricity used.
- pts_ 7y agoGosh yes. These slow languages offload thinking to burning fossil fuels. INEFFICIENT.
- pjmlp 7y agoTell that to accounting that is paying for AWS instance usage. Facebook has an interesting talk about how much electricity 1% performance improvement saves.
- Daishiman 7y agoYou're not Facebook. The carbon footprint of a single developer is comparatively gigantic for the vast majority of projects most people work on.
- pjmlp 7y agoIf it is running on its own computer, for shell scripting. If it is trying to process ML data, or running in some cloud provider, or deployed in some IoT device supposed to run for years without maintenance, then maybe yes.
- Daishiman 7y agoRight, but when you're at that point in performance considerations you already have a team of specialists working on multiple angles in performance. And precisely, for ML code all python libraries run extremely optimized natively compiled code. The language overhead is a minimal consideration. And for business domain code language performance is rarely the limiting factor.
- hyperpallium 7y agoIs there a model dependency management system for some other language that addresses all these issues? Dependency hell is everywhere.
- dehrmann 7y agoIt's not a "model," but if you're able to 1) use fewer dependencies 2) use stable dependencies 3) use dependencies with fewer dependencies, it helps with dependency hell. I've even made commits to projects to reduce their dependency count.
- adrianN 7y agoI find rust with cargo and go with go-modules to be pretty nice to work with.
- RobertRoberts 7y agoDoes it have to be? I have found that I would rather code my own versions of some libraries so I have control over it. Even if there is some extra long term maintenance and some up front dev costs, it's paid off already a number of times.
- mark_l_watson 7y agoA little off topic, but this is why I really like a Common Lisp with Quicklisp: library dependencies are stored in a convenient location locally and the libraries I write can be treated the same way (with a trivial config change to add Quicklisp load paths to my own library project directories).
- itronitron 7y agomaven for java
- ajmurmann 7y agoRust's cargo, JS's yarn and the grand daddy of them all Ruby's bundler address all these issues. Even newer versions of Gradle support a workflow where you specify the versions you know you want and just on everything else, including transitive dependencies, down.
- zys5945 7y agoI agree that builtin tools suck for dependency management. However a lot of the issues that you mentioned (such as lock file and transitive dependencies) can be handled by pipenv, which should be the default package manager
- j88439h84 7y agoPoetry is way way better imho. https://poetry.eustace.io https://poetry.eustace.io
- j88439h84 7y agoThe solution to this is Poetry. https://poetry.eustace.io https://poetry.eustace.io It's good. Projects should use it.
- Soulsbane 7y agoThanks! Didn't realize this existed.
- memorysafety 7y ago> curl ... | python Ah goddamnit. 868 lines, including os.rmtree calls and stuff. Also installable via pip, but... "not recommended", and: [RuntimeError] Poetry was not installed with the recommended installer. Cannot update automatically.
- StavrosK 7y agoYeah, I hate this trend. Unfortunately, you can't pip install poetry because it needs to manage packages, so I guess a different way was necessary. Still, OS-specific packages would be nice, I guess they just need volunteers.
- heavenlyblue 7y agoEven pip is pip-installable. What makes poetry any different?
- acdha 7y agoIt’s running over HTTPS from an auditable source. Is that _really_ so much worse than a pip install, and can you explain in detail why you believe that to be true?
- chrisfinazzo 7y agoSomewhere, I can hear John Siracusa saying, 'curl piped into a shell? No thanks.'
- alwaysanon 7y agoIt is funny - half of the real desire/need for containers comes back to these sorts of issue with both node and Python. And then they bring in their own different challenges.
- bengalister 7y agoI have been programming with node for the last 3 years and I never had any dependency issues with node (at least for 3rd party dependencies). I cannot say that with python that requires using some tool be it docker or virtualenv to isolate them from the already installed ones. Node's dependency managers npm/yarn just copy the versioned dependencies from their cache folder into the local node_modules folder and remove transitive dependencies duplicates when possible by flattening them into node_modules.
- firepoet 7y agoLucky! I wrote a small internal app in node for my company that relied on an IMAP library. 3 months after launch, someone upgraded the library and my app stopped working. Stack traces were incomprehensible. No “how to upgrade” documentation in sight. So I spent 2 hours and rewrote it in Java 8 with Maven. Issues all gone. Node has some work to do before I’ll consider touching it again.
- ChrisRackauckas 7y agoPython's dependency hell is what made me first look at Julia. I develop on Windows (someone has to :) ), and it was just impossible to get all of the numerical libraries like pydstool, scipy, FEniCS, Daedalus, etc. playing nicely together... so I gave Julia a try. And now the only time I have issues getting a package to run are Julia packages which have a Python dependency. Python is a good language, but having everything in one language and binary-free is just a blessing for getting code to run on someone else's computer.
- 6thaccount2 7y agoJulia really is nice in this way. It is nice to see the package manager that actually works for me unlike a lot of what I try with pip.
- jpalomaki 7y agoVisual Studio Code brings some tooling to make it easier to work with code running in Docker container.
- sosodev 7y agoPipenv has felt to me like a pretty solid solution. It's not perfect but it's a lot better than the other options.
- j88439h84 7y agoIt is not as good as Poetry.
- Chris2048 7y agofrom poetry docs: https://github.com/sdispater/poetry/blob/master/README.md#dependency-resolution https://github.com/sdispater/poetry/blob/master/README.md#de... " Let's take an example: pipenv install oslo.utils==1.4.0 will fail with this error: Could not find a version that matches pbr!=0.7,!=2.1.0,<1.0,>=0.6,>=2.0.0 while Poetry will get you the right set of packages "
- yrro 7y agoLast time I gave it a go, I found it was pretty strongly welded to virtualenv, rather than using Python's own (much less problematic) venv. I came away less than enthused as a result. (To be fair, modifying it to use venv is... non-trivial).
- 89qh4p 7y agoPipenv has a good approach to management, similar to npm, but the implementation is buggy. It gained popularity by being recommended very early on in the official python documentation while being misleadingly advertised as production-ready. If you look at the issues section in its github repo, you'll see that there are some pretty basic bugs there which are very annoying or disruptive. Moreover it seems the author has almost left the boat and a handful of contributors have to tidy things up. Just to illustrate my point. I think a package manager that takes a few minutes to install a single tiny package, or don't prevent you from adding non-existing packages (e.g. spelling mistake), or doesn't let you install a new package without trying to upgrade all other packages isn't really production-ready. These are known issues since November last year.
- deleted 7y ago[deleted]
- adev_ 7y agoSolution ? Use Nix. > nix-shell -p python3Packages.numpy python3Packages.my_important_package It solves every problem you quoted before.
- yawaramin 7y agoOr something that works with PyPI directly https://news.ycombinator.com/item?id=20672329 https://news.ycombinator.com/item?id=20672329
- adev_ 7y ago> Or something that works with PyPI directly Poetry is python specific and does not solve the problems that pip/pypi has with native C/C++/Rust/etc modules. Nix/guix solves all of that
- danieldk 7y agoIndeed. We have some Python modules written in Rust. It needs Rust nightly, because pyo3 requires Rust nightly. The Rust crate relies on libtensorflow. Unit tests for the Python module use Python and pytest. And we use our own build of libtensorflow (optimizations for AVX and FMA). The dependencies of such projects are easy to specify in Nix. Moreover, it's easy to reproduce the environment across machines by pinning nixpkgs to a specific version.
- mark_l_watson 7y agoI installed Nix operating system on an old lap top earlier this year, and indeed it does solve a lot of development and devops problems. I retired this spring, so I only played with Nix out of curiosity, but if I still had an active career as a developer I would use Nix.
- antupis 7y agoDependency management system is a definite problem but making executables is downright horrible or even impossible.
- marble-drink 7y agoIt's not that bad if you use the right tools. The two main options are an all-in-one solution like poetry or pipenv, and an ensemble of tools like pyenv, virtualenvwrapper, versioneer and pip-tools. I prefer the latter because it feels more like the Unix way. Why should Python have some "official" method to do this? Flexibility is a strength, not a weakness. Nobody ever suggests that C should have some official package manager. Instead the developers build a build system for their project. After a while every project seems to get its own unique requirements so trying to use a cookie-cutter system seems pointless.
- andybak 7y ago> Flexibility is a strength, not a weakness. "There should be one-- and preferably only one --obvious way to do it.": https://www.python.org/dev/peps/pep-0020/ https://www.python.org/dev/peps/pep-0020/
- RhysU 7y agoThey almost rescinded that in PEP 357 and ultimately did so in PEPs 468/469. PEP -23 updated the standard library to match, but not until 3.9.1.a. Until then, beware the various blog posts you'll find on Google talking about this concept on 1.x.
- andybak 7y agoIt was always an ideal to aim for rather than a strict rule. I don't see any of those PEPs changing the balance enough to claim the principle was dead (maybe a bit injured...)
- mft_ 7y agoIn general, leaving such things open leads to a proliferation of different 'solutions', as multiple people try to solve the issue... leading to the additional confusion and cognitive load of trying to find a single solution which suits your use-case and works, when often none of them are perfect. Sometimes a 'benign dictator' single approach has benefits...
- wasnthere 7y agosaddened to see this poorly constructed comment berating python at the top of this thread. the author seems to have some personal issues with the language given the generally frustrated tone of the comment. the entire comment could have just been 1 line "We need official support for a modern package management system, from the Python org itself." which would be consumed as constructive feedback by all readers with the right context. but somehow the author chooses to "vent" adding unnecessary drama to something that does not get in the way of writing high quality production grade python apps (general purpose, web, ai or otherwise) there is no language that is devoid of shortcomings - so to any new (<3 yrs exp) python users, please ignore the above comment entirely as it has no bearing on anything practical that you are doing/will do. and all experienced python users know that there are ways to work around the shortcomings listed here and beyond. this is my psa for the python community!
- saagarjha 7y ago> the entire comment could have just been 1 line "We need official support for a modern package management system, from the Python org itself." I personally would consider this to be a strictly worse comment because it does not go into detail about what’s lacking like the parent comment does.
- Chris2048 7y ago> the author seems to have some personal issues with the language given the generally frustrated tone of the comment What "personal issues" do you think the author has? The frustrated tone comes from the frustrations the author explicitly outlines; unless you think this shouldn't be so, you are turning this into an ad-hom. > the entire comment could have just been 1 line "We need official support for a modern package management system, from the Python org itself." Why? because you don't appreciate the detail on why we need such a thing? These issues certainly get in the way of producing production apps; not in the sense that they make it impossible, but they make the process harder and slower than it needs to be.
- rezeroed 7y agoI actually quite liked that post. I use python maybe once a year or less, and don't enjoy the experience. That post distinguished some of the details which in my rare usage I see simply as a gloopy mess.
- wisty 7y agoDocker is one option. Don't worry about sorting out that mess, just clone the whole operating system. sigh
- moron4hire 7y ago.NETs solution to this was the project file, a configuration file that lists the compiler version, framework version, and dependencies (now including NuGet packages and their versions).
- fearface 7y agoI've always seen it like this: Not everyone builds reproducible software with Python (or in general) and how you handle dependencies can vary. Python leaves it open how you do it: globally installed packages, local packages, or a mix of both. In the end, it needs to find the import in the PYTHONPATH, so there's no magic involved, and there are multiple robust options to choose from. So instead of bashing Python for not shoveling down an opinion on you, it's up to the developers to choose which tools they want to use. If they don't choose one and are unable to freeze their dependencies, it's not a Python problem, but IMO lack of skill and seniority.
- busfahrer 7y ago> Python leaves it open how you do it Are you saying “There’s more than one way to do it”?
- fearface 7y agoYes :-) It's fair to say Python's approach to dependency management doesn't follow the Zen of Python, but there's a simple way documented in the tutorial: https://docs.python.org/3/tutorial/venv.html https://docs.python.org/3/tutorial/venv.html
- WilliamEdward 7y agoThe fact that there's more than one way to do things in Python is why i've found it so easy and flexible, I have no idea why that goober put this motto in the zen
- dragonsh 7y agoIt's general design guideline and I like Zen of python PEP-20. Explicit is better than implicit and most packaging system in python are explicit which I like. Been using it for over 15 years after perl and been happy with it. Nothing to complaint as every language has their own set of good and bad. This is what makes it interesting, there is always a room to improve and make things better.
- arvinsim 7y ago> The biggest issue, in my opinion, is in dependency management. Python has a horrible dependency management system, from top-to-bottom. Yup, I love Python over my current language in my job(JS/TS). But I really dislike handling conflicts using pip, requirements.txt and virtualenv. So much so that I will take JS node_modules over it.
- Chris2048 7y agoI recently found this: https://dephell.org/docs/ https://dephell.org/docs/ I seems to have some neat functionality wrt dep handling (and I'd never really heard of it before).
- tsss 7y agoPython was a scripting language. All those problems are caused by people using it like something it isn't. Python has way outlived it's usefulness and it's about time we move on to something better.
- oehtXRwMkIs 7y agoDo you think that language exists right now? I can't think of a good alternative.
- void445be54d48a 7y agoThe virtual env is really the thing that has stopped me from using python. It's a lovely language but the tooling around it needs a lot of help. I'm sure it will get there though. I mean if the js folks can do it, certainly python can.
- chrisfinazzo 7y agoThe virtualenv thing just galls me. Sure, pipenv aped rbenv - appropriately, I might add - but until they supplant virtualenv as the recommended way to have separate environments, I'll pass.
- takeda 7y ago> The biggest issue, in my opinion, is in dependency management. Python has a horrible dependency management system, from top-to-bottom. I agree, although a lot of it has to do that there's so much misinformation about the web, and many articles recommending bad solutions. This is because python went through many packaging solutions. IMO the setuptools one is the one that's most common and available by default. It has a weakness though, it started with people writing setup.py file and defining all parameters there. Because setup.py is actually a python program it encourages you to write it as a program and that creates issues, setuptools though for a wile had a declarative way to declare packages using setup.cfg file, you should use that and your setup.py should contain nothing more than a call to setup(). > Why do I need to make a "virtual environment" to have separate dependencies, and then source it my shell? Because chances are that your application A uses different versions than application B. Yes this could be solved by allowing python to keep multiple versions of the same packages, but if virtualenv is bothering you you would like to count on system package manager to keep care of that, and rpm, deb don't offer this functionality by default. So you would once again have to use some kind of virtualenv like environment that's disconnected from the system packages. > Why do I need to manually add version numbers to a file? You don't have to, this is one of the things that there's a lot of misinformation about how to package application. You should create setup.py/cfg and declare your immediate dependencies, then you can optionally provide version _ranges_ that are acceptable. I highly recommend to install pip-tools and use pip-compile to generate requirements.txt, that file then works like a lock file and it is essentially picking the latest versions within restrictions in setup.cfg > Why isn't there any builtin way to automatically define a lock file (currently, most Python projects just don't even specify indirect dependency versions, many Python developers probably don't even realize this is an issue!!!!!)? Because Python is old (it's older than Java) it wasn't a thing in the past. > Why can't I parallelize dependency installation? Not sure I understand this one. yum, apt-get etc don't parallelize either because it's prone to errors? TBH I never though of this as an issue, because python packages are relatively small and it installs quickly. The longest part was always downloading dependencies, but caching solves that. > Why isn't there a builtin way to create a redistributable executable with all my dependencies? Some people are claiming that python has a kitchen sink and that made it more complex, you're claiming it should have even more things built in, I don't see a problem, there are several solutions to package it as an executable. Also it is a difficult problem to solve, because Python also works on almost all platforms including Windows and OS X. > Why do I need to have fresh copies of my dependencies, even if they are the same versions, in each virtual environment? You don't you can install your dependencies in system directory and configure virtualenv to see these packages as well, I prefer though to have it completly isolated from the system. > There is so much chaos, I've seen very few projects that actually have reproducible builds. Most people just cross their fingers and hope dependencies don't change, and they just "deal with" the horrible kludge that is a virtual environment. Not sure what to say, it works predictable to me and I actually really like virtualenv > We need official support for a modern package management system, from the Python org itself. Third party solutions don't cut it, because they just end up being incompatible with each other. setuptools with declarative setup.cfg is IMO very close there. > Example: if the Python interpreter knew just a little bit about dependencies, it could pull in the correct version from a global cache - no need to reinstall the same module over and over again, just use the shared copy. Imagine how many CPU cycles would be saved. No more need for special wrapper tools like "tox". There is a global cache already and pip utilizes it even withing an virtualenv. I actually never needed to use tox myself. I think most of your problems is that there are a lot of bad information about how to package a python app. Sadly even the page from PPA belongs there. I think people should start with this: https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files https://setuptools.readthedocs.io/en/latest/setuptools.html#... Yes it still has some of the problems you mentioned, but it fixes some others.
- jakeogh 7y agoPortage solves a few of these. http://michael.orlitzky.com/articles/motherfuckers_need_package_management.xhtml http://michael.orlitzky.com/articles/motherfuckers_need_pack...
- m463 7y agoPython is the WORST language (except for all the rest)
- fiedzia 7y ago> Why isn't there a builtin way to create a redistributable executable with all my dependencies? There is and it's called docker. The other issues could indeed be fixed with something like poetry.