5 ms·

The same issue occurs with Node and NPM. I don't want to speak for Ruby as I haven't used Ruby extensively but I can't imagine that Gem's and Ruby versions don'

by cfors 7y ago

The same issue occurs with Node and NPM. I don't want to speak for Ruby as I haven't used Ruby extensively but I can't imagine that Gem's and Ruby versions don't have the same issue. Honestly virtualenv's solve most of these problems. If using a virtualenv seem to low level or a pain, I can't recommend Poetry [0] enough. It uses virtualenv's in the background, and has from my experience a great dependency resolver. [0] https://poetry.eustace.io/ https://poetry.eustace.io/

heavenlyblue 7y ago
What about jvm, rust, scala, haskell? All of them require similar "cludges".
- EdwardDiego 7y ago
  Hmm? Which cludges? Dependency management in JVM languages is painless compared to Python.
  - brianwawok 7y ago
    What? How far back in time are you going? As far as I remember, it went Ant to Maven to Gradle? With some less popular offshoots like SBT and Ivy. All of those build tools are pretty complicated. I honestly prefer python and virtual environment. Simple requirements.txt file.
    - EdwardDiego 7y ago
      They're complex, because they handle a wide variety of tasks. Maven, for example, supports multiple languages - you want to have a project mixing Scala, Kotlin, Java and Clojure? No problem. Well, maybe some problems, depends on how well you've kept your exposed APIs interoperable. Want to ship all your app and all it's dependencies in a single file, sweet as, use the shade plugin, or maybe the capsule plugin. etc. Etc. The Python equivalent is, well, take your pick: https://docs.python-guide.org/shipping/freezing/#freezing-your-code-ref https://docs.python-guide.org/shipping/freezing/#freezing-yo... > Simple requirements.txt file I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole. Not to mention that it was impossible to generate a pipfile.lock for Airflow because its dependencies had rules demanding incompatible versions of dependencies (one wanted <= 7.0.0, the other wanted > 7.0.0) - in Maven I can easily exclude one of the two competing dependencies. Oh, and then there was the time that a Python dependency used by Airflow required me to set an env var to handle a licensing issue. Can't even do that in Maven, which strikes me as a feature. pip3 install apache-airflow RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE Also, your simple requirements.txt didn't mention how you manage virtualenvs. Lastly, Gradle is being used quite a bit, but Maven is still pretty dominant. It really depends on if you want a purely declarative build tool like Maven, or one you can drop scripts into like Gradle. I prefer purely declarative for reproducible buils.
      - m45t3r 7y ago
        > I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole. Pip explicitly has no API (except for its command line interface), even the Pip namespace implicates this (pip._internal). So if some dependency is using a Pip "API" they're wrong. Now, I am developing an application that needs to interface with Pip, in this case it may be appropriate to use the internal Pip API if you pin your Pip version in requirements.txt. However a library shouldn't do this ever.
        EdwardDiego 7y ago
        Sure, it's wrong, not going to disagree. But it happened with a "plain ol requirements.txt", hence why I don't rate Python dependency management as highly as you do.
        m45t3r 7y ago
        What do you mean "plain ol requirements.txt" in this case? Either you're pinning your dependencies in requirements.txt (in this case, you wouldn't have a problem since requirements.txt is acting like a lockfile) or you're doing it wrong. I mean, there is no problem to depend in a internal API if you're locking the versions, if not, just assume the consequences. BTW, I don't rate Python dependency management very high: most other languages have better solutions. However it is manageable.
  - theamk 7y ago
    Say what? Every JVM program I have seen has an attached shell script to set up dependency path. Here is a first example: https://github.com/apache/spark/blob/121f9338cefbb1c800fabfea5152899a58176b00/bin/spark-class https://github.com/apache/spark/blob/121f9338cefbb1c800fabfe... At least half of that is figuring out what the classpath should be. And the worst part, it is all exposed to user, too -- when I was setting up Spark, I had to muck with class paths a lot. So JVM has 3 big problems: (1) No single way to set up dependency path, each project is different; (2) No way to set up classpath in JVM language, has to use some other one (like bash); (3) Somehow, user has to fix it for many semin-advanced programs. No other ecosystem I know of is this insane! Python, node.js, even C++ are all better!
    - EdwardDiego 7y ago
      Okay, so you've changed the context from "I'm a developer" to "I'm an end user" - how many Python end users are having to bugger about with virtualenvs? Okay, so, speaking for myself, we generally ship fat jars or capsules these Docker days. I can't speak as to why Apache Spark chose to do it that way, I don't run their project. But if I'm guessing I suspect it's so you can provide your own Hadoop dependencies as needed, as there's a bunch of varying distributions that people use (Cloudera, Hortonworks etc.) and they're trying to play nice with the existing Hadoop ecosystem - probably also to make it possible to extend it with any other jars that you want to use (although spark-shell has a nice feature where you provide additional dependencies on the command line, or within the shell, using Maven artifact coordinates) For an example of what's involved using an uberjar https://docs.cyclopsgroup.org/jmxterm https://docs.cyclopsgroup.org/jmxterm > java -jar jmxterm-1.0.0-uber.jar That's it.
      - m45t3r 7y ago
        No, OP case is for developers setting up their development machines. I feel his pain since I work with Clojure, that 99% of time it simply works however in the 1% of time that it breaks its is painful to fix. BTW, the uberjar equivalent in Python is something like PEX/shiv/zipapps.
      - theamk 7y ago

by cfors 7y ago

heavenlyblue 7y ago

What about jvm, rust, scala, haskell? All of them require similar "cludges".

EdwardDiego 7y ago
Hmm? Which cludges? Dependency management in JVM languages is painless compared to Python.
- brianwawok 7y ago
  What? How far back in time are you going? As far as I remember, it went Ant to Maven to Gradle? With some less popular offshoots like SBT and Ivy. All of those build tools are pretty complicated. I honestly prefer python and virtual environment. Simple requirements.txt file.
  - EdwardDiego 7y ago
    They're complex, because they handle a wide variety of tasks. Maven, for example, supports multiple languages - you want to have a project mixing Scala, Kotlin, Java and Clojure? No problem. Well, maybe some problems, depends on how well you've kept your exposed APIs interoperable. Want to ship all your app and all it's dependencies in a single file, sweet as, use the shade plugin, or maybe the capsule plugin. etc. Etc. The Python equivalent is, well, take your pick: https://docs.python-guide.org/shipping/freezing/#freezing-your-code-ref https://docs.python-guide.org/shipping/freezing/#freezing-yo... > Simple requirements.txt file I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole. Not to mention that it was impossible to generate a pipfile.lock for Airflow because its dependencies had rules demanding incompatible versions of dependencies (one wanted <= 7.0.0, the other wanted > 7.0.0) - in Maven I can easily exclude one of the two competing dependencies. Oh, and then there was the time that a Python dependency used by Airflow required me to set an env var to handle a licensing issue. Can't even do that in Maven, which strikes me as a feature. pip3 install apache-airflow RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE Also, your simple requirements.txt didn't mention how you manage virtualenvs. Lastly, Gradle is being used quite a bit, but Maven is still pretty dominant. It really depends on if you want a purely declarative build tool like Maven, or one you can drop scripts into like Gradle. I prefer purely declarative for reproducible buils.
    - m45t3r 7y ago
      > I've never had a JVM dependency break because I upgraded Maven. I have had that exact issue with Python - we're using Apache Airflow, and anyone who installed the latest version of Pip hit an issue - Pip had changed an API that Python dependencies depended on, so Airflow could no longer be installed by that version of Pip, because it was written for the old API. Have fun googling "TypeError: 'module' object is not callable" to enjoy that rabbit hole. Pip explicitly has no API (except for its command line interface), even the Pip namespace implicates this (pip._internal). So if some dependency is using a Pip "API" they're wrong. Now, I am developing an application that needs to interface with Pip, in this case it may be appropriate to use the internal Pip API if you pin your Pip version in requirements.txt. However a library shouldn't do this ever.
      - EdwardDiego 7y ago
        Sure, it's wrong, not going to disagree. But it happened with a "plain ol requirements.txt", hence why I don't rate Python dependency management as highly as you do.
        m45t3r 7y ago
        What do you mean "plain ol requirements.txt" in this case? Either you're pinning your dependencies in requirements.txt (in this case, you wouldn't have a problem since requirements.txt is acting like a lockfile) or you're doing it wrong. I mean, there is no problem to depend in a internal API if you're locking the versions, if not, just assume the consequences. BTW, I don't rate Python dependency management very high: most other languages have better solutions. However it is manageable.
- theamk 7y ago
  Say what? Every JVM program I have seen has an attached shell script to set up dependency path. Here is a first example: https://github.com/apache/spark/blob/121f9338cefbb1c800fabfea5152899a58176b00/bin/spark-class https://github.com/apache/spark/blob/121f9338cefbb1c800fabfe... At least half of that is figuring out what the classpath should be. And the worst part, it is all exposed to user, too -- when I was setting up Spark, I had to muck with class paths a lot. So JVM has 3 big problems: (1) No single way to set up dependency path, each project is different; (2) No way to set up classpath in JVM language, has to use some other one (like bash); (3) Somehow, user has to fix it for many semin-advanced programs. No other ecosystem I know of is this insane! Python, node.js, even C++ are all better!
  - EdwardDiego 7y ago
    Okay, so you've changed the context from "I'm a developer" to "I'm an end user" - how many Python end users are having to bugger about with virtualenvs? Okay, so, speaking for myself, we generally ship fat jars or capsules these Docker days. I can't speak as to why Apache Spark chose to do it that way, I don't run their project. But if I'm guessing I suspect it's so you can provide your own Hadoop dependencies as needed, as there's a bunch of varying distributions that people use (Cloudera, Hortonworks etc.) and they're trying to play nice with the existing Hadoop ecosystem - probably also to make it possible to extend it with any other jars that you want to use (although spark-shell has a nice feature where you provide additional dependencies on the command line, or within the shell, using Maven artifact coordinates) For an example of what's involved using an uberjar https://docs.cyclopsgroup.org/jmxterm https://docs.cyclopsgroup.org/jmxterm > java -jar jmxterm-1.0.0-uber.jar That's it.
    - m45t3r 7y ago
      No, OP case is for developers setting up their development machines. I feel his pain since I work with Clojure, that 99% of time it simply works however in the 1% of time that it breaks its is painful to fix. BTW, the uberjar equivalent in Python is something like PEX/shiv/zipapps.
    - theamk 7y ago