5 ms·

I work on Python every day on a reasonably large code base and have none of the issues you’re talking about. I’m 10x more productive than similar C or Java pro

I work on Python every day on a reasonably large code base and have none of the issues you’re talking about. I’m 10x more productive than similar C or Java projects. Dependency management is about as easy as it is going to get. We have problems with our dependencies breaking stuff, but who doesn’t? People talk as if packaging is a solved problem. It isn’t in any language. And then they complain that Python packaging changes too much. That’s because folks are iterating on a hard problem.

disgruntledphd2 6y ago
Do you handle deployment of this Python application? For me, that's where the pain points arise. I love writing Python, but deploying it does not spark joy at all, at all.
- erjiang 6y ago
  Curious to know what issues you have with deploying Python codebases. Out of all of the minor and major gripes I have with Python, deployment is not one of them.
  - tetha 6y ago
    To me, python deployments are painless, as long as you can stick to pure dependencies and possibly wheels. Once a pip install needs to start compiling C, things do go way south very quickly. At that point you can install the union of all common C development tools, kernel headers and prepare for hours of header hunting. I've done that too much to like python anymore.
    - disgruntledphd2 6y ago
      Yup, yup. I deploy statistical models with Python, and these always have C dependencies. Additionally, they are part of a larger application, which is mostly managed by pip, which means that I need both pip and conda which is where things get really, really hairy. I actually blame Google and FB here, as neither of them use standard python dependency management tools, and many of their frameworks bring in the world, thus increasing the risk of breakage.
  - JAlexoid 6y ago
    Adding data files via setuptools.... And putting them into a common shared directory. Try doing that without writing convoluted code in your setup.py.
    - xapata 6y ago
      It is funky, but importlib package resources helps.
- d0mine 6y ago
  Here's some of the ways to deploy Python code: - `curl -L https://app.example.com/install https://app.example.com/install | sh` that downloads installer and runs for instance: apt/yum install <your-package> - in CI environment on a VM: `git checkout` & `pipenv install --deploy` - `pipx install glances` on a home computer - just `pip install` e.g., in a docker container [possibly in a virtualenv]. For pure Python packages, it can work even in Pythonista for iOS (iphone/ipad) - just copy a python module/archive (PyInstaller and the like) - give a link to a web app (deployed somewhere via e.g., git push) - for education: there are python in the browser options e.g., brython, repl.it, trinket.io, pythontutor.com - just write a snippet in my favourite editor for literate devops tasks/research (jupyter-emacs + tramp + Org Babel) or give a link to a Jupyter notebook - a useful work can be done even in a REPL (e.g., Python as a powerful calculator)
  - lmm 6y ago
    The fact that you have 9 different ways all with their own different problems is exactly the problem here.
    - d0mine 6y ago
      Do you use a single program on all of your devices for all possible computer-related tasks? Do you see a fault in such logic? Why do you think the deployment space is any different: do you use kubernetes for everything? I dare you. Do mention any tool/any language that handles all the above use cases without sacrificing the requirements for each use-case.
      - disgruntledphd2 6y ago
        My core problem is with C/C++ depedencies. Can you describe to me how you handle these when you deploy Python?
        d0mine 6y ago
        - system packages (deb/rpm/etc) - binary wheels (manylinux) - building from source plus some caching if appropriate
- wenc 6y ago
  I use Anaconda exclusively and deployments (with virtual environments) have been fairly ok. That said, I do run into trouble when I have a dependency that requires compilation on Windows (i.e. like the popular turbodbc) because say, a wheel isn't available for a particular Python version. Any time a compilation is needed, it's a headache. Windows machines don't come with compilers, so one has to download and install a multigigabyte Visual Studio Build Essentials package just to compile. Sometimes the compilation fails for various reasons. Require gcc compilation is headache for installing dependencies inside Docker containers too -- you have to install gcc in order to install Python dependencies and then remove gcc after. I think requiring local compilation (instead of just delivering the binary) is a UNIX-mindset that is holding back many packaging solutions. I think a lot of pain would be alleviated if we could somehow mandate centralized wheel creation for all Python versions, otherwise the package manager marks a package as broken or unavailable and defaults to the last available wheel. Also if only we applied some standards like R's CRAN repo does -- ie. if it doesn't pass error checks or doesn't build on certain architectures (institute a centralized CI/CD build pipeline in the package repo), it doesn't get published -- the Python packaging experience would be much improved.
  - disgruntledphd2 6y ago
    Yeah, if PyPi was as annoying as Cran with respect to new versions, then a lot of this pain would go away. For those who don't realise, when there's a new version of R, anything that doesn't build without errors/warnings is removed from the archive. This is really annoying if you want something to keep running, but it prevents the kind of dependency rot common to Python (recently I found a dependency that was four years out of date).
- jnwatson 6y ago
  Production deployment is Docker all the time. Deployment for development is just pyenv and virtualenv.

disgruntledphd2 6y ago

Do you handle deployment of this Python application? For me, that's where the pain points arise. I love writing Python, but deploying it does not spark joy at all, at all.

erjiang 6y ago
Curious to know what issues you have with deploying Python codebases. Out of all of the minor and major gripes I have with Python, deployment is not one of them.
- tetha 6y ago
  To me, python deployments are painless, as long as you can stick to pure dependencies and possibly wheels. Once a pip install needs to start compiling C, things do go way south very quickly. At that point you can install the union of all common C development tools, kernel headers and prepare for hours of header hunting. I've done that too much to like python anymore.
  - disgruntledphd2 6y ago
    Yup, yup. I deploy statistical models with Python, and these always have C dependencies. Additionally, they are part of a larger application, which is mostly managed by pip, which means that I need both pip and conda which is where things get really, really hairy. I actually blame Google and FB here, as neither of them use standard python dependency management tools, and many of their frameworks bring in the world, thus increasing the risk of breakage.
- JAlexoid 6y ago
  Adding data files via setuptools.... And putting them into a common shared directory. Try doing that without writing convoluted code in your setup.py.
  - xapata 6y ago
    It is funky, but importlib package resources helps.
d0mine 6y ago
Here's some of the ways to deploy Python code: - `curl -L https://app.example.com/install https://app.example.com/install | sh` that downloads installer and runs for instance: apt/yum install <your-package> - in CI environment on a VM: `git checkout` & `pipenv install --deploy` - `pipx install glances` on a home computer - just `pip install` e.g., in a docker container [possibly in a virtualenv]. For pure Python packages, it can work even in Pythonista for iOS (iphone/ipad) - just copy a python module/archive (PyInstaller and the like) - give a link to a web app (deployed somewhere via e.g., git push) - for education: there are python in the browser options e.g., brython, repl.it, trinket.io, pythontutor.com - just write a snippet in my favourite editor for literate devops tasks/research (jupyter-emacs + tramp + Org Babel) or give a link to a Jupyter notebook - a useful work can be done even in a REPL (e.g., Python as a powerful calculator)
- lmm 6y ago
  The fact that you have 9 different ways all with their own different problems is exactly the problem here.
  - d0mine 6y ago
    Do you use a single program on all of your devices for all possible computer-related tasks? Do you see a fault in such logic? Why do you think the deployment space is any different: do you use kubernetes for everything? I dare you. Do mention any tool/any language that handles all the above use cases without sacrificing the requirements for each use-case.
    - disgruntledphd2 6y ago
      My core problem is with C/C++ depedencies. Can you describe to me how you handle these when you deploy Python?
      - d0mine 6y ago
        - system packages (deb/rpm/etc) - binary wheels (manylinux) - building from source plus some caching if appropriate
wenc 6y ago
I use Anaconda exclusively and deployments (with virtual environments) have been fairly ok. That said, I do run into trouble when I have a dependency that requires compilation on Windows (i.e. like the popular turbodbc) because say, a wheel isn't available for a particular Python version. Any time a compilation is needed, it's a headache. Windows machines don't come with compilers, so one has to download and install a multigigabyte Visual Studio Build Essentials package just to compile. Sometimes the compilation fails for various reasons. Require gcc compilation is headache for installing dependencies inside Docker containers too -- you have to install gcc in order to install Python dependencies and then remove gcc after. I think requiring local compilation (instead of just delivering the binary) is a UNIX-mindset that is holding back many packaging solutions. I think a lot of pain would be alleviated if we could somehow mandate centralized wheel creation for all Python versions, otherwise the package manager marks a package as broken or unavailable and defaults to the last available wheel. Also if only we applied some standards like R's CRAN repo does -- ie. if it doesn't pass error checks or doesn't build on certain architectures (institute a centralized CI/CD build pipeline in the package repo), it doesn't get published -- the Python packaging experience would be much improved.
- disgruntledphd2 6y ago
  Yeah, if PyPi was as annoying as Cran with respect to new versions, then a lot of this pain would go away. For those who don't realise, when there's a new version of R, anything that doesn't build without errors/warnings is removed from the archive. This is really annoying if you want something to keep running, but it prevents the kind of dependency rot common to Python (recently I found a dependency that was four years out of date).
jnwatson 6y ago
Production deployment is Docker all the time. Deployment for development is just pyenv and virtualenv.