10 ms·

Doing Python development under Mac OS

bdcravens 6y ago
This is why I gave up on playing super-nerd and just moving all development to docker-compose.
- random5634 6y ago
  I'm in Windows / Linux, my wife is on MacOS. She wanted to help with some python programming. Yep - playing super nerd is LAME. I thought getting environment setup to match mine would be pip requirements.txt but no go. Really facinating. I think things like go / rust etc? may work better - you compile everything and compile to binary on platform, so don't need to build and carry around environments everywhere. I moved to docker deploys / dockerfiles locally for pretty much everything. Totally overkill, but when I switched to working from home was so glad I could just get up and go without getting my overall system sorted out. I haven't totally figured out the WSL2 python remote interpreter / debugger code flow yet however, so still using docker. I'm sure WSL2 will or is an option. I've been tempted to just vnc or similar into a cloud instance, but my local machine is to fast by comparison for the price.
- dinkleberg 6y ago
  Using PyCharm with docker-compose as the interpreter works like a dream (relative to Python development). No more dependency problems and then you can also have a full setup with postgres, Celery, redis, etc. all available on the local machine. My mac is a couple years old and I don't have the fan of death problem everyone seems to complain about. The best thing is allowing my team to know with confidence we're developing against the exact same environment.
ryan-duve 6y ago
The author concludes the best solution is using brew to install different Python dependencies for different packages. I had a similar solution when I started using a Mac for work back in 2017, but around six months ago I found another way: 1. Install Docker CE on the Mac. 2. Start a container for your work, e.g., docker run python:3.8 3. When you need to get back into it, use docker exec -it <first_three_container_characters> ipython 4. If the container is shut down from a reboot or something, start it with docker start <first_three_container_characters> What's really cool about this is `bash` can be run instead of `ipython` to get a Bash prompt and install any Linux dependency. In theory, at some point if one project has conflicting dependencies with another project a new container can be spun up, but this is not something that's come up in practice yet. This has worked so well for me these past few months, I actually replicated it on my Linux computer. I took a few extra convenience steps like mounting `/home/` and aliasing `ipython` to a command that starts the container before executing the command, but other than that it's been a pretty seamless experience and I never have to worry about OS-level Python installation stuff any more. I use a scipy image with Jupyter Lab in a similar way for my graphical notebook needs. The only time I would recommend against this is when performance is an issue (like training ML models), but at work we use AWS machines with their own Docker environments for that. For everything else, Mac, Linux or maybe even Windows, Docker is the only way I now recommend firing up Python.
- TheRealPomax 6y ago
  And then enjoy the jet engine that your macbook fans turn into, because holy shit Docker is bad at resource management.
  - vorpalhex 6y ago
    This depends on your base image and what it's doing. If you're running like a large elasticsearch cluster.. you're still running a large elasticsearch cluster. Docker overhead should generally be minimal even with the VM that's running, but make sure to use alpine/slim images.
  - powerapple 6y ago
    Agree. I have never enjoyed running docker on my laptop. If I have to use Docker, I use it on a remote Linux server. I had good experience with Vagrant if I have to use a linux environment locally. The only good thing with Docker on macOS is that you don't need to sudo it.
  - Wowfunhappy 6y ago
    Is the problem that Docker is bad at resource management, or that this is just what happens when you're actually running an entire second OS under the hood? I appreciate Docker's approach, but it's definitely a bit of a nuclear option (at least on non-Linux systems).
- cutler 6y ago
  Surely it's an admission of failure if one has to resort to Docker just to get a scripting language to function on a local machine? It's only a matter of time before someone finds a use for Kubernetes on a local machine.
SeanLuke 6y ago
> Yamale’s tox.ini, for example, depends upon two different versions of Python, 3.6 and 3.8. What? It seems to me that this awful design choice is root of the problem, and not MacOS at all.
- jimminy 6y ago
  Tox is designed for environment isolation, and primarily for testing against multiple release targets. It's not an awful choice, when it's exactly what you want to do. The dependency is that tox will run the tests against a Python 3.6 target environment and a Python 3.8 target environment. The dual requirement is asking for test assurance that Yamale will work with 3.6 or 3.8.
- saila 6y ago
  This just means that tox is configured to run the project's test suite against those two Python versions.
  - SeanLuke 6y ago
    Ah, then move along.
Demiurge 6y ago
This is why I started using Conda recently. It works quite well to create isolated python environments, having many prebuilt packages, and most importantly, system level dependencies. So, you can install some of those, and for anything that is not in conda-forge, pip install works and builds.
- dnautics 6y ago
  just be careful. I don't know if things have improved in conda-land, but when I last used it a few years ago I table flipped and quit conda (and, ultimately, python) when the system-level dependency management got so broken that I was forced to reinstall linux from scratch and start COMPLETELY over (it had broken things outside of its domain). Tensorflow being hard-pinned to CUDA versions probably didn't help.
  - xapata 6y ago
    I've been using conda for quite some time, maybe version 0, I forget, and this has never happened to me. I hope you submitted an issue.
  - icedistilled 6y ago
    Having gone through the nightmare of setting up tensor flow with cuda, I can't help but think you're letting that color your view of python as a whole. Tensorflow, cuda, and nvidia were a nightmare that took me 5 days to resolve on ubuntu 18 since it was my first time setting it up ever. There was literally no working guides or people posting with my exact issue, but tons of posts about other people having a nightmare of a time. In the end I figured out I had to go find this one file/library nvidia had moved around between versions of drivers/cuda even though documentation suggested they hadn't and I was using the versions that were supposed to work fine. In the end, fixing it was either by setting an environment variable because the default path was wrong or editing the path in a config file. I'm not remembering which now.
    - dnautics 6y ago
      I dunno, I think the problem is not having a project-specific location for your deps. Obviously you want global deps for a console scripting language, but if you are doing any sort of bigger project, having a global deps location (which is implicitly resolved by the language, no less) is unacceptable for reproducible development for larger projects today. The env hack that python managers like conda and venv use is, IMO, extremely dangerous, because it's hiding state. These days I use languages that don't do this for my daily drivers, it's better for my sanity.
brundolf 6y ago
https://media.tenor.com/images/af0610f778ecdc1b7ab5a750c2d98615/tenor.gif https://media.tenor.com/images/af0610f778ecdc1b7ab5a750c2d98...
FunnyLookinHat 6y ago
I used to play this game but realized pyenv / nvm / etc. are all just anti-patterns nowadays. Use docker ( or some other containerized solution ) to run code in your local directory in a specific environment. I think the author is doing the best they can considering all of these tools and resources aren't updated to this pattern yet, but I'd hate to see this become the new normal for any developer entering into the field. What a way to scare off new dev friends!
singron 6y ago
I highly recommend nix for this type of problem, both on linux and macos. You can make all kinds of dependencies available simultaneously in the same file system. You don't have to run vms, or copy in and out of containers. The build cache works great and is much better than a linear Dockerfile. Tweaking dependencies at any level is pretty easy.
- emptysea 6y ago
  It's been a while since I last ventured into nix with python, but when I last tried it seemed way less user friendly compared to the current python solutions like poetry[1]. With poetry I can `poetry add foo` and `foo` will be added as a dependency and importantly all its dependencies will be locked in the `poetry.lock` file. With Nix either you write all your definitions from scratch, which feels like maintaining a lock file by hand, or you specify a package like `numpy` in your dependencies that is defined in the nix packages and hope that the next time you update nix packages nothing breaks. Also if you want a package that isn't in nix packages you have to write the definition yourself, which isn't a big deal if you start a project off with nix and gradually add dependencies as you need them, but if you are trying to migrate your project with 200+ dependencies it feels really tedious. [1]: https://python-poetry.org/ https://python-poetry.org/
they4kman 6y ago
For complicated situations like this, nothing beats MacPorts. Where brew is generally based on recipes (read: collection of bash scripts) that may or may not play well together, MacPorts is a package system with well-defined dependencies. The drawback to MacPorts is that everything is built from source, and nothing previously installed on the system is utilized -- so installing Python for the first time may take 30-60 minutes. But the next time is 5 minutes, and you can install all the versions you want, where they'll live side-by-side in peace and love and harmony. MacPorts is slower, but offers peace of mind -- 'cause when I'm trying to develop something, the last thing I want to worry about is dependency issues (gross)
- icedchai 6y ago
  I've seen MacPorts uses pre-built binaries, at least in some cases. Not sure if this applies to the Python ports.
- pseudalopex 6y ago
  MacPorts has binary packages. Homebrew uses Ruby not bash. MacPorts uses Tcl. MacPorts just has better QA.
- cutler 6y ago
  I don't understand. I've been using MacPorts for 16 years and I've never had to wait more than a few minutes for a Python install to complete. Most packages are pre-built. I can even get clang without having to compile. Plus one for MacPorts though. It rarely lets me down and unlike Homebrew stays in /opt/local instead of polluting /usr.
deleted 6y ago
[deleted]
zemo 6y ago
Python used to be my tool of choice for all of my small programming tasks, but nowadays I find messing with Python to be an absolutely massive time-waster for me. It used to be my favorite programming language. Now I avoid it entirely because of this nonsense.
- nickt 6y ago
  Where did you go from Python?
  - zemo 6y ago
    I just write everything in Go nowadays, since I didn't have any hard Python requirements like numpy to worry about.
    - nickt 6y ago
      Cool, thanks.
- deleted 6y ago
  [deleted]
moocowtruck 6y ago
pipenv
majormunky 6y ago
Python development on a Mac works pretty well if you install it from the installer that you get from python.org, at least it has in my experience.
powerapple 6y ago
I was able to use venv and install tox and tox-pyenv locally to use tox. I also installed pyenv with brew. I had many problem with scientific packages in python projects mostly, the incompatibility between different versions of libraries, I use Conda for those projects, and venv for other projects not dependent on numpy and scikit-learn.
dmart 6y ago
I suspect that the terrible installation/package management/distribution experience is going to be the thing that ultimately drives Python into irrelevance. Beginners are not going to want to figure this stuff out when Go/Rust/etc. make it so easy to get started and share your work.
- sigzero 6y ago
  I absolutely hate the way Go does packages. I've heard they've improved it, so maybe I need to take another look.
  - zemo 6y ago
    I dunno, I've been programming Go for 9 years and have had far fewer problems with dependencies than I ever did with pip, gem, or npm. I don't think it's perfect by any means, but none of them are.
- yodelshady 6y ago
  Small Rust projects are... urgh. I spend more time downloading packages, and cleaning up the sometimes hundreds of megabytes of random crap, than I do developing. Easily finished small python projects in less time than the download-compile-cycle of Rust. My python projects usually rely almost entirely on the same 3 or 4 already downloaded libraries, plus the large stdlib. They run 3.8. That experience is not comparable, and not in favour of Rust. Maybe it's because I don't use whatever Tox is, but I don't. So no. Not beginner friendly.
saila 6y ago
I think it's better to create a virtualenv for each project and install tox and other project dependencies there. In addition, I'm not sure tox-pyenv is required. I use tox with multiple pyenv-installed Python versions and have never used it. Base: - install homebrew - brew install openssl readline sqlite3 xz zlib # recommended - brew install pyenv - pyenv install <version> # for one or more versions Project: - create/clone project directory - cd project directory - pyenv local x.y.z x.y.z ... # include each version the project supports - python3 -m venv .venv # create virtualenv for project using one of the pyenv versions - install the project into the virtualenv - install any additional dependencies into this virtualenv too, if necessary Edit: I'd also add that if you do need a tool to be installed globally, it's better to use pipx or pip with --user. It's generally not a good idea to install things into a system-level Python. https://pipxproject.github.io/pipx/ https://pipxproject.github.io/pipx/
angrais 6y ago
I'm quite confused as python development under OSX is extremely simple: 1. Install pyenv [1] and the specific version you plan to use. 2. Use poetry [2] for dependency management, which pins the python version to the one defined, ideally the one you've set via pyenv. -- [1]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv [2]: https://python-poetry.org/ https://python-poetry.org/
- the_local_host 6y ago
  Assuming this combination of tools does work, it may be that it's just too hard to discover, given the large number of competing approaches described in the article and these comments.

This is why I gave up on playing super-nerd and just moving all development to docker-compose.

random5634 6y ago
I'm in Windows / Linux, my wife is on MacOS. She wanted to help with some python programming. Yep - playing super nerd is LAME. I thought getting environment setup to match mine would be pip requirements.txt but no go. Really facinating. I think things like go / rust etc? may work better - you compile everything and compile to binary on platform, so don't need to build and carry around environments everywhere. I moved to docker deploys / dockerfiles locally for pretty much everything. Totally overkill, but when I switched to working from home was so glad I could just get up and go without getting my overall system sorted out. I haven't totally figured out the WSL2 python remote interpreter / debugger code flow yet however, so still using docker. I'm sure WSL2 will or is an option. I've been tempted to just vnc or similar into a cloud instance, but my local machine is to fast by comparison for the price.
dinkleberg 6y ago
Using PyCharm with docker-compose as the interpreter works like a dream (relative to Python development). No more dependency problems and then you can also have a full setup with postgres, Celery, redis, etc. all available on the local machine. My mac is a couple years old and I don't have the fan of death problem everyone seems to complain about. The best thing is allowing my team to know with confidence we're developing against the exact same environment.

The author concludes the best solution is using brew to install different Python dependencies for different packages. I had a similar solution when I started using a Mac for work back in 2017, but around six months ago I found another way: 1. Install Docker CE on the Mac. 2. Start a container for your work, e.g., docker run python:3.8 3. When you need to get back into it, use docker exec -it <first_three_container_characters> ipython 4. If the container is shut down from a reboot or something, start it with docker start <first_three_container_characters> What's really cool about this is `bash` can be run instead of `ipython` to get a Bash prompt and install any Linux dependency. In theory, at some point if one project has conflicting dependencies with another project a new container can be spun up, but this is not something that's come up in practice yet. This has worked so well for me these past few months, I actually replicated it on my Linux computer. I took a few extra convenience steps like mounting `/home/` and aliasing `ipython` to a command that starts the container before executing the command, but other than that it's been a pretty seamless experience and I never have to worry about OS-level Python installation stuff any more. I use a scipy image with Jupyter Lab in a similar way for my graphical notebook needs. The only time I would recommend against this is when performance is an issue (like training ML models), but at work we use AWS machines with their own Docker environments for that. For everything else, Mac, Linux or maybe even Windows, Docker is the only way I now recommend firing up Python.

TheRealPomax 6y ago
And then enjoy the jet engine that your macbook fans turn into, because holy shit Docker is bad at resource management.
- vorpalhex 6y ago
  This depends on your base image and what it's doing. If you're running like a large elasticsearch cluster.. you're still running a large elasticsearch cluster. Docker overhead should generally be minimal even with the VM that's running, but make sure to use alpine/slim images.
- powerapple 6y ago
  Agree. I have never enjoyed running docker on my laptop. If I have to use Docker, I use it on a remote Linux server. I had good experience with Vagrant if I have to use a linux environment locally. The only good thing with Docker on macOS is that you don't need to sudo it.
- Wowfunhappy 6y ago
  Is the problem that Docker is bad at resource management, or that this is just what happens when you're actually running an entire second OS under the hood? I appreciate Docker's approach, but it's definitely a bit of a nuclear option (at least on non-Linux systems).
cutler 6y ago
Surely it's an admission of failure if one has to resort to Docker just to get a scripting language to function on a local machine? It's only a matter of time before someone finds a use for Kubernetes on a local machine.

SeanLuke 6y ago

> Yamale’s tox.ini, for example, depends upon two different versions of Python, 3.6 and 3.8. What? It seems to me that this awful design choice is root of the problem, and not MacOS at all.

jimminy 6y ago
Tox is designed for environment isolation, and primarily for testing against multiple release targets. It's not an awful choice, when it's exactly what you want to do. The dependency is that tox will run the tests against a Python 3.6 target environment and a Python 3.8 target environment. The dual requirement is asking for test assurance that Yamale will work with 3.6 or 3.8.
saila 6y ago
This just means that tox is configured to run the project's test suite against those two Python versions.
- SeanLuke 6y ago
  Ah, then move along.

Demiurge 6y ago

This is why I started using Conda recently. It works quite well to create isolated python environments, having many prebuilt packages, and most importantly, system level dependencies. So, you can install some of those, and for anything that is not in conda-forge, pip install works and builds.

dnautics 6y ago
just be careful. I don't know if things have improved in conda-land, but when I last used it a few years ago I table flipped and quit conda (and, ultimately, python) when the system-level dependency management got so broken that I was forced to reinstall linux from scratch and start COMPLETELY over (it had broken things outside of its domain). Tensorflow being hard-pinned to CUDA versions probably didn't help.
- xapata 6y ago
  I've been using conda for quite some time, maybe version 0, I forget, and this has never happened to me. I hope you submitted an issue.
- icedistilled 6y ago
  Having gone through the nightmare of setting up tensor flow with cuda, I can't help but think you're letting that color your view of python as a whole. Tensorflow, cuda, and nvidia were a nightmare that took me 5 days to resolve on ubuntu 18 since it was my first time setting it up ever. There was literally no working guides or people posting with my exact issue, but tons of posts about other people having a nightmare of a time. In the end I figured out I had to go find this one file/library nvidia had moved around between versions of drivers/cuda even though documentation suggested they hadn't and I was using the versions that were supposed to work fine. In the end, fixing it was either by setting an environment variable because the default path was wrong or editing the path in a config file. I'm not remembering which now.
  - dnautics 6y ago
    I dunno, I think the problem is not having a project-specific location for your deps. Obviously you want global deps for a console scripting language, but if you are doing any sort of bigger project, having a global deps location (which is implicitly resolved by the language, no less) is unacceptable for reproducible development for larger projects today. The env hack that python managers like conda and venv use is, IMO, extremely dangerous, because it's hiding state. These days I use languages that don't do this for my daily drivers, it's better for my sanity.

brundolf 6y ago

https://media.tenor.com/images/af0610f778ecdc1b7ab5a750c2d98615/tenor.gif https://media.tenor.com/images/af0610f778ecdc1b7ab5a750c2d98...

FunnyLookinHat 6y ago

I used to play this game but realized pyenv / nvm / etc. are all just anti-patterns nowadays. Use docker ( or some other containerized solution ) to run code in your local directory in a specific environment. I think the author is doing the best they can considering all of these tools and resources aren't updated to this pattern yet, but I'd hate to see this become the new normal for any developer entering into the field. What a way to scare off new dev friends!

singron 6y ago

I highly recommend nix for this type of problem, both on linux and macos. You can make all kinds of dependencies available simultaneously in the same file system. You don't have to run vms, or copy in and out of containers. The build cache works great and is much better than a linear Dockerfile. Tweaking dependencies at any level is pretty easy.

emptysea 6y ago
It's been a while since I last ventured into nix with python, but when I last tried it seemed way less user friendly compared to the current python solutions like poetry[1]. With poetry I can `poetry add foo` and `foo` will be added as a dependency and importantly all its dependencies will be locked in the `poetry.lock` file. With Nix either you write all your definitions from scratch, which feels like maintaining a lock file by hand, or you specify a package like `numpy` in your dependencies that is defined in the nix packages and hope that the next time you update nix packages nothing breaks. Also if you want a package that isn't in nix packages you have to write the definition yourself, which isn't a big deal if you start a project off with nix and gradually add dependencies as you need them, but if you are trying to migrate your project with 200+ dependencies it feels really tedious. [1]: https://python-poetry.org/ https://python-poetry.org/

they4kman 6y ago

For complicated situations like this, nothing beats MacPorts. Where brew is generally based on recipes (read: collection of bash scripts) that may or may not play well together, MacPorts is a package system with well-defined dependencies. The drawback to MacPorts is that everything is built from source, and nothing previously installed on the system is utilized -- so installing Python for the first time may take 30-60 minutes. But the next time is 5 minutes, and you can install all the versions you want, where they'll live side-by-side in peace and love and harmony. MacPorts is slower, but offers peace of mind -- 'cause when I'm trying to develop something, the last thing I want to worry about is dependency issues (gross)

icedchai 6y ago
I've seen MacPorts uses pre-built binaries, at least in some cases. Not sure if this applies to the Python ports.
pseudalopex 6y ago
MacPorts has binary packages. Homebrew uses Ruby not bash. MacPorts uses Tcl. MacPorts just has better QA.
cutler 6y ago
I don't understand. I've been using MacPorts for 16 years and I've never had to wait more than a few minutes for a Python install to complete. Most packages are pre-built. I can even get clang without having to compile. Plus one for MacPorts though. It rarely lets me down and unlike Homebrew stays in /opt/local instead of polluting /usr.

deleted 6y ago

[deleted]

zemo 6y ago

Python used to be my tool of choice for all of my small programming tasks, but nowadays I find messing with Python to be an absolutely massive time-waster for me. It used to be my favorite programming language. Now I avoid it entirely because of this nonsense.

nickt 6y ago
Where did you go from Python?
- zemo 6y ago
  I just write everything in Go nowadays, since I didn't have any hard Python requirements like numpy to worry about.
  - nickt 6y ago
    Cool, thanks.
deleted 6y ago
[deleted]

moocowtruck 6y ago

pipenv

majormunky 6y ago

Python development on a Mac works pretty well if you install it from the installer that you get from python.org, at least it has in my experience.

powerapple 6y ago

I was able to use venv and install tox and tox-pyenv locally to use tox. I also installed pyenv with brew. I had many problem with scientific packages in python projects mostly, the incompatibility between different versions of libraries, I use Conda for those projects, and venv for other projects not dependent on numpy and scikit-learn.

dmart 6y ago

I suspect that the terrible installation/package management/distribution experience is going to be the thing that ultimately drives Python into irrelevance. Beginners are not going to want to figure this stuff out when Go/Rust/etc. make it so easy to get started and share your work.

sigzero 6y ago
I absolutely hate the way Go does packages. I've heard they've improved it, so maybe I need to take another look.
- zemo 6y ago
  I dunno, I've been programming Go for 9 years and have had far fewer problems with dependencies than I ever did with pip, gem, or npm. I don't think it's perfect by any means, but none of them are.
yodelshady 6y ago
Small Rust projects are... urgh. I spend more time downloading packages, and cleaning up the sometimes hundreds of megabytes of random crap, than I do developing. Easily finished small python projects in less time than the download-compile-cycle of Rust. My python projects usually rely almost entirely on the same 3 or 4 already downloaded libraries, plus the large stdlib. They run 3.8. That experience is not comparable, and not in favour of Rust. Maybe it's because I don't use whatever Tox is, but I don't. So no. Not beginner friendly.

saila 6y ago

I think it's better to create a virtualenv for each project and install tox and other project dependencies there. In addition, I'm not sure tox-pyenv is required. I use tox with multiple pyenv-installed Python versions and have never used it. Base: - install homebrew - brew install openssl readline sqlite3 xz zlib # recommended - brew install pyenv - pyenv install <version> # for one or more versions Project: - create/clone project directory - cd project directory - pyenv local x.y.z x.y.z ... # include each version the project supports - python3 -m venv .venv # create virtualenv for project using one of the pyenv versions - install the project into the virtualenv - install any additional dependencies into this virtualenv too, if necessary Edit: I'd also add that if you do need a tool to be installed globally, it's better to use pipx or pip with --user. It's generally not a good idea to install things into a system-level Python. https://pipxproject.github.io/pipx/ https://pipxproject.github.io/pipx/

angrais 6y ago

I'm quite confused as python development under OSX is extremely simple: 1. Install pyenv [1] and the specific version you plan to use. 2. Use poetry [2] for dependency management, which pins the python version to the one defined, ideally the one you've set via pyenv. -- [1]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv [2]: https://python-poetry.org/ https://python-poetry.org/

the_local_host 6y ago
Assuming this combination of tools does work, it may be that it's just too hard to discover, given the large number of competing approaches described in the article and these comments.