17 ms·

Python Best Practices for a New Project in 2021

276 points by jedwhite 5y ago

gjvc 5y ago
Every time I see one of these type of posts I look to see the section on how they deploy their work to different environments. Every time I fail to find such a section.
- reacharavindh 5y ago
  With Python, the only sane choice is probably through Docker images :-( This is the reason why I consciously attempt to move away from Python and choose Go or Rust for new projects if possible. Of course, on existing projects, Python deployment is a pain.
  - lvncelot 5y ago
    Last I've heard, the nix experience for python was kind of lacking, sadly - but I don't have any first-hand experience to go with that statement.
  - ramtatatam 5y ago
    Why is deploying through docker images not OK? I always thought that's what docker is meant to solve.
    - willis936 5y ago
      Docker was made for development and testing of code, not production. It has been co-opted for production with good success. It seems fine to do it for hobbyist projects, but still feels dirty in a commercial setting.
      - ramtatatam 5y ago
        I'm using docker in commercial environments for years and cannot complain... But I don't distribute software, which is where the problem is probably more visible.
      - iainmerrick 5y ago
        Why?
        willis936 5y ago
        Why what?
      - evgen 5y ago
        The largest internet services on the planet were using LXC containers for production ops before you ever heard of the term 'docker'. In its earliest iterations it was easier to push a container to production than to run it on your laptop.
        KaiserPro 5y ago
        > using LXC containers for production which isn't docker. From what I recall, spotify were the first company with a large footprint to use docker. However for some reason they skipped VMs and went screaming into docker when it was _very_ new. Personally that seemed like a mistake, but you know, each to their own. If you're wanting to get into a bun fight about containers, then IBM 360 and JCL has some time for you.
      - Daishiman 5y ago
        If we stopped using software in prod because it feels "dirty" we would not be using computers at all.
    - stult 5y ago
      Desktop/local usage, especially for non technical users or people working in environments where they can’t install docker (eg locked down corporate machine). There isn’t a great way to pass around a single executable or installer for a Python app. There’s pyinstaller, but it’s finicky and doesn’t quite work cross platform (you have to build the executable on the target OS). On the other hand, OP’s setup makes it very easy to publish packages. So you can create a tarball and have a user pip install that, then run your app with a simple CLI. Or publish to pypi if you want it public. The downside is you assume the user has the right version of Python and knows how to switch versions if need be and all the weirdness that comes with that. But for web apps, packaging your app also makes it easy to wrap in a simple dockerfile that basically just installs the package and then runs it.
      - marderfarker2 5y ago
        I’ve deployed Python desktop apps quite successfully by using onedir with pysintaller and then package the contents using NSIS.
      - codethief 5y ago
        > There isn’t a great way to pass around a single executable or installer for a Python app. Have you tried Nuitka or mypyc? (Haven't tried them myself but I've heard very good things about Nuitka.)
        nerdponx 5y ago
        Also Briefcase: https://beeware.org/project/projects/tools/briefcase/ https://beeware.org/project/projects/tools/briefcase/. "Convert a Python project into a standalone native application."
        stult 5y ago
        Personally I have not. I have only had one particularly weird use case where I really needed to deliver a Python executable alongside an Electron app, and made it work with Pyinstaller. We’re actually reworking things to use Pyodide to compile the Python to WASM so we don’t have to deliver any separate Python code at all. It’s a real Rube Goldberg machine of an application, but we had a lot of weird constraints we had to meet (eg, we couldn’t use sockets to communicate between Python and Node, so had to package the Python as an executable). It works beautifully but it was a mess to get running.
  - sitkack 5y ago
    tarballs, rpms or debs also work. One should probably run their own package server like https://github.com/pypiserver/pypiserver https://github.com/pypiserver/pypiserver All of that said, containers are nice because you have a log of what is running, easy to transport and coordinate. When you use Go and Rust over Python, does the use of Docker disappear? What replaces it?
    - amrox 5y ago
      > One should probably run their own package server like https://github.com/pypiserver/pypiserver https://github.com/pypiserver/pypiserver Never used pypiserver but I’ve had a good experience with https://github.com/devpi/devpi https://github.com/devpi/devpi
  - asplake 5y ago
    I wouldn’t write off platforms such as Elastic Beanstalk and Heroku for web projects, smaller ones of bog-standard shape anyway
    - KaiserPro 5y ago
      Heroku, yes I would push for that. Its expensive, but it'll save you an entire devop function right up til you hit the 30 people mark. Elastic beanstalk is just a horrid dev environment. Lots of waiting, lots of non-obvious options, and very little reward. I would personally push for lambda and zappa (https://github.com/zappa/Zappa https://github.com/zappa/Zappa) for python, as it seems to be much easier to deploy and debug.
  - KaiserPro 5y ago
    > the only sane choice is probably through Docker images Its really not. You either tailor your dependencies to match the host (clue, you should be doing this anyway, it makes things much easier in the long run), or use virtual envs. pyenv also gives you some (non obvious) flexibility as well However, having said that, the chances are, your go or rust binary is going to be in a docker env as well(so is your python), so its basically all the same.
  - blooalien 5y ago
    I'm personally a big fan of (especially) the most recent builds of Python 3, but lately I've been interested in learning Go or Rust just because there's a few things I really like about them both vs some other languages I've looked at. So, seein' that you've coded in both Go and Rust apparently, my question to you is: Which do you prefer of the two (and why)? I personally lean a bit toward Go, but I haven't learned enough of either to decide absolutely which of the two I should learn first.
    - reacharavindh 5y ago
      I think it is better to not _decide_ which one to learn. They both have their perks and quirks. If you can, I’d recommend to dip your toes on both. For network services, Go feels super nice because of the high quality libraries in its ecosystem. I feel like it is better Java. Rust on the hand, feels like a better C++... with a steep learning curve, and useful when you can/need to get something correct/efficient/safe at the cost of increased developer time and cognitive load. They both have their place IMHO.
      - blooalien 5y ago
        Nice. Thank you for the info. Do they both have pretty good GUI toolkit support available? I currently use PyQt5 in Python, but I'm not totally against GTK if that's what is easier for Go or Rust. Still think I kinda feel like Go's the one of the two I'm prolly gonna learn first, although you've got me leanin' hard toward also learning Rust in addition to Go.
  - alephu5 5y ago
    If you're deploying a REST API with authentication, caching and so on then the KISS solution is python with a framework (such as Django) in a docker container. I've spent many many hours (years) trying nix, rust, Haskell, go, spring framework and all sorts of other things which are a lot of fun but not so good for getting shit done. For other domains this doesn't apply of course; lower-level network stuff, portable CLIs, CPU-intensive workloads and so on are much better in go/rust but you can either integrate them with a network call, spawning an OS process or an FFI in the case of rust.
  - aitchnyu 5y ago
    I'm using Pycharm Pro and it can use Docker for code completion. I use Docker AND venv on host filesystem for friends using Pycharm Community and VSCode. Is this still necessary?
- inopinatus 5y ago
  Most production deployment goes like this: 1. Put files on a server, 2. Start a process. How much ancillary scaffolding you put around that is a matter of taste.
  - thundergolfer 5y ago
    If your Python code has native code under the hood (eg. Numpy) this two-step is very much a 'step1: circle, step 2: draw the rest of owl' description of production deployment.
    - Doxin 5y ago
      If you're deploying to a server you'll want to pip install dependencies on the server. If you're using docker you'll want to pip install dependencies in your container. If you're deploying to an end users computer you'll want to use pyinstaller, which admittedly is not trivial to get working in all cases.
      - gjvc 5y ago
        This is exactly my point. .venv/ needs to live in the project deployment directory, and needs a wrapper script to use the python binary/symlink therein. Isolate the project from the python runtime it uses, and you'll always have the right set of packages installed. None of this is perfect, but an in-tree .venv/ and convenience scripts seems to be the least-worst option.
- marderfarker2 5y ago
  For Conda, you can transplant the envs sub folder (e.g. C:\Users\<user>\Conda\envs\<env name>) to another computer with Conda installed and activate the env there. Make sure the environment is setup empty before development begins.
- StevePerkins 5y ago
  If this is a company project, a REST API or task service of some kind, then you are probably using Docker in the year 2021. If this is anything else, or if you work for a shop that hasn't embraced containerization, then you use PyInstaller (http://www.pyinstaller.org http://www.pyinstaller.org) to bundle your application. Either into a directory that contains your full Python virtual environment (only 5-10 megs!), or into a single executable file. The latter is most convenient for a Go/Rust type experience. But the former will startup faster, because that single-file executable has to first uncompress itself to the system temp directory.
- gjvc 5y ago
  I was overly brief here. I meant to say "I look to see how they are going to deploy and use the virtualenv they have created." Hint: what works in development -- just telling people to type "source .venv/bin/activate.sh" or such, doesn't fly in an unattended environment. All that it requires, of course, is a bin/venv-python wrapper (bash) script to reference the created .venv/ directory, so this is hardly ground-breaking stuff, but as I mentioned originally, this (crucial) section is missed every time. I use a minimal-but-complete pairing of venv and pip, and a couple of location-independent wrapper scripts, and I can run things the same across all environments.
Dunedan 5y ago
The headline is pretty misleading, as the mentioned "best practices" are highly opinionated, as the author also states: > This is my very opinionated attempt to compile some of the best practices on setting up a new Python environment for local development.
- EamonnMR 5y ago
  These are fairly close to our working environment (we're using pylint instead of Flake8, haven't switched most of our projects to poetry yet, and we use Pedantic but not MyPy yet.)
  - tlocke 5y ago
    I think you mean Flake8 rather than Sense8? Anyway, gives me a chance to plug Flake8 Alphabetize, a Flake8 plugin for import ordering https://github.com/tlocke/flake8-alphabetize https://github.com/tlocke/flake8-alphabetize
    - claytonjy 5y ago
      How does this compare/contrast with isort?
      - tlocke 5y ago
        Flake8 Alphabetize will just give you warnings about import order, whereas isort will actually change the code itself. So as I see it there are two types of tool, formatters and checkers. A formatter doesn't alter the Abstract Syntax Tree (AST). In other words it doesn't alter the meaning of the code, just how it looks (eg. the formatter Black). A checker on the other hand looks only at the AST and just gives warnings in your editor, and then it's up to you to change it or not. The import order is fixed in the AST, so it falls outside the scope of Black (which never changes the AST). So I felt there was a need for a tool that worked as a checker that just gives warnings in your editor if your imports don't conform to PEP8, and hence Flake8 Alphabetize. It's worth mentioning that Flake8 Alphabetize follows Black's philosophy of having only one way of doing things, so a project can standardise on Flake8 Alphabetize and everyone's imports will look the same. Anyway, any feedback is welcomed:-)
yvan 5y ago
In the article the author mention that VS Code doesn't support Poetry venv. This can be solve by configuring Poetry to create the virtualenvironment directly in the project folder (which in my opinion is neater). Here is the config https://python-poetry.org/docs/configuration/#virtualenvsin-project https://python-poetry.org/docs/configuration/#virtualenvsin-...
- anewhnaccount2 5y ago
  You can also just create .venv as a symlink.
stult 5y ago
I use this exact set up and it’s great. Only thing I would add is using cookiecutter, which makes adding all these bells and whistles extremely easy.
wwwhizz 5y ago
"best practices" `curl https://pyenv.run https://pyenv.run | bash` hmmm...
- maccard 5y ago
  What's the problem here? The script is served over https, so it's not going to be tanpwred with (unless you have a malicious cert, but at that point you can't trust anyone), and curl | bash isn't any worse than downloading a script and just running it, or running a precompiled binary you don't trust.
  - Cthulhu_ 5y ago
    The request itself won't be tampered with, but what if the host was? That endpoint could be compromised and send you a different script. They should offer a download with signature validation instead. Signed by Apple, Microsoft, etc if possible.
    - maccard 5y ago
      If you're afraid the host may be untrusted then you would be wrong to download any of their code at all. The safety is in reviewing the code there, not in avoiding curl | bash. Running pip install or npm install is just as dangerous. > They should offer a download with signature validation instead. Signed by Apple, Microsoft, etc if possible. If the host is compromised, the attacker will just get Microsoft to sign their malware instead; see [0]. If the host is compromised, and you run the code without reviwing it, you're hosed regardless. [0] https://arstechnica.com/gadgets/2021/06/microsoft-digitally-signs-malicious-rootkit-driver/ https://arstechnica.com/gadgets/2021/06/microsoft-digitally-...
    - dragonwriter 5y ago
      > The request itself won't be tampered with, but what if the host was? What if your distro package repository was?
  - doix 5y ago
    pyenv could get taken over and you won't know. It's also possible to detect when someone is piping to bash (on the server) and serve a different payload [0]. You're better off piping curl to a file, reviewing the file and then running it manually. [0]: https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...
Tipewryter 5y ago
I prefer containers over pyenv and poetry. This way not only python version and dependencies are "in one place" but also all other stuff that comes along with a new project. The OS, the database etc. The one thing I dislike about Python projects is that Python plasters the compile cache files all over the place. Is there a reason to change that? Currently I use the -B flag for all my scripts. But that makes it slow. I wish Python would have an option to perform like PHP and keep cached compilations in memory instead on disk. Or at least somewhere in /tmp/.
- willis936 5y ago
  If you need a specific OS then isn't a VM the solution rather than a container? Also at what point do people just realize that all of this overhead is a gigantic waste of time and just use a better language?
  - skohan 5y ago
    Yeah to be honest if you need containers to make a project reproducible this is just a sign of failure. You're basically saying you need to encapsulate the entire system for your code to run correctly.
    - cpfohl 5y ago
      Loads of tools have external dependencies that are hard dependencies. And library search paths is my machine vary from those on prod... I could go on, but I'm not sure I understand why using a container to manage all that is a failure?
      - skohan 5y ago
        There are plenty of reasons why you might want to containerize a project. If you have a lot of system dependencies, for example, you might want to consider including a Dockerfile in your project to make it portable. However what makes Python a failure is that people feel they need this to dependably run a python program which only has pure-python dependencies. Compare this to a language like Rust, or the NPM ecosystem. In those cases, the tools have managed to dependably encapsulate projects such that you only need the package manager to make a project fully repeatable. With either of those ecosystems, there's basically one system dependency, and you can find any repository online and dependably do `git clone ...` then `cargo build` etc. to make it work. With Python, you effectively have to reproduce the original developer's system, and that is a failure.
santiagobasulto 5y ago
This is pretty much the same thing that I use. Great write up. If you use ohmyzsh, there are plugins for poetry and pyenv too.
kuu 5y ago
I'm a bit surprised that conda is not mentioned anywhere (I'm not saying as best practice, but as an alternative or why envs are better)
- karxxm 5y ago
  Yup, especially when you are working with os-level dependencies like cuda, conda is the way to go. I think every tensorflow user can approve
  - codethief 5y ago
    …unless you're working on a project for arm64-based IoT devices (like Nvidia Jetson), where Conda is not available.
    - hantusk 5y ago
      Does it not work with miniforge/mambaforge? (https://github.com/conda-forge/miniforge https://github.com/conda-forge/miniforge) I use this with my m1 mac, and it works great.
    - wdroz 5y ago
      conda is now available[1] for arm64 (since March[2]). [1] -- https://docs.conda.io/en/latest/miniconda.html#linux-installers https://docs.conda.io/en/latest/miniconda.html#linux-install... [2] -- https://github.com/conda/conda/issues/8297 https://github.com/conda/conda/issues/8297
      - codethief 5y ago
        Awww man. I must have missed that announcement by only a few days.
  - dijksterhuis 5y ago
    Quite confused about this comment as many of the latest tensorflow V2 releases aren't reliably uploaded to conda forge. IIRC there's only like 4 or so of the V2 releases uploaded. You can conda install the CUDA dependencies and then install the required Tensorflow version via conda pip. But that's not much different to installing CUDA manually and then installing tf from system pip. It's much faster and easier to pull a tf Docker image as it's their "officially supported" way to get up and running. So... as a tf user and a sysadmin... Nah. No conda for me thanks.
Donckele 5y ago
Please remove “Best Practices” and year from the title. Completely irrelevant and misleading.
- codethief 5y ago
  Why? The year in particular seems very relevant because the "recommended Python project setup" (if there ever was such a thing) changes every other year or so.
  - Donckele 5y ago
    Well at least next year it will be officially out of date.
- plesiv 5y ago
  You should expect that "it's my opinion" is implied for these kind of blog posts.
  - Donckele 5y ago
    Maybe we should have a prefix “My Opinion” for those easily confused.
nxpnsv 5y ago
Very close to my setup actually, there is a lot of good tips in this article.
- Spivak 5y ago
  Agreed, this is a pretty solid setup that will get you 90% of the way there for any Python project.
kissgyorgy 5y ago
For VSCode to recognize the virtualenvs created by Poetry, you just have to add a setting for the Python extension: "python.venvFolders": [ "~/.cache/pypoetry/virtualenvs", "~/.pyenv/versions", ], This will make VSCode automatically recognize virtualenv interpreters, so you can select them without starting code from an activated shell.
petulla 5y ago
why poetry over pyenv-virtualenv?
- EamonnMR 5y ago
  What poetry replaces is pip and requirements.txt. Our team had a lively discussion about this, but here are some good reasons: * Keep test frameworks out of production (in case they've busted something; the two points below can happen to test libraries too.) * pip recently changed the way it's resolver works[1], breaking numerous projects. Yeah yeah, it's a major version bump, but lots of containers and environments installed pip latest just sort of assuming that it would always handle requirements.txt the same way. With that contract broken, now using pip directly isn't a non-decision anymore. * Locking specific versions of dependencies can you roll back Bad News in production. Let's say you have a project with library A which has dependency B. If A asks for the latest version of B, you can be in a situation where a new version of B breaks your project and you won't know about it until you do a release _and_ if you try to roll back you'll still be in trouble. We recently had to deal with a similar issue and had to fall back on an older container until we could figure it out. [1] https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html?m=1 https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.h...
  - mvanbaak 5y ago
    pip is not to blame for the above points. * if your code is under src/ and your tests under test/, you use a requirements-test (or better, tox.ini) your tests including the dependencies like pytest dont end up in a wheel * if a container depends on 'latest' and not some semver major number, it's 100% the containers fault when they blindly update to a new major version * 'pip freeze' is your friend
    - EamonnMR 5y ago
      Agreed, Pip isn't to blame, treating `pip install -r requirements.txt` as one's sole dependency management is. You can choose to build your own dependency management practice around pip, or use one someone else has already created. I think that it is easier to get a team on the same page with something like Poetry, especially if they're used to bundler or npm. You're right about the docker containers as well, but upstream does what it wants and downstream has a strong tendency to not mess with upstream's choices.
EamonnMR 5y ago
One thing I'd add herr here is Pydantic. It compliments the type hinting very nicely and lets you get more static with your classes. But that you don't just install, you need to use. Which is another omission, I suppose; python's type hints are opt in, so mypy won't do you any good unless you actually add type hints, right?
- daviddavis 5y ago
  There are a bunch of mypy configuration options to disallow certain things like --disallow-untyped-defs. There’s also --strict which forces typing on everything.
  - kortex 5y ago
    Man, developing python with pydantic and "mypy --strict" (I follow pydantic's config [0] where I can) is such Type 2 Fun. It feels like a totally different language. Yeah it takes a little more time at first but then type inference and autocomplete starts to kick in and then you're screaming fast. And you "compile" it and everything just works. No hunting down edge cases or tracebacks cause you forgot to catch a None. I find it super satisfying. Much easier to stay in flow state when you aren't having to stop every few minutes to test stuff and dig through tracebacks. [0] https://github.com/samuelcolvin/pydantic/blob/master/setup.cfg#L48 https://github.com/samuelcolvin/pydantic/blob/master/setup.c...
dpatterbee 5y ago
> some Linux distributions already switched to version 3 At the lightning pace of only 12 years since the release of python 3 (and 15 months since python 2's deprecation), some linux distribution have ALREADY switched to it.
- skohan 5y ago
  The developer story around python 2/3 is emblematic of why I don't consider it a suitable language for software development outside of very controlled cases like notebooks.
  - leetrout 5y ago
    Eh. That’s all passed. Now we’re arguing about pip vs pipenv / pip file or poetry.
    - kissgyorgy 5y ago
      No we are not :) poetry won, it's just way better than pipenv. In fact, pipenv was just always shit.
      - leetrout 5y ago
        Have you seen PDM? https://pdm.fming.dev/ https://pdm.fming.dev/
  - smcl 5y ago
    I dunno, they made some big breaking changes but continued supporting 2.x for over a decade after 3.x. There are definitely reasons I would not consider Python for some projects, but the 2/3 transition is not one of them. To be honest the event is so infamous (and IMO a little overblown) that they basically couldn't do this anymore, and I think it's been openly stated that they wouldn't.
    - skohan 5y ago
      What gets me about it is that the 2/3 transition was just a massive unsolved problem, and one which caused pain over a decade after 3.x was released. Other ecosystems managed to solve it - like Rust for example handles editions in a very sound way - but with Python you had so many problems cropping up over the years over python versions.
rwmark 5y ago
I wonder why VSCode is at least name-dropped in these Python setup tutorials lately. VSCode is spying on you and sending back "telemetry" data. Given the current situation in Python, where there is little development, the old boys have totalitarian control, and new contributors are smart enough to avoid that mess: Try out Rust, Go, Elixir or Lua instead. It might save you a lot of trouble. Heck, if you are willing to put in a lot of time to create carefully written objects, C++11 code can look a lot like Python (if you are into that).
- hetspookjee 5y ago
  My experience is also that PyCharm is much more pleasureable to work with and on a safer side. I checked 1.5 year ago, so could be changed, for remote development on a server and had to download some 3rd party Rsync tools to be able to do so. For PyCharm is just works out of the box with a professional license.
  - rwmark 5y ago
    Indeed, anything but VSCode: "What data is collected. The GitHub Copilot collects activity from the user's Visual Studio Code editor, tied to a timestamp, and metadata." PyCharm, Emacs, Vim, are all better and don't spy.
    - excitednumber 5y ago
      I came here to read this.
    - btdmaster 5y ago
      PyCharm has similarly bad privacy: https://www.jetbrains.com/legal/docs/privacy/privacy.html https://www.jetbrains.com/legal/docs/privacy/privacy.html. I'm really happy though that I learned to use Emacs, really useful and language independent.
  - 0x008 5y ago
    VS Code experience has definitely improved a LOT in the past 1.5 years with the new python language server (pylance) and remote development being improved quite a bot. We use it daily in production in my company with almost zero problems so far.
hazbot 5y ago
Every time I see one of these threads I think "maybe Python isn't the best choice if we still need threads like this". But it's just so useful...
thundergolfer 5y ago
The lack of discussion of lock files is a big omission in any 'best practices' page. "Lock" only appears in a log output of Poetry. Even if the answer is "Poetry handles it" you certainly want to explain why they're important just like is being done in the rest of the "Why use..." sections.
- nerdponx 5y ago
  For a generic lockfile, use the pip-compile tool from https://pypi.org/project/pip-tools https://pypi.org/project/pip-tools
  - beaugunderson 5y ago
    Huge +1 for this; pip-compile is so much faster and less surprising than poetry. Poetry's #1 footgun is that it updates the version of unrelated dependencies when you add a new dependency. No thanks.
  - pmkiwi 5y ago
    Yeap I am using pip-compile too because I always managed to mess up my environment when using poetry!
- 0x008 5y ago
  Yes! Lockfiles are basically the only reason to use poetry over pip+venv or am I missing something?
cutler 5y ago
Why is all this unnecessary with Ruby considering the languages are so similar?
- e12e 5y ago
  Much similar work is required for ruby, but gem/bundler is less fraught with fragmentation than pip/poetry/venv.
  - wink 5y ago
    I think a big part of the problem is that bundler has been around for ages whereas half of the tools mentioned here didn't even exist just 5-6 years ago. Which can be seen by this hilarious sentence: > By default, Python packages are installed with pip install. In reality nobody uses it this way. It installs all your dependencies into one version of Python interpreter which messes up dependencies. Why hilarious? Because everyone I know uses `pip install` if they're not using poetry. And because poetry is kinda new, that's a lot of people.
    - dmurray 5y ago
      The quote isn't clear, but I think the author means people don't generally run pip with their system python. I.e. they don't run /usr/bin/pip or whatever, but use some virtual environment solution.
      - e12e 5y ago
        Which is the same for "gem install" - you don't want to go "sudo gem install".
        dmurray 5y ago
        gem catches this by default and warns you. Pip doesn't, but it would probably be good if it did and the pip guys explicitly took the position that "installing stuff on your system python is not a normal use of it".
mikesabbagh 5y ago
I am surprised that containers are not mentioned here. We do similar but use containers instead of env.
- amrox 5y ago
  Containers are great for web services or anything that primarily communicates through a port. Less so for CLI style projects that read and write to the local file system. Yes I know you can bind mount directories but it’s clunky and you also usually have to fight with file permissions issues.
deleted 5y ago
[deleted]
gillesjacobs 5y ago
Great recommendations! Exactly my setup, except add pipx for global system tools, i.e. command line tools distributed through pypy (youtube-dl, black, etc.). I also add Docker or Songularity containers as needed for deep learning deployment but that is quite computing platform and application specific.
- blooalien 5y ago
  Love pipx. Great tool. I personally use that and poetry both quite heavily.
- bobuk 5y ago
  probably you've meant pypi not pypy
  - gillesjacobs 5y ago
    Yes, indeed I mean the Python Package Index (PyPI). Mypy, pypy, pypi, pyenv, pipenv, pip, pipx. Easily confusable names are a core tenet of the Python ecosystem!
jtdev 5y ago
venv + requirements.txt is simple and works. Why are people so obsessed with replacing this simple toolset/process with poorly maintained tools like Poetry/PyEnv?
- philote 5y ago
  Why do you think those tools are poorly maintained? How do you handle locking in specific versions of libraries and keeping those properly maintained?
  - nomdep 5y ago
    With pip-tools. https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools And you can still use standard setup.cuff and pip install -e unlike Poetry. Also, much faster.
  - jtdev 5y ago
    990 open issues… bugs going back to 2018 still open… https://github.com/python-poetry/poetry/issues https://github.com/python-poetry/poetry/issues Please help me understand why I would want to lock specific versions of libraries in my Python projects. I’ve worked extensively with Python for a number of years and have never needed a lock file for my dependencies.
claytonjy 5y ago
Why use the external pre-commit repos instead of pointing to the poetry-managed local installations for flake8, black, isort, etc? The repos approach results in a much simpler and cleaner pre-commit config, but since you end up with two pinned versions of each, managed by different updaters, it can invite drift which can be very hard to track down. I'd prefer to only use the pre-commit versions of these libs, but then I'd sacrifice editor integration.
super_mario 5y ago
Why use pyenv when venv is built into Python 3? Also, pip installs your dependencies into active venv, so I don't see a point in using anything else.
- blooalien 5y ago
  Because pyenv [1] and venv [2] are entirely different things that do two completely different jobs? [1]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv [2]: https://docs.python.org/3/library/venv.html https://docs.python.org/3/library/venv.html
edumucelli 5y ago
That is a great write-up! One extra bit I'd recommend to this list is using https://github.com/ikamensh/flynt https://github.com/ikamensh/flynt to convert string format into f-strings. It requires Python 3.6.
WillDaSilva 5y ago
Great recommendations all around. It'd be nice if they went into how they handle dealing with C extensions, as that adds a large amount of complexity to a Python project, and most of the published advice on the topic is quite old. EDIT: Mentioning `pyproject.toml` and the relevant PEPs would also be great (i.e. PEP 621, PEP 517, etc.). Fortunately Poetry is compliant with these PEPs.
- gww 5y ago
  I really struggle with this part of Python package development. I have a rather complex C++ library I have been calling with Cython but I am struggling to find the best practices for either compiling the library with setup tools or calling cmake and copying the library into the package etc.
  - thundergolfer 5y ago
    Look into Bazel. Bazel particularly suited to building Python+{C/C++/Rust} codebases. It's still early days in the Bazel ecosystem, so you won't find a ready-made solution, but the fundamentals are solid.
    - gww 5y ago
      Oh cool thanks, I will check it out!
- paulross 5y ago
  This might help: https://pythonextensionpatterns.readthedocs.io/en/latest/index.html https://pythonextensionpatterns.readthedocs.io/en/latest/ind...
  - WillDaSilva 5y ago
    That's a great resource in general for Python extensions, but doesn't have much to say about packaging and distributing them. For that the best resources I've found have been to look at what large/complex projects that use them do, as they've often had to deal with many of the odd cases that often come up. Using Cython adds another layer of complexity to the packaging/distribution that that link does not address. Fortunately now that you can specify build requirements in `pyproject.toml` Cython has become significantly easier to use on that front, but there are still some less than obvious bits to say the least. Maybe I should publish an overview of Python best practices for publishing C extensions (with or without Cython).
daniellarusso 5y ago
Is there a node/react/js equivalent of this article?
- DrammBA 5y ago
  I'm also interested in this, but I'm afraid the answer will be "CRA/Next/Nest or deploy your own webpack/babel".
pizza 5y ago
Awesome article but damn if that isn't some cognitive overhead just to get started (for newbies, at least)
rbanffy 5y ago
The pain and suffering I see with different versions of Python installed with Pyenv in my team and surrounding ones is huge. I advise anyone to just stay away from it. On Macs, you can install MacPorts and it'll bring in every version of Python, from ancient to newest, in a sane BSD-like way (not in a user-writable folder, ffs). On Linux the default package managers offer pretty much every version of Python that's not outright dangerous to run. On Windows there are binary installers and every Python goes in a separate folder. Not only it's not needed, it creates a layer of "magic" the user has to understand on top of the environment. And, if you really want to use brew, you can still continue using it, knowing it only has python 3.9 at the moment. Plus, it can coexist mostly peacefully with Macports.
- mdaniel 5y ago
  > knowing it only has python 3.9 at the moment. Is incorrect, they just move older versions to their "@" nomenclature, for example https://formulae.brew.sh/formula/python@3.7 https://formulae.brew.sh/formula/python@3.7
  - rbanffy 5y ago
    Interesting. Thank you for correcting me. Can you install multiple versions this way? My colleagues are suffering with that and I'd prefer not to move them from Homebrew if avoidable.
    - mdaniel 5y ago
      For sure, yes, they're distinctly named formulae and thus subject to their own install, remove, and version bump lifecycle management IIRC the trick is that the "at" versions don't get "brew link"-ed by default, since they'd almost certainly smash on top of the non-at binaries or manpages or whatever. Using `PATH=$(brew --prefix "python@3.7")/bin:$PATH` or ones favorite context switching gizmo will help, or (with the python ones specifically) using `$(brew --prefix "python@3.7")/bin/python -m venv ...` is a great way to avoid having a lot of special env-var silliness
therealmrmatt 5y ago
If anyone is interested I have a GitHub template that is essentially a simpler of the scaffolding the author lays out (https://github.com/mplemay/python-template https://github.com/mplemay/python-template).
foobarbecue 5y ago
I use conda environments and have found they do everything described here. Is there a reason I should try pyenv & poetry?
jonnycomputer 5y ago
pyenv + poetry ?? I thought you just needed poetry? I've been using conda, but have begun contemplating a switch to poetry. So what am I missing?
peterthehacker 5y ago
This title is misleading to me. It’s not about “best practices” but instead about “best tools”.
deleted 5y ago
[deleted]
aunty_helen 5y ago
There should be an article like this for every language, as the author mentions it's not just syntax to change a language. Much of it is experience based knowledge. However, this is how you would setup for a new project only and in a piecemeal manner, if you didn't already have some existing template. Compatibility issues with your main dev tool would rule out using Poetry, in the same way running the latest versions of any software without reason is a rookie mistake. Chasing a higher version number is a jr / intermediate folley. If you're developing python professionally, save your money and just pay for pycharm. The morning of fluffing with a dozen new tools which then have their own maintanence overheads is less cost effective than buying a product that gives you >70% of what the author has recommended and it does so in a generally consistent manner. This blog post was a great reminder of how much pycharm gives me on a day to day basis, how people get lost in their +1 more tool mindset and how switching languages is an intial cognitive overload.
l0b0 5y ago
The current work project[1] has all of these, with one substitution: Pyenv, Poetry, Pytest, pytest-cov with 100% branch coverage, pre-commit, Pylint rather than Flake8, Black, mypy (with a stricter configuration than recommended here), and finally isort. These are all super helpful. There's also a simpler template repo[2] with almost all of these. [1] https://github.com/linz/geostore/ https://github.com/linz/geostore/ [2] https://github.com/linz/template-python-hello-world https://github.com/linz/template-python-hello-world
0x008 5y ago
What is almost never mentioned is that the default virtual env will always set the python binary it is called from as the python interpreter for that environment. So if I want to use Python3.9 i go python3.9 -m venv .venv, if I want to use python3.6 i go python3.6 -m venv .venv etc. Yes, lockfiles are great, but while everyone can execute the above commands, it is a lot harder to convince everyone in my organization to switch to poetry.
bluewalt 5y ago
> Some people work with code from 13" laptops. And viewing diff of two files of 79 char per line max becomes very convenient. I think it's not the only reason. Limiting columns forces developers to read more vertically, which is less tiring for eyes. For example, try to read a book on a 27' monitor with no max width. Going to the next line will quickly become painful.
bluewalt 5y ago
I replaced all the pyenv (and any version management in my projects) by the awesome VS Code remote development with containers. This is a game changer to me. 1 container = 1 clean installation, completely isolated, and you can share with your teammates. Then I disable the use of virtual environments in Poetry, because it's a useless overhead. Then for updates, I just change versions in my Dockerfile or .toml and rebuild the container from scratch in a few time, which is cleaner than manual updates for everything IMO.

276 points by jedwhite 5y ago

gjvc 5y ago

Every time I see one of these type of posts I look to see the section on how they deploy their work to different environments. Every time I fail to find such a section.

reacharavindh 5y ago
With Python, the only sane choice is probably through Docker images :-( This is the reason why I consciously attempt to move away from Python and choose Go or Rust for new projects if possible. Of course, on existing projects, Python deployment is a pain.
- lvncelot 5y ago
  Last I've heard, the nix experience for python was kind of lacking, sadly - but I don't have any first-hand experience to go with that statement.
- ramtatatam 5y ago
  Why is deploying through docker images not OK? I always thought that's what docker is meant to solve.
  - willis936 5y ago
    Docker was made for development and testing of code, not production. It has been co-opted for production with good success. It seems fine to do it for hobbyist projects, but still feels dirty in a commercial setting.
    - ramtatatam 5y ago
      I'm using docker in commercial environments for years and cannot complain... But I don't distribute software, which is where the problem is probably more visible.
    - iainmerrick 5y ago
      Why?
      - willis936 5y ago
        Why what?
    - evgen 5y ago
      The largest internet services on the planet were using LXC containers for production ops before you ever heard of the term 'docker'. In its earliest iterations it was easier to push a container to production than to run it on your laptop.
      - KaiserPro 5y ago
        > using LXC containers for production which isn't docker. From what I recall, spotify were the first company with a large footprint to use docker. However for some reason they skipped VMs and went screaming into docker when it was _very_ new. Personally that seemed like a mistake, but you know, each to their own. If you're wanting to get into a bun fight about containers, then IBM 360 and JCL has some time for you.
    - Daishiman 5y ago
      If we stopped using software in prod because it feels "dirty" we would not be using computers at all.
  - stult 5y ago
    Desktop/local usage, especially for non technical users or people working in environments where they can’t install docker (eg locked down corporate machine). There isn’t a great way to pass around a single executable or installer for a Python app. There’s pyinstaller, but it’s finicky and doesn’t quite work cross platform (you have to build the executable on the target OS). On the other hand, OP’s setup makes it very easy to publish packages. So you can create a tarball and have a user pip install that, then run your app with a simple CLI. Or publish to pypi if you want it public. The downside is you assume the user has the right version of Python and knows how to switch versions if need be and all the weirdness that comes with that. But for web apps, packaging your app also makes it easy to wrap in a simple dockerfile that basically just installs the package and then runs it.
    - marderfarker2 5y ago
      I’ve deployed Python desktop apps quite successfully by using onedir with pysintaller and then package the contents using NSIS.
    - codethief 5y ago
      > There isn’t a great way to pass around a single executable or installer for a Python app. Have you tried Nuitka or mypyc? (Haven't tried them myself but I've heard very good things about Nuitka.)
      - nerdponx 5y ago
        Also Briefcase: https://beeware.org/project/projects/tools/briefcase/ https://beeware.org/project/projects/tools/briefcase/. "Convert a Python project into a standalone native application."
      - stult 5y ago
        Personally I have not. I have only had one particularly weird use case where I really needed to deliver a Python executable alongside an Electron app, and made it work with Pyinstaller. We’re actually reworking things to use Pyodide to compile the Python to WASM so we don’t have to deliver any separate Python code at all. It’s a real Rube Goldberg machine of an application, but we had a lot of weird constraints we had to meet (eg, we couldn’t use sockets to communicate between Python and Node, so had to package the Python as an executable). It works beautifully but it was a mess to get running.
- sitkack 5y ago
  tarballs, rpms or debs also work. One should probably run their own package server like https://github.com/pypiserver/pypiserver https://github.com/pypiserver/pypiserver All of that said, containers are nice because you have a log of what is running, easy to transport and coordinate. When you use Go and Rust over Python, does the use of Docker disappear? What replaces it?
  - amrox 5y ago
    > One should probably run their own package server like https://github.com/pypiserver/pypiserver https://github.com/pypiserver/pypiserver Never used pypiserver but I’ve had a good experience with https://github.com/devpi/devpi https://github.com/devpi/devpi
- asplake 5y ago
  I wouldn’t write off platforms such as Elastic Beanstalk and Heroku for web projects, smaller ones of bog-standard shape anyway
  - KaiserPro 5y ago
    Heroku, yes I would push for that. Its expensive, but it'll save you an entire devop function right up til you hit the 30 people mark. Elastic beanstalk is just a horrid dev environment. Lots of waiting, lots of non-obvious options, and very little reward. I would personally push for lambda and zappa (https://github.com/zappa/Zappa https://github.com/zappa/Zappa) for python, as it seems to be much easier to deploy and debug.
- KaiserPro 5y ago
  > the only sane choice is probably through Docker images Its really not. You either tailor your dependencies to match the host (clue, you should be doing this anyway, it makes things much easier in the long run), or use virtual envs. pyenv also gives you some (non obvious) flexibility as well However, having said that, the chances are, your go or rust binary is going to be in a docker env as well(so is your python), so its basically all the same.
- blooalien 5y ago
  I'm personally a big fan of (especially) the most recent builds of Python 3, but lately I've been interested in learning Go or Rust just because there's a few things I really like about them both vs some other languages I've looked at. So, seein' that you've coded in both Go and Rust apparently, my question to you is: Which do you prefer of the two (and why)? I personally lean a bit toward Go, but I haven't learned enough of either to decide absolutely which of the two I should learn first.
  - reacharavindh 5y ago
    I think it is better to not _decide_ which one to learn. They both have their perks and quirks. If you can, I’d recommend to dip your toes on both. For network services, Go feels super nice because of the high quality libraries in its ecosystem. I feel like it is better Java. Rust on the hand, feels like a better C++... with a steep learning curve, and useful when you can/need to get something correct/efficient/safe at the cost of increased developer time and cognitive load. They both have their place IMHO.
    - blooalien 5y ago
      Nice. Thank you for the info. Do they both have pretty good GUI toolkit support available? I currently use PyQt5 in Python, but I'm not totally against GTK if that's what is easier for Go or Rust. Still think I kinda feel like Go's the one of the two I'm prolly gonna learn first, although you've got me leanin' hard toward also learning Rust in addition to Go.
- alephu5 5y ago
  If you're deploying a REST API with authentication, caching and so on then the KISS solution is python with a framework (such as Django) in a docker container. I've spent many many hours (years) trying nix, rust, Haskell, go, spring framework and all sorts of other things which are a lot of fun but not so good for getting shit done. For other domains this doesn't apply of course; lower-level network stuff, portable CLIs, CPU-intensive workloads and so on are much better in go/rust but you can either integrate them with a network call, spawning an OS process or an FFI in the case of rust.
- aitchnyu 5y ago
  I'm using Pycharm Pro and it can use Docker for code completion. I use Docker AND venv on host filesystem for friends using Pycharm Community and VSCode. Is this still necessary?
inopinatus 5y ago
Most production deployment goes like this: 1. Put files on a server, 2. Start a process. How much ancillary scaffolding you put around that is a matter of taste.
- thundergolfer 5y ago
  If your Python code has native code under the hood (eg. Numpy) this two-step is very much a 'step1: circle, step 2: draw the rest of owl' description of production deployment.
  - Doxin 5y ago
    If you're deploying to a server you'll want to pip install dependencies on the server. If you're using docker you'll want to pip install dependencies in your container. If you're deploying to an end users computer you'll want to use pyinstaller, which admittedly is not trivial to get working in all cases.
    - gjvc 5y ago
      This is exactly my point. .venv/ needs to live in the project deployment directory, and needs a wrapper script to use the python binary/symlink therein. Isolate the project from the python runtime it uses, and you'll always have the right set of packages installed. None of this is perfect, but an in-tree .venv/ and convenience scripts seems to be the least-worst option.
marderfarker2 5y ago
For Conda, you can transplant the envs sub folder (e.g. C:\Users\<user>\Conda\envs\<env name>) to another computer with Conda installed and activate the env there. Make sure the environment is setup empty before development begins.
StevePerkins 5y ago
If this is a company project, a REST API or task service of some kind, then you are probably using Docker in the year 2021. If this is anything else, or if you work for a shop that hasn't embraced containerization, then you use PyInstaller (http://www.pyinstaller.org http://www.pyinstaller.org) to bundle your application. Either into a directory that contains your full Python virtual environment (only 5-10 megs!), or into a single executable file. The latter is most convenient for a Go/Rust type experience. But the former will startup faster, because that single-file executable has to first uncompress itself to the system temp directory.
gjvc 5y ago
I was overly brief here. I meant to say "I look to see how they are going to deploy and use the virtualenv they have created." Hint: what works in development -- just telling people to type "source .venv/bin/activate.sh" or such, doesn't fly in an unattended environment. All that it requires, of course, is a bin/venv-python wrapper (bash) script to reference the created .venv/ directory, so this is hardly ground-breaking stuff, but as I mentioned originally, this (crucial) section is missed every time. I use a minimal-but-complete pairing of venv and pip, and a couple of location-independent wrapper scripts, and I can run things the same across all environments.

Dunedan 5y ago

The headline is pretty misleading, as the mentioned "best practices" are highly opinionated, as the author also states: > This is my very opinionated attempt to compile some of the best practices on setting up a new Python environment for local development.

EamonnMR 5y ago
These are fairly close to our working environment (we're using pylint instead of Flake8, haven't switched most of our projects to poetry yet, and we use Pedantic but not MyPy yet.)
- tlocke 5y ago
  I think you mean Flake8 rather than Sense8? Anyway, gives me a chance to plug Flake8 Alphabetize, a Flake8 plugin for import ordering https://github.com/tlocke/flake8-alphabetize https://github.com/tlocke/flake8-alphabetize
  - claytonjy 5y ago
    How does this compare/contrast with isort?
    - tlocke 5y ago
      Flake8 Alphabetize will just give you warnings about import order, whereas isort will actually change the code itself. So as I see it there are two types of tool, formatters and checkers. A formatter doesn't alter the Abstract Syntax Tree (AST). In other words it doesn't alter the meaning of the code, just how it looks (eg. the formatter Black). A checker on the other hand looks only at the AST and just gives warnings in your editor, and then it's up to you to change it or not. The import order is fixed in the AST, so it falls outside the scope of Black (which never changes the AST). So I felt there was a need for a tool that worked as a checker that just gives warnings in your editor if your imports don't conform to PEP8, and hence Flake8 Alphabetize. It's worth mentioning that Flake8 Alphabetize follows Black's philosophy of having only one way of doing things, so a project can standardise on Flake8 Alphabetize and everyone's imports will look the same. Anyway, any feedback is welcomed:-)

yvan 5y ago

In the article the author mention that VS Code doesn't support Poetry venv. This can be solve by configuring Poetry to create the virtualenvironment directly in the project folder (which in my opinion is neater). Here is the config https://python-poetry.org/docs/configuration/#virtualenvsin-project https://python-poetry.org/docs/configuration/#virtualenvsin-...

anewhnaccount2 5y ago
You can also just create .venv as a symlink.

stult 5y ago

I use this exact set up and it’s great. Only thing I would add is using cookiecutter, which makes adding all these bells and whistles extremely easy.

wwwhizz 5y ago

"best practices" `curl https://pyenv.run https://pyenv.run | bash` hmmm...

maccard 5y ago
What's the problem here? The script is served over https, so it's not going to be tanpwred with (unless you have a malicious cert, but at that point you can't trust anyone), and curl | bash isn't any worse than downloading a script and just running it, or running a precompiled binary you don't trust.
- Cthulhu_ 5y ago
  The request itself won't be tampered with, but what if the host was? That endpoint could be compromised and send you a different script. They should offer a download with signature validation instead. Signed by Apple, Microsoft, etc if possible.
  - maccard 5y ago
    If you're afraid the host may be untrusted then you would be wrong to download any of their code at all. The safety is in reviewing the code there, not in avoiding curl | bash. Running pip install or npm install is just as dangerous. > They should offer a download with signature validation instead. Signed by Apple, Microsoft, etc if possible. If the host is compromised, the attacker will just get Microsoft to sign their malware instead; see [0]. If the host is compromised, and you run the code without reviwing it, you're hosed regardless. [0] https://arstechnica.com/gadgets/2021/06/microsoft-digitally-signs-malicious-rootkit-driver/ https://arstechnica.com/gadgets/2021/06/microsoft-digitally-...
  - dragonwriter 5y ago
    > The request itself won't be tampered with, but what if the host was? What if your distro package repository was?
- doix 5y ago
  pyenv could get taken over and you won't know. It's also possible to detect when someone is piping to bash (on the server) and serve a different payload [0]. You're better off piping curl to a file, reviewing the file and then running it manually. [0]: https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-bash-server-side/ https://www.idontplaydarts.com/2016/04/detecting-curl-pipe-b...

Tipewryter 5y ago

I prefer containers over pyenv and poetry. This way not only python version and dependencies are "in one place" but also all other stuff that comes along with a new project. The OS, the database etc. The one thing I dislike about Python projects is that Python plasters the compile cache files all over the place. Is there a reason to change that? Currently I use the -B flag for all my scripts. But that makes it slow. I wish Python would have an option to perform like PHP and keep cached compilations in memory instead on disk. Or at least somewhere in /tmp/.

willis936 5y ago
If you need a specific OS then isn't a VM the solution rather than a container? Also at what point do people just realize that all of this overhead is a gigantic waste of time and just use a better language?
- skohan 5y ago
  Yeah to be honest if you need containers to make a project reproducible this is just a sign of failure. You're basically saying you need to encapsulate the entire system for your code to run correctly.
  - cpfohl 5y ago
    Loads of tools have external dependencies that are hard dependencies. And library search paths is my machine vary from those on prod... I could go on, but I'm not sure I understand why using a container to manage all that is a failure?
    - skohan 5y ago
      There are plenty of reasons why you might want to containerize a project. If you have a lot of system dependencies, for example, you might want to consider including a Dockerfile in your project to make it portable. However what makes Python a failure is that people feel they need this to dependably run a python program which only has pure-python dependencies. Compare this to a language like Rust, or the NPM ecosystem. In those cases, the tools have managed to dependably encapsulate projects such that you only need the package manager to make a project fully repeatable. With either of those ecosystems, there's basically one system dependency, and you can find any repository online and dependably do `git clone ...` then `cargo build` etc. to make it work. With Python, you effectively have to reproduce the original developer's system, and that is a failure.

santiagobasulto 5y ago

This is pretty much the same thing that I use. Great write up. If you use ohmyzsh, there are plugins for poetry and pyenv too.

kuu 5y ago

I'm a bit surprised that conda is not mentioned anywhere (I'm not saying as best practice, but as an alternative or why envs are better)

karxxm 5y ago
Yup, especially when you are working with os-level dependencies like cuda, conda is the way to go. I think every tensorflow user can approve
- codethief 5y ago
  …unless you're working on a project for arm64-based IoT devices (like Nvidia Jetson), where Conda is not available.
  - hantusk 5y ago
    Does it not work with miniforge/mambaforge? (https://github.com/conda-forge/miniforge https://github.com/conda-forge/miniforge) I use this with my m1 mac, and it works great.
  - wdroz 5y ago
    conda is now available[1] for arm64 (since March[2]). [1] -- https://docs.conda.io/en/latest/miniconda.html#linux-installers https://docs.conda.io/en/latest/miniconda.html#linux-install... [2] -- https://github.com/conda/conda/issues/8297 https://github.com/conda/conda/issues/8297
    - codethief 5y ago
      Awww man. I must have missed that announcement by only a few days.
- dijksterhuis 5y ago
  Quite confused about this comment as many of the latest tensorflow V2 releases aren't reliably uploaded to conda forge. IIRC there's only like 4 or so of the V2 releases uploaded. You can conda install the CUDA dependencies and then install the required Tensorflow version via conda pip. But that's not much different to installing CUDA manually and then installing tf from system pip. It's much faster and easier to pull a tf Docker image as it's their "officially supported" way to get up and running. So... as a tf user and a sysadmin... Nah. No conda for me thanks.

Donckele 5y ago

Please remove “Best Practices” and year from the title. Completely irrelevant and misleading.

codethief 5y ago
Why? The year in particular seems very relevant because the "recommended Python project setup" (if there ever was such a thing) changes every other year or so.
- Donckele 5y ago
  Well at least next year it will be officially out of date.
plesiv 5y ago
You should expect that "it's my opinion" is implied for these kind of blog posts.
- Donckele 5y ago
  Maybe we should have a prefix “My Opinion” for those easily confused.

nxpnsv 5y ago

Very close to my setup actually, there is a lot of good tips in this article.

Spivak 5y ago
Agreed, this is a pretty solid setup that will get you 90% of the way there for any Python project.

kissgyorgy 5y ago

For VSCode to recognize the virtualenvs created by Poetry, you just have to add a setting for the Python extension: "python.venvFolders": [ "~/.cache/pypoetry/virtualenvs", "~/.pyenv/versions", ], This will make VSCode automatically recognize virtualenv interpreters, so you can select them without starting code from an activated shell.

petulla 5y ago

why poetry over pyenv-virtualenv?

EamonnMR 5y ago
What poetry replaces is pip and requirements.txt. Our team had a lively discussion about this, but here are some good reasons: * Keep test frameworks out of production (in case they've busted something; the two points below can happen to test libraries too.) * pip recently changed the way it's resolver works[1], breaking numerous projects. Yeah yeah, it's a major version bump, but lots of containers and environments installed pip latest just sort of assuming that it would always handle requirements.txt the same way. With that contract broken, now using pip directly isn't a non-decision anymore. * Locking specific versions of dependencies can you roll back Bad News in production. Let's say you have a project with library A which has dependency B. If A asks for the latest version of B, you can be in a situation where a new version of B breaks your project and you won't know about it until you do a release _and_ if you try to roll back you'll still be in trouble. We recently had to deal with a similar issue and had to fall back on an older container until we could figure it out. [1] https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html?m=1 https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.h...
- mvanbaak 5y ago
  pip is not to blame for the above points. * if your code is under src/ and your tests under test/, you use a requirements-test (or better, tox.ini) your tests including the dependencies like pytest dont end up in a wheel * if a container depends on 'latest' and not some semver major number, it's 100% the containers fault when they blindly update to a new major version * 'pip freeze' is your friend
  - EamonnMR 5y ago
    Agreed, Pip isn't to blame, treating `pip install -r requirements.txt` as one's sole dependency management is. You can choose to build your own dependency management practice around pip, or use one someone else has already created. I think that it is easier to get a team on the same page with something like Poetry, especially if they're used to bundler or npm. You're right about the docker containers as well, but upstream does what it wants and downstream has a strong tendency to not mess with upstream's choices.

EamonnMR 5y ago

One thing I'd add herr here is Pydantic. It compliments the type hinting very nicely and lets you get more static with your classes. But that you don't just install, you need to use. Which is another omission, I suppose; python's type hints are opt in, so mypy won't do you any good unless you actually add type hints, right?

daviddavis 5y ago
There are a bunch of mypy configuration options to disallow certain things like --disallow-untyped-defs. There’s also --strict which forces typing on everything.
- kortex 5y ago
  Man, developing python with pydantic and "mypy --strict" (I follow pydantic's config [0] where I can) is such Type 2 Fun. It feels like a totally different language. Yeah it takes a little more time at first but then type inference and autocomplete starts to kick in and then you're screaming fast. And you "compile" it and everything just works. No hunting down edge cases or tracebacks cause you forgot to catch a None. I find it super satisfying. Much easier to stay in flow state when you aren't having to stop every few minutes to test stuff and dig through tracebacks. [0] https://github.com/samuelcolvin/pydantic/blob/master/setup.cfg#L48 https://github.com/samuelcolvin/pydantic/blob/master/setup.c...

dpatterbee 5y ago

> some Linux distributions already switched to version 3 At the lightning pace of only 12 years since the release of python 3 (and 15 months since python 2's deprecation), some linux distribution have ALREADY switched to it.

skohan 5y ago
The developer story around python 2/3 is emblematic of why I don't consider it a suitable language for software development outside of very controlled cases like notebooks.
- leetrout 5y ago
  Eh. That’s all passed. Now we’re arguing about pip vs pipenv / pip file or poetry.
  - kissgyorgy 5y ago
    No we are not :) poetry won, it's just way better than pipenv. In fact, pipenv was just always shit.
    - leetrout 5y ago
      Have you seen PDM? https://pdm.fming.dev/ https://pdm.fming.dev/
- smcl 5y ago
  I dunno, they made some big breaking changes but continued supporting 2.x for over a decade after 3.x. There are definitely reasons I would not consider Python for some projects, but the 2/3 transition is not one of them. To be honest the event is so infamous (and IMO a little overblown) that they basically couldn't do this anymore, and I think it's been openly stated that they wouldn't.
  - skohan 5y ago
    What gets me about it is that the 2/3 transition was just a massive unsolved problem, and one which caused pain over a decade after 3.x was released. Other ecosystems managed to solve it - like Rust for example handles editions in a very sound way - but with Python you had so many problems cropping up over the years over python versions.

rwmark 5y ago

I wonder why VSCode is at least name-dropped in these Python setup tutorials lately. VSCode is spying on you and sending back "telemetry" data. Given the current situation in Python, where there is little development, the old boys have totalitarian control, and new contributors are smart enough to avoid that mess: Try out Rust, Go, Elixir or Lua instead. It might save you a lot of trouble. Heck, if you are willing to put in a lot of time to create carefully written objects, C++11 code can look a lot like Python (if you are into that).

hetspookjee 5y ago
My experience is also that PyCharm is much more pleasureable to work with and on a safer side. I checked 1.5 year ago, so could be changed, for remote development on a server and had to download some 3rd party Rsync tools to be able to do so. For PyCharm is just works out of the box with a professional license.
- rwmark 5y ago
  Indeed, anything but VSCode: "What data is collected. The GitHub Copilot collects activity from the user's Visual Studio Code editor, tied to a timestamp, and metadata." PyCharm, Emacs, Vim, are all better and don't spy.
  - excitednumber 5y ago
    I came here to read this.
  - btdmaster 5y ago
    PyCharm has similarly bad privacy: https://www.jetbrains.com/legal/docs/privacy/privacy.html https://www.jetbrains.com/legal/docs/privacy/privacy.html. I'm really happy though that I learned to use Emacs, really useful and language independent.
- 0x008 5y ago
  VS Code experience has definitely improved a LOT in the past 1.5 years with the new python language server (pylance) and remote development being improved quite a bot. We use it daily in production in my company with almost zero problems so far.

hazbot 5y ago

Every time I see one of these threads I think "maybe Python isn't the best choice if we still need threads like this". But it's just so useful...

thundergolfer 5y ago

The lack of discussion of lock files is a big omission in any 'best practices' page. "Lock" only appears in a log output of Poetry. Even if the answer is "Poetry handles it" you certainly want to explain why they're important just like is being done in the rest of the "Why use..." sections.

nerdponx 5y ago
For a generic lockfile, use the pip-compile tool from https://pypi.org/project/pip-tools https://pypi.org/project/pip-tools
- beaugunderson 5y ago
  Huge +1 for this; pip-compile is so much faster and less surprising than poetry. Poetry's #1 footgun is that it updates the version of unrelated dependencies when you add a new dependency. No thanks.
- pmkiwi 5y ago
  Yeap I am using pip-compile too because I always managed to mess up my environment when using poetry!
0x008 5y ago
Yes! Lockfiles are basically the only reason to use poetry over pip+venv or am I missing something?

cutler 5y ago

Why is all this unnecessary with Ruby considering the languages are so similar?

e12e 5y ago
Much similar work is required for ruby, but gem/bundler is less fraught with fragmentation than pip/poetry/venv.
- wink 5y ago
  I think a big part of the problem is that bundler has been around for ages whereas half of the tools mentioned here didn't even exist just 5-6 years ago. Which can be seen by this hilarious sentence: > By default, Python packages are installed with pip install. In reality nobody uses it this way. It installs all your dependencies into one version of Python interpreter which messes up dependencies. Why hilarious? Because everyone I know uses `pip install` if they're not using poetry. And because poetry is kinda new, that's a lot of people.
  - dmurray 5y ago
    The quote isn't clear, but I think the author means people don't generally run pip with their system python. I.e. they don't run /usr/bin/pip or whatever, but use some virtual environment solution.
    - e12e 5y ago
      Which is the same for "gem install" - you don't want to go "sudo gem install".
      - dmurray 5y ago
        gem catches this by default and warns you. Pip doesn't, but it would probably be good if it did and the pip guys explicitly took the position that "installing stuff on your system python is not a normal use of it".

mikesabbagh 5y ago

I am surprised that containers are not mentioned here. We do similar but use containers instead of env.

amrox 5y ago
Containers are great for web services or anything that primarily communicates through a port. Less so for CLI style projects that read and write to the local file system. Yes I know you can bind mount directories but it’s clunky and you also usually have to fight with file permissions issues.

deleted 5y ago

[deleted]

gillesjacobs 5y ago

Great recommendations! Exactly my setup, except add pipx for global system tools, i.e. command line tools distributed through pypy (youtube-dl, black, etc.). I also add Docker or Songularity containers as needed for deep learning deployment but that is quite computing platform and application specific.

blooalien 5y ago
Love pipx. Great tool. I personally use that and poetry both quite heavily.
bobuk 5y ago
probably you've meant pypi not pypy
- gillesjacobs 5y ago
  Yes, indeed I mean the Python Package Index (PyPI). Mypy, pypy, pypi, pyenv, pipenv, pip, pipx. Easily confusable names are a core tenet of the Python ecosystem!

jtdev 5y ago

venv + requirements.txt is simple and works. Why are people so obsessed with replacing this simple toolset/process with poorly maintained tools like Poetry/PyEnv?

philote 5y ago
Why do you think those tools are poorly maintained? How do you handle locking in specific versions of libraries and keeping those properly maintained?
- nomdep 5y ago
  With pip-tools. https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools And you can still use standard setup.cuff and pip install -e unlike Poetry. Also, much faster.
- jtdev 5y ago
  990 open issues… bugs going back to 2018 still open… https://github.com/python-poetry/poetry/issues https://github.com/python-poetry/poetry/issues Please help me understand why I would want to lock specific versions of libraries in my Python projects. I’ve worked extensively with Python for a number of years and have never needed a lock file for my dependencies.

claytonjy 5y ago

Why use the external pre-commit repos instead of pointing to the poetry-managed local installations for flake8, black, isort, etc? The repos approach results in a much simpler and cleaner pre-commit config, but since you end up with two pinned versions of each, managed by different updaters, it can invite drift which can be very hard to track down. I'd prefer to only use the pre-commit versions of these libs, but then I'd sacrifice editor integration.

super_mario 5y ago

Why use pyenv when venv is built into Python 3? Also, pip installs your dependencies into active venv, so I don't see a point in using anything else.

blooalien 5y ago
Because pyenv [1] and venv [2] are entirely different things that do two completely different jobs? [1]: https://github.com/pyenv/pyenv https://github.com/pyenv/pyenv [2]: https://docs.python.org/3/library/venv.html https://docs.python.org/3/library/venv.html

edumucelli 5y ago

That is a great write-up! One extra bit I'd recommend to this list is using https://github.com/ikamensh/flynt https://github.com/ikamensh/flynt to convert string format into f-strings. It requires Python 3.6.

WillDaSilva 5y ago

Great recommendations all around. It'd be nice if they went into how they handle dealing with C extensions, as that adds a large amount of complexity to a Python project, and most of the published advice on the topic is quite old. EDIT: Mentioning `pyproject.toml` and the relevant PEPs would also be great (i.e. PEP 621, PEP 517, etc.). Fortunately Poetry is compliant with these PEPs.

gww 5y ago
I really struggle with this part of Python package development. I have a rather complex C++ library I have been calling with Cython but I am struggling to find the best practices for either compiling the library with setup tools or calling cmake and copying the library into the package etc.
- thundergolfer 5y ago
  Look into Bazel. Bazel particularly suited to building Python+{C/C++/Rust} codebases. It's still early days in the Bazel ecosystem, so you won't find a ready-made solution, but the fundamentals are solid.
  - gww 5y ago
    Oh cool thanks, I will check it out!
paulross 5y ago
This might help: https://pythonextensionpatterns.readthedocs.io/en/latest/index.html https://pythonextensionpatterns.readthedocs.io/en/latest/ind...
- WillDaSilva 5y ago
  That's a great resource in general for Python extensions, but doesn't have much to say about packaging and distributing them. For that the best resources I've found have been to look at what large/complex projects that use them do, as they've often had to deal with many of the odd cases that often come up. Using Cython adds another layer of complexity to the packaging/distribution that that link does not address. Fortunately now that you can specify build requirements in `pyproject.toml` Cython has become significantly easier to use on that front, but there are still some less than obvious bits to say the least. Maybe I should publish an overview of Python best practices for publishing C extensions (with or without Cython).

daniellarusso 5y ago

Is there a node/react/js equivalent of this article?

DrammBA 5y ago
I'm also interested in this, but I'm afraid the answer will be "CRA/Next/Nest or deploy your own webpack/babel".

pizza 5y ago

Awesome article but damn if that isn't some cognitive overhead just to get started (for newbies, at least)

rbanffy 5y ago

The pain and suffering I see with different versions of Python installed with Pyenv in my team and surrounding ones is huge. I advise anyone to just stay away from it. On Macs, you can install MacPorts and it'll bring in every version of Python, from ancient to newest, in a sane BSD-like way (not in a user-writable folder, ffs). On Linux the default package managers offer pretty much every version of Python that's not outright dangerous to run. On Windows there are binary installers and every Python goes in a separate folder. Not only it's not needed, it creates a layer of "magic" the user has to understand on top of the environment. And, if you really want to use brew, you can still continue using it, knowing it only has python 3.9 at the moment. Plus, it can coexist mostly peacefully with Macports.

mdaniel 5y ago
> knowing it only has python 3.9 at the moment. Is incorrect, they just move older versions to their "@" nomenclature, for example https://formulae.brew.sh/formula/python@3.7 https://formulae.brew.sh/formula/python@3.7
- rbanffy 5y ago
  Interesting. Thank you for correcting me. Can you install multiple versions this way? My colleagues are suffering with that and I'd prefer not to move them from Homebrew if avoidable.
  - mdaniel 5y ago
    For sure, yes, they're distinctly named formulae and thus subject to their own install, remove, and version bump lifecycle management IIRC the trick is that the "at" versions don't get "brew link"-ed by default, since they'd almost certainly smash on top of the non-at binaries or manpages or whatever. Using `PATH=$(brew --prefix "python@3.7")/bin:$PATH` or ones favorite context switching gizmo will help, or (with the python ones specifically) using `$(brew --prefix "python@3.7")/bin/python -m venv ...` is a great way to avoid having a lot of special env-var silliness

therealmrmatt 5y ago

If anyone is interested I have a GitHub template that is essentially a simpler of the scaffolding the author lays out (https://github.com/mplemay/python-template https://github.com/mplemay/python-template).

foobarbecue 5y ago

I use conda environments and have found they do everything described here. Is there a reason I should try pyenv & poetry?

jonnycomputer 5y ago

pyenv + poetry ?? I thought you just needed poetry? I've been using conda, but have begun contemplating a switch to poetry. So what am I missing?

peterthehacker 5y ago

This title is misleading to me. It’s not about “best practices” but instead about “best tools”.

deleted 5y ago

[deleted]

aunty_helen 5y ago

There should be an article like this for every language, as the author mentions it's not just syntax to change a language. Much of it is experience based knowledge. However, this is how you would setup for a new project only and in a piecemeal manner, if you didn't already have some existing template. Compatibility issues with your main dev tool would rule out using Poetry, in the same way running the latest versions of any software without reason is a rookie mistake. Chasing a higher version number is a jr / intermediate folley. If you're developing python professionally, save your money and just pay for pycharm. The morning of fluffing with a dozen new tools which then have their own maintanence overheads is less cost effective than buying a product that gives you >70% of what the author has recommended and it does so in a generally consistent manner. This blog post was a great reminder of how much pycharm gives me on a day to day basis, how people get lost in their +1 more tool mindset and how switching languages is an intial cognitive overload.

l0b0 5y ago

The current work project[1] has all of these, with one substitution: Pyenv, Poetry, Pytest, pytest-cov with 100% branch coverage, pre-commit, Pylint rather than Flake8, Black, mypy (with a stricter configuration than recommended here), and finally isort. These are all super helpful. There's also a simpler template repo[2] with almost all of these. [1] https://github.com/linz/geostore/ https://github.com/linz/geostore/ [2] https://github.com/linz/template-python-hello-world https://github.com/linz/template-python-hello-world

0x008 5y ago

What is almost never mentioned is that the default virtual env will always set the python binary it is called from as the python interpreter for that environment. So if I want to use Python3.9 i go python3.9 -m venv .venv, if I want to use python3.6 i go python3.6 -m venv .venv etc. Yes, lockfiles are great, but while everyone can execute the above commands, it is a lot harder to convince everyone in my organization to switch to poetry.

bluewalt 5y ago

> Some people work with code from 13" laptops. And viewing diff of two files of 79 char per line max becomes very convenient. I think it's not the only reason. Limiting columns forces developers to read more vertically, which is less tiring for eyes. For example, try to read a book on a 27' monitor with no max width. Going to the next line will quickly become painful.

bluewalt 5y ago

I replaced all the pyenv (and any version management in my projects) by the awesome VS Code remote development with containers. This is a game changer to me. 1 container = 1 clean installation, completely isolated, and you can share with your teammates. Then I disable the use of virtual environments in Poetry, because it's a useless overhead. Then for updates, I just change versions in my Dockerfile or .toml and rebuild the container from scratch in a few time, which is cleaner than manual updates for everything IMO.