4 ms·
>figure out how internal ML library A and dependent CLI tools package B can express their dependencies in ways that they can be installed as regular pip package
by monkeybutton 5y ago
>figure out how internal ML library A and dependent CLI tools package B can express their dependencies in ways that they can be installed as regular pip packages from where they live in our private GitHub repos.
Does this mean that you are keeping forks of the projects in your repo?
What I've been using but is probably not best practice is:
- A private PyPi instance (Nexus) for caching external packages
- All our projects, services, everything live in a monorepo
- All our code is developed as packages with a setup.py that lists the minimal required dependencies for each project
- "builds" of services are docker images that copy the necessary sources and do a `pip install -e <package for our service>`
The downside of this is if requirements are changed, updating your personal development environment means trashing and remaking the environment (relatively quick) or rerunning `pip install -e` for the given project.
I would only recommend using conda if you are developing on Windows. Even today there's still packages missing binary wheels and compiling other people's code on Windows to build/install a package sucks.
At another company I've seen:
- Each service and internally developed package had its own git repository
- Projects used dependency links[1] to get the packages from the git
I didn't like this approach because it split up tasks and created a lot of PRs. In the first setup with a monorepo you could see all the changes in one PR.
[1]https://stackoverflow.com/questions/36544700/how-to-pip-install-a-package-that-has-git-dependencies https://stackoverflow.com/questions/36544700/how-to-pip-inst...