5 ms·

At my last job, my (Data Science) team all deployed production code including our models. We were using Scala, which made it fairly easy since we could essentia

At my last job, my (Data Science) team all deployed production code including our models. We were using Scala, which made it fairly easy since we could essentially just build fat JARs and upload them. In contrast, I've found trying to do the same thing with Python nightmarish - simple things like having Pandas .20 on one machine and .21 on another will result in silent changes in how some metrics are calculated. Anaconda mostly, but not entirely, solves that problem with its superior environment management (significantly better than pip IME.) But given that it's imperfect, I'll probably look more into Docker soon.

iamwil 9y ago
How did you manage the dependency versioning problem? Did you set up automated tests for that? Or did you just make sure everyone installed the same version with a requirements.txt file?
- SatvikBeri 9y ago
  Anaconda has a better tool for cloning identical environments as long as both machines are on the same OS: https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments https://conda.io/docs/user-guide/tasks/manage-environments.h... In theory, pip lets you do this with pip freeze, but I found that pip often broke on some finicky libraries, while Anaconda failed much less frequently (though still has occasional issues.) The main drawback is the same OS requirement – this makes it harder to e.g. develop on a mac and then launch a pipeline on an Ubuntu machine in AWS.
  - pwang 9y ago
    What do you mean by "same OS requirement"? Can't you take your environment.yml from a Mac and recreate the environment on Linux?
    - SatvikBeri 9y ago
      Anaconda has two environment-copying tools, one is environment.yml (which is analogous to pip's requirements.txt), and the other is the "explicit" spec file, which actually points to specific files for specific packages. I've found environment.yml to fail with finicky libraries, same as requirements.txt, while the explicit spec file almost always works.

iamwil 9y ago

How did you manage the dependency versioning problem? Did you set up automated tests for that? Or did you just make sure everyone installed the same version with a requirements.txt file?

SatvikBeri 9y ago
Anaconda has a better tool for cloning identical environments as long as both machines are on the same OS: https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments https://conda.io/docs/user-guide/tasks/manage-environments.h... In theory, pip lets you do this with pip freeze, but I found that pip often broke on some finicky libraries, while Anaconda failed much less frequently (though still has occasional issues.) The main drawback is the same OS requirement – this makes it harder to e.g. develop on a mac and then launch a pipeline on an Ubuntu machine in AWS.
- pwang 9y ago
  What do you mean by "same OS requirement"? Can't you take your environment.yml from a Mac and recreate the environment on Linux?
  - SatvikBeri 9y ago
    Anaconda has two environment-copying tools, one is environment.yml (which is analogous to pip's requirements.txt), and the other is the "explicit" spec file, which actually points to specific files for specific packages. I've found environment.yml to fail with finicky libraries, same as requirements.txt, while the explicit spec file almost always works.