4 ms·

As someone dabbling in ML for the first time, setting up the correct Python env has been a nightmare just to run some sample code. I am familiar with Docker and

As someone dabbling in ML for the first time, setting up the correct Python env has been a nightmare just to run some sample code. I am familiar with Docker and would love if this got more mainstream adoption. Fighting with dependencies sucks when you aren’t intimately familiar with the ecosystem: python, node and other js, and even go.

smnrchrds 9y ago
I haven't worked in ML, but I've used Python extensively for scientific computing. Setting up the environment used to be a headache, but Anaconda solved that issue. Are ML tools not available in Anaconda?
- robszumski 9y ago
  This is on my list of tools to investigate next. Glad I’m headed in the right direction.
  - xapata 9y ago
    I know you said "dabbling" but it seems like Anaconda is recommended by nearly every guide to ML with Python. What did you stumble across that recommended something else?
  - JPKab 9y ago
    Move it to the top of your list. Anaconda is absolutely where you should start if you are trying to mess with ML/Predictive.
- jimwalsh 9y ago
  ML tools are available in Anaconda and is where we direct our data scientists until they are comfortable customizing further.
mindhash 9y ago
Does anyone see value in tool like michaelangelo from uber? What if a data scientist could deploy models right from browser?
- JPKab 9y ago
  Check out Yhat. They were a Ycombinator alumni, and they offer that service. They were purchased by an enterprise software company called Alteryx, due entirely to the technical excellence of their product.
pletnes 9y ago
What you need is the conda package manager (from anaconda python), in my opinion. Docker is very powerful, but is useful for other purposes (e.g deploying code) than simply grabbing a bunch of python modules and running analyses in a notebook.
- thenaturalist 9y ago
  I'm asking as someone who is mid level Python & DS experienced and out of real curiosity (not a "mine is better than yours" thought): Could you elaborate what in your opinion makes anaconda superior? I am frequently doing data munging & low level ML things and - so far - I am more happy with the pip side of things as a combination of virtualenv, autoenv [1], pip tools [2], pyup [3] and the rest of the eco system. Standard procedure, takes two minutes and I have a Jupyter notebook up, smartly automated base processes for dependency management, full control of environment variables and can deploy/ integrate this in any other Python setup (if I were to port IPython code to pure Python). Is this more a thing of each his own or am I missing a crucial advantage of Anaconda? [1] https://github.com/kennethreitz/autoenv https://github.com/kennethreitz/autoenv [2] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools [3] https://github.com/pyupio/pyup https://github.com/pyupio/pyup
  - joshvm 9y ago
    Ease of use. I run outreach events and when you have sporadic internet access, being able to give new programmers an exe to install Python with a load of useful packages is a godsend. Also, it takes a lot of the pain out of setting up a Python environment on Windows when you need funny packages that require compilation (e.g OpenCV). I've been to enough software carpentry talks where people nod and smile, and then promptly ignore the "how to be a responsible developer" advice. They don't care. They just want to run Python and plot some stuff or classify some data without worrying about multiple CLI tools. People seem to accept Anaconda because it comes across as a monolith and it's easier to understand for new programmers than virtualenvs and worrying about pinning pip installs. That said, I don't use Anaconda on Mac or Linux.
- JPKab 9y ago
abhirag 9y ago
Have a look at pipenv - [https://docs.pipenv.org/ https://docs.pipenv.org/], it provides deterministic builds and sane package management for python and is the officially recommended Python packaging tool now. If dependency management is your major problem, Docker might be an overkill. P.S. -- Even if you are on Windows, whole SciPy ecosystem can be easily installed with pipenv, no need of conda for that now.
- j2kun 9y ago
  This looks pretty, but I've never personally had trouble with `pip freeze > requirements.txt` What benefits does pipenv have beyond storing the hashes as well? In what way is it more sane? For example, does it get around the issues with, e.g., matplotlib being installed in a virtualenv?
  - abhirag 9y ago
    I assume you are referring to the issues related to installation of external GUI frameworks in virtual environments mentioned here -- [https://matplotlib.org/faq/virtualenv_faq.html https://matplotlib.org/faq/virtualenv_faq.html], I am afraid pipenv doesn't solve that any better than virtualenv’s --system-site-packages option -- [https://docs.pipenv.org/advanced/#working-with-platform-provided-python-components https://docs.pipenv.org/advanced/#working-with-platform-prov...]. If you are happy with your pip and venv workflow you can happily ignore pipenv, I personally prefer it because: - It combines all the features of pip and venv with a better UI(in my opinion). - I can specify my environments in more detail using Pipfile [https://docs.pipenv.org/advanced/#specifying-basically-anything https://docs.pipenv.org/advanced/#specifying-basically-anyth...] - It is integrated with pyenv [https://docs.pipenv.org/advanced/#automatic-python-installation https://docs.pipenv.org/advanced/#automatic-python-installat...] - The promise of deterministic builds
- pwang 9y ago
  Pipenv is only as good as the wheels that get installed. And the wheels are only as good as the assumption that your runtime/deployment environment matches the buildtime environment of the person who made those wheels. Oftentimes with ML and compute-intensive workloads, performance is an aspect of correctness. It is nontrivial to get the right accelerated libraries, and the right GPU libraries linking against the right Numpy version for the particular notebook you're running. If you are doing work just for yourself, or if you have total control over the deployment environment (and your collaborator's environments), then you might be able to get away with just using pip for these things.
drraid0 9y ago
How? what is there besides running pip?
- iamwil 9y ago
  It can be a pain if you're running in a windows environment, even with pip.

smnrchrds 9y ago

I haven't worked in ML, but I've used Python extensively for scientific computing. Setting up the environment used to be a headache, but Anaconda solved that issue. Are ML tools not available in Anaconda?

robszumski 9y ago
This is on my list of tools to investigate next. Glad I’m headed in the right direction.
- xapata 9y ago
  I know you said "dabbling" but it seems like Anaconda is recommended by nearly every guide to ML with Python. What did you stumble across that recommended something else?
- JPKab 9y ago
  Move it to the top of your list. Anaconda is absolutely where you should start if you are trying to mess with ML/Predictive.
jimwalsh 9y ago
ML tools are available in Anaconda and is where we direct our data scientists until they are comfortable customizing further.

mindhash 9y ago

Does anyone see value in tool like michaelangelo from uber? What if a data scientist could deploy models right from browser?

JPKab 9y ago
Check out Yhat. They were a Ycombinator alumni, and they offer that service. They were purchased by an enterprise software company called Alteryx, due entirely to the technical excellence of their product.

pletnes 9y ago

What you need is the conda package manager (from anaconda python), in my opinion. Docker is very powerful, but is useful for other purposes (e.g deploying code) than simply grabbing a bunch of python modules and running analyses in a notebook.

thenaturalist 9y ago
I'm asking as someone who is mid level Python & DS experienced and out of real curiosity (not a "mine is better than yours" thought): Could you elaborate what in your opinion makes anaconda superior? I am frequently doing data munging & low level ML things and - so far - I am more happy with the pip side of things as a combination of virtualenv, autoenv [1], pip tools [2], pyup [3] and the rest of the eco system. Standard procedure, takes two minutes and I have a Jupyter notebook up, smartly automated base processes for dependency management, full control of environment variables and can deploy/ integrate this in any other Python setup (if I were to port IPython code to pure Python). Is this more a thing of each his own or am I missing a crucial advantage of Anaconda? [1] https://github.com/kennethreitz/autoenv https://github.com/kennethreitz/autoenv [2] https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools [3] https://github.com/pyupio/pyup https://github.com/pyupio/pyup
- joshvm 9y ago
  Ease of use. I run outreach events and when you have sporadic internet access, being able to give new programmers an exe to install Python with a load of useful packages is a godsend. Also, it takes a lot of the pain out of setting up a Python environment on Windows when you need funny packages that require compilation (e.g OpenCV). I've been to enough software carpentry talks where people nod and smile, and then promptly ignore the "how to be a responsible developer" advice. They don't care. They just want to run Python and plot some stuff or classify some data without worrying about multiple CLI tools. People seem to accept Anaconda because it comes across as a monolith and it's easier to understand for new programmers than virtualenvs and worrying about pinning pip installs. That said, I don't use Anaconda on Mac or Linux.
JPKab 9y ago

abhirag 9y ago

Have a look at pipenv - [https://docs.pipenv.org/ https://docs.pipenv.org/], it provides deterministic builds and sane package management for python and is the officially recommended Python packaging tool now. If dependency management is your major problem, Docker might be an overkill. P.S. -- Even if you are on Windows, whole SciPy ecosystem can be easily installed with pipenv, no need of conda for that now.

j2kun 9y ago
This looks pretty, but I've never personally had trouble with `pip freeze > requirements.txt` What benefits does pipenv have beyond storing the hashes as well? In what way is it more sane? For example, does it get around the issues with, e.g., matplotlib being installed in a virtualenv?
- abhirag 9y ago
  I assume you are referring to the issues related to installation of external GUI frameworks in virtual environments mentioned here -- [https://matplotlib.org/faq/virtualenv_faq.html https://matplotlib.org/faq/virtualenv_faq.html], I am afraid pipenv doesn't solve that any better than virtualenv’s --system-site-packages option -- [https://docs.pipenv.org/advanced/#working-with-platform-provided-python-components https://docs.pipenv.org/advanced/#working-with-platform-prov...]. If you are happy with your pip and venv workflow you can happily ignore pipenv, I personally prefer it because: - It combines all the features of pip and venv with a better UI(in my opinion). - I can specify my environments in more detail using Pipfile [https://docs.pipenv.org/advanced/#specifying-basically-anything https://docs.pipenv.org/advanced/#specifying-basically-anyth...] - It is integrated with pyenv [https://docs.pipenv.org/advanced/#automatic-python-installation https://docs.pipenv.org/advanced/#automatic-python-installat...] - The promise of deterministic builds
pwang 9y ago
Pipenv is only as good as the wheels that get installed. And the wheels are only as good as the assumption that your runtime/deployment environment matches the buildtime environment of the person who made those wheels. Oftentimes with ML and compute-intensive workloads, performance is an aspect of correctness. It is nontrivial to get the right accelerated libraries, and the right GPU libraries linking against the right Numpy version for the particular notebook you're running. If you are doing work just for yourself, or if you have total control over the deployment environment (and your collaborator's environments), then you might be able to get away with just using pip for these things.

drraid0 9y ago

How? what is there besides running pip?

iamwil 9y ago
It can be a pain if you're running in a windows environment, even with pip.