5 ms·

Not only upgrading is hard, but also installation (on Windows at least). For each Tensorflow version you need a specific python version, a specific CUDA version

by XCSme 7y ago

Not only upgrading is hard, but also installation (on Windows at least). For each Tensorflow version you need a specific python version, a specific CUDA version, specific tensorflow-gpu version, and many other easy to get wrong things. The problem is not the requirements, but that it's very hard to know what versions are compatible. There are endless threads on Github of people trying to use Tensorflow but failing after spending days trying to install it.

nicklarsennz 7y ago
I've been using containers for this, and mounting the source code.
- jacquesm 7y ago
  I've been using Anaconda and it finally made Python work without endless rounds of 'incompatible libary bingo'. It also works well for machine learning projects.
  - tudelo 7y ago
    +1 for Anaconda, worth trying for anyone who has a problem with package versions.
    - echelon 7y ago
      As a software engineer and non-data scientist, I hate Anaconda because it feels like it's a tool that tries to be the be-all, end-all package management tool for everyone in the data science field, yet it feels like a sloppily built, bloated whale. It's even managed to overwrite PATH on some of my Linux machines, which is where I drew the line. I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead. Sadly, many of the ML models I investigate on Github don't even have their package requirements frozen. It's an uphill battle...
      - jacquesm 7y ago
        > I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead. I suspect you have a lot of time on your hands. But for me the 'batteries included' approach really nails it, why repeat the headache over and over again when a single entity can take care of that in such a way that incompatibilities are almost impossible to create? The hardest time I've had was to re-create an environment that ran some python code from a while ago, with Anaconda it was super easy. I'm sure it has its limitations and just like every other tool there are situations where it is best to avoid it but for now it suits me very well.
      - mstokholm 7y ago
        I would suggest you try out Miniconda (https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html). It comes with just the basics, and let's you install TF with GPU support by simply doing: conda install -c anaconda tensorflow-gpu
  - dekhn 7y ago
    it's incredibly slow.
    - xxxtentachyon 7y ago
      What’s incredibly slow? Installing things with conda?
      - dekhn 7y ago
        conda install for our environment.yml: about 3-5 minutes solving, then 5-10 minutes installing. pip install with almost exactly the same set of packages: 3-5 minutes total.
  - jwilber 7y ago
    Anaconda may work well as a virtual environment for some ml projects, but it is by no means a solution for getting a gpu-working installation of tensorflow on Windows.
    - jacquesm 7y ago
      > getting a gpu-working installation of tensorflow on Windows I think part of your problem is the last word there. Windows is a bad match for such an environment. On Ubuntu it is pretty much painless.
      - jwilber 7y ago
        I agree, and prefer Linux myself, but some clients only allow for solutions based in Windows (and no containers) :/
    - deleted 7y ago
      [deleted]
ghop02 7y ago
They're (slowly) making this better. Starting with 1.15, there is only one tensorflow pip package, no tensorflow/tensorflow-gpu hell any longer.
- n8henrie 7y ago
  Really? https://www.tensorflow.org/install/gpu https://www.tensorflow.org/install/gpu says to `pip install tensorflow-gpu`.
  - ghop02 7y ago
    https://groups.google.com/a/tensorflow.org/forum/#!topic/developers/iRCt5m4qUz0 https://groups.google.com/a/tensorflow.org/forum/#!topic/dev... Ah I guess it's on the way but not fully there yet
mikehollinger 7y ago
Try using these containers[1] from my peer team at IBM. They run on a variety of architectures (x86, ppc64le, with and without GPUs in most cases). In addition if you don't want to fiddle with the containers, there's also a conda channel[2] that lines stuff up. I work on a peer team in machine vision, and use these for personal and professional projects. [1] https://hub.docker.com/r/ibmcom/powerai/ https://hub.docker.com/r/ibmcom/powerai/ [2] https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.6.1/navigation/wmlce_install.htm https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.6.1/...
hadlock 7y ago
Tensorflow sounds like an ideal candidate for running in a container. List out all your approved compatible versions in the Dockerfile and distribute it with your source code and anyone can reproduce your results with the exact same setup.
- XCSme 7y ago
  Yes, but for personal use, unless someone else already made those containers, I will still have to go myself through the trial-and-error process of finding the right combination of versions. Yes, second installation will be easier, but if I just want it on my PC it doesn't really help.
  - hadlock 7y ago
    You can `docker pull username/mytensorflowcontainer` and start from someone elses' work. Looks like Tensorflow has a how-to on the site: https://www.tensorflow.org/install/docker https://www.tensorflow.org/install/docker including working cpu-only and gpu-enabled examples.

by XCSme 7y ago

nicklarsennz 7y ago

I've been using containers for this, and mounting the source code.

jacquesm 7y ago
I've been using Anaconda and it finally made Python work without endless rounds of 'incompatible libary bingo'. It also works well for machine learning projects.
- tudelo 7y ago
  +1 for Anaconda, worth trying for anyone who has a problem with package versions.
  - echelon 7y ago
    As a software engineer and non-data scientist, I hate Anaconda because it feels like it's a tool that tries to be the be-all, end-all package management tool for everyone in the data science field, yet it feels like a sloppily built, bloated whale. It's even managed to overwrite PATH on some of my Linux machines, which is where I drew the line. I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead. Sadly, many of the ML models I investigate on Github don't even have their package requirements frozen. It's an uphill battle...
    - jacquesm 7y ago
      > I vastly prefer creating hermetic environments with either venv or Docker. They're much cleaner and easier to work with. I wish data scientists would adopt these tools instead. I suspect you have a lot of time on your hands. But for me the 'batteries included' approach really nails it, why repeat the headache over and over again when a single entity can take care of that in such a way that incompatibilities are almost impossible to create? The hardest time I've had was to re-create an environment that ran some python code from a while ago, with Anaconda it was super easy. I'm sure it has its limitations and just like every other tool there are situations where it is best to avoid it but for now it suits me very well.
    - mstokholm 7y ago
      I would suggest you try out Miniconda (https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html). It comes with just the basics, and let's you install TF with GPU support by simply doing: conda install -c anaconda tensorflow-gpu
- dekhn 7y ago
  it's incredibly slow.
  - xxxtentachyon 7y ago
    What’s incredibly slow? Installing things with conda?
    - dekhn 7y ago
      conda install for our environment.yml: about 3-5 minutes solving, then 5-10 minutes installing. pip install with almost exactly the same set of packages: 3-5 minutes total.
- jwilber 7y ago
  Anaconda may work well as a virtual environment for some ml projects, but it is by no means a solution for getting a gpu-working installation of tensorflow on Windows.
  - jacquesm 7y ago
    > getting a gpu-working installation of tensorflow on Windows I think part of your problem is the last word there. Windows is a bad match for such an environment. On Ubuntu it is pretty much painless.
    - jwilber 7y ago
      I agree, and prefer Linux myself, but some clients only allow for solutions based in Windows (and no containers) :/
  - deleted 7y ago
    [deleted]

ghop02 7y ago

They're (slowly) making this better. Starting with 1.15, there is only one tensorflow pip package, no tensorflow/tensorflow-gpu hell any longer.

n8henrie 7y ago
Really? https://www.tensorflow.org/install/gpu https://www.tensorflow.org/install/gpu says to `pip install tensorflow-gpu`.
- ghop02 7y ago
  https://groups.google.com/a/tensorflow.org/forum/#!topic/developers/iRCt5m4qUz0 https://groups.google.com/a/tensorflow.org/forum/#!topic/dev... Ah I guess it's on the way but not fully there yet

mikehollinger 7y ago

Try using these containers[1] from my peer team at IBM. They run on a variety of architectures (x86, ppc64le, with and without GPUs in most cases). In addition if you don't want to fiddle with the containers, there's also a conda channel[2] that lines stuff up. I work on a peer team in machine vision, and use these for personal and professional projects. [1] https://hub.docker.com/r/ibmcom/powerai/ https://hub.docker.com/r/ibmcom/powerai/ [2] https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.6.1/navigation/wmlce_install.htm https://www.ibm.com/support/knowledgecenter/en/SS5SF7_1.6.1/...

hadlock 7y ago

Tensorflow sounds like an ideal candidate for running in a container. List out all your approved compatible versions in the Dockerfile and distribute it with your source code and anyone can reproduce your results with the exact same setup.

XCSme 7y ago
Yes, but for personal use, unless someone else already made those containers, I will still have to go myself through the trial-and-error process of finding the right combination of versions. Yes, second installation will be easier, but if I just want it on my PC it doesn't really help.
- hadlock 7y ago
  You can `docker pull username/mytensorflowcontainer` and start from someone elses' work. Looks like Tensorflow has a how-to on the site: https://www.tensorflow.org/install/docker https://www.tensorflow.org/install/docker including working cpu-only and gpu-enabled examples.