3 ms·

The intended audience of other ML developers and data scientists will likely be more familiar with Anaconda[1] than docker. I recommend starting with miniconda[

by pavon 4y ago

The intended audience of other ML developers and data scientists will likely be more familiar with Anaconda[1] than docker. I recommend starting with miniconda[2] which will install just the packages needed to run this project (as declared in that environment.yml file), whereas Anaconda is a giant batteries-included install. [1] https://www.anaconda.com/ https://www.anaconda.com/ [2] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html

TekMol 4y ago
Thanks I might try it later. But I will probably not have a big enough GPU? Can I rent a VM to run it, or do I need to rent a dedicated server?
- klodolph 4y ago
  It's not just a question of whether the GPU is big enough. Sometimes you need to tweak this stuff so it runs on the type of GPU that you have. I would suggest trying it out first, and then looking at a VM later. Yes, you can rent a VM, you just need to get a VM with enough memory, and with a GPU attached. This is how I do it. Just remember to turn the VM off. A VM with a GPU may cost something like $0.45 an hour, which is over $300 per month.
  - TekMol 4y ago
    I'm totally fine with $0.45 per hour. Where do you rent your VM?
    - klodolph 4y ago
      I use GCS. There are some weird steps you have to go through--like your account might have a limit of 0 GPU instances, so you have to raise it. I also added storage read/write scope to the instance, and made a bucket that the VM's service account could write to, just to make it easier to get the output out. The Nvidia driver has randomly stopped working. I've often run out of system or GPU RAM. I created the VM through the cloud dashboard and then ssh'd to it to set it up. When I'm done, I stop the instance. While it's stopped, I only accrue charges for the disk volume.
      - TekMol 4y ago
        What is GCS?
        dharma1 4y ago
        Google
Y_Y 4y ago
I respectfully disagree. I find docker to be more-or-less unavoidable in ML work, but thankfully conda is still avoidable. My recommendation is not to touch conda, unless you're on Windows and not don't know about cuda drivers or WSL. Otherwise IMHO it's far less trouble to stick with pip (and virtualenv/pipenv/poetry) and it's not difficult to translate the environment.
- stared 4y ago
  While your millage (clearly) varies from mine, Anaconda is in practice the standard environment for deep learning (and, generally, in most of the Python data science ecosystem). For example, when you go to the front page of PyTorch (https://pytorch.org/ https://pytorch.org/), the default way to go is with Anaconda. It precisely makes it easy to install things regardless of the system and with matching versions. For example, out of box, it gives GPU support for Apple Silicon - not extra installation instructions. Pip installers don't work with non-Python dependencies. Of course, you can manually install things any way you like (including inside Docker), but it is up to you to make sure that all dependencies are compatible. And it is a non-trivial task, given frequent updates of all things involved (including CUDA kernels, Python versions, PyTorch/TF versions, and all libraries related to them one way or the other). I know times before Anaconda matured. Docker was often necessary to make code reusable - and I am grateful I don't need to go back to these times. Of course, now you CAN use Docker. Quite a lot of decent installation scripts place conda (usually miniconda) instructions in a Dockerfile.
  - Y_Y 4y ago
    Pip certainly does install plenty of non-python dependencies, but also leaves to the system package manager many of the things managed by conda. I don't know if there's a good definition for "in practice the standard" but I think conda is a heap of shit and I wish people would use it less.
stared 4y ago
To add to that, Anaconda is for the data science ecosystem as NPM for TypeScript. For that reason, it is often assumed that people do know. (Also, automatic tests with GitHub Actions, and similar tools, do wonders to make dependencies explicit AND tested.) However, personally, I always link to such dependencies. "Obviousness" is a subjective criterion. It takes someone coming from a different language (or ecosystem), and it takes quite a lot of guesswork to figure out what to install. And if things do not work - it gives no clue if they are missing dependencies, incompatible versions of software, or maybe neither - there is something wrong with the code.

by pavon 4y ago

TekMol 4y ago

Thanks I might try it later. But I will probably not have a big enough GPU? Can I rent a VM to run it, or do I need to rent a dedicated server?

klodolph 4y ago
It's not just a question of whether the GPU is big enough. Sometimes you need to tweak this stuff so it runs on the type of GPU that you have. I would suggest trying it out first, and then looking at a VM later. Yes, you can rent a VM, you just need to get a VM with enough memory, and with a GPU attached. This is how I do it. Just remember to turn the VM off. A VM with a GPU may cost something like $0.45 an hour, which is over $300 per month.
- TekMol 4y ago
  I'm totally fine with $0.45 per hour. Where do you rent your VM?
  - klodolph 4y ago
    I use GCS. There are some weird steps you have to go through--like your account might have a limit of 0 GPU instances, so you have to raise it. I also added storage read/write scope to the instance, and made a bucket that the VM's service account could write to, just to make it easier to get the output out. The Nvidia driver has randomly stopped working. I've often run out of system or GPU RAM. I created the VM through the cloud dashboard and then ssh'd to it to set it up. When I'm done, I stop the instance. While it's stopped, I only accrue charges for the disk volume.
    - TekMol 4y ago
      What is GCS?
      - dharma1 4y ago
        Google

Y_Y 4y ago

I respectfully disagree. I find docker to be more-or-less unavoidable in ML work, but thankfully conda is still avoidable. My recommendation is not to touch conda, unless you're on Windows and not don't know about cuda drivers or WSL. Otherwise IMHO it's far less trouble to stick with pip (and virtualenv/pipenv/poetry) and it's not difficult to translate the environment.

stared 4y ago
While your millage (clearly) varies from mine, Anaconda is in practice the standard environment for deep learning (and, generally, in most of the Python data science ecosystem). For example, when you go to the front page of PyTorch (https://pytorch.org/ https://pytorch.org/), the default way to go is with Anaconda. It precisely makes it easy to install things regardless of the system and with matching versions. For example, out of box, it gives GPU support for Apple Silicon - not extra installation instructions. Pip installers don't work with non-Python dependencies. Of course, you can manually install things any way you like (including inside Docker), but it is up to you to make sure that all dependencies are compatible. And it is a non-trivial task, given frequent updates of all things involved (including CUDA kernels, Python versions, PyTorch/TF versions, and all libraries related to them one way or the other). I know times before Anaconda matured. Docker was often necessary to make code reusable - and I am grateful I don't need to go back to these times. Of course, now you CAN use Docker. Quite a lot of decent installation scripts place conda (usually miniconda) instructions in a Dockerfile.
- Y_Y 4y ago
  Pip certainly does install plenty of non-python dependencies, but also leaves to the system package manager many of the things managed by conda. I don't know if there's a good definition for "in practice the standard" but I think conda is a heap of shit and I wish people would use it less.

stared 4y ago

To add to that, Anaconda is for the data science ecosystem as NPM for TypeScript. For that reason, it is often assumed that people do know. (Also, automatic tests with GitHub Actions, and similar tools, do wonders to make dependencies explicit AND tested.) However, personally, I always link to such dependencies. "Obviousness" is a subjective criterion. It takes someone coming from a different language (or ecosystem), and it takes quite a lot of guesswork to figure out what to install. And if things do not work - it gives no clue if they are missing dependencies, incompatible versions of software, or maybe neither - there is something wrong with the code.