6 ms·

The "code" link takes you to a GitHub repo. But I don't know how to set up the needed environment. I wish authors of such tools would provide a Dockerfile to s

by TekMol 4y ago

The "code" link takes you to a GitHub repo. But I don't know how to set up the needed environment. I wish authors of such tools would provide a Dockerfile to set up the environment needed to run their creations. Their setup section starts with conda env create -f environment.yml But that gives me bash: conda: command not found So I tried apt install conda and apt install python3-conda But nothing worked. Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? Guess I will wait until some SaaS company provides an API to use this on their machines then. Any ideas, which company might do this first?

teruakohatu 4y ago
> Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? I think this is unfair criticism. Conda is extremely common in data science, as common as Docker is in software development. While I agree docker is easier to use as a consumer, Conda automatically sets up the environment and a quick Google would show you how to install conda.
- TekMol 4y ago
  I just said Dockerfile because when I say "shellscript", people think that is oldschool. A dockerfile shows me every step of what has to be done in an unmistakable way. I can easily turn it into a shellscript. The conda file (environment.yml, right?) does not tell me how to set up the environment. It contains a list of .. I don't know what. It starts with: channels: - pytorch - nvidia - defaults I don't know what to do with that. It does not look like computer code. > a quick Google would show you how to install conda No. It leads me to yet another tutorial with a bunch of manual steps: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html https://docs.conda.io/projects/conda/en/latest/user-guide/in... It even starts with decisions I shall make of which I know nothing: Download the installer: - Miniconda installer for Linux. - Anaconda installer for Linux. How do I know which one the software in question needs? How far down this rabbit hole is someone who just wants to run a software supposed to go?
  - extr 4y ago
    [flagged]
    - rowanG077 4y ago
      Conda is a special type of hell though. I have read the manual. Multiple tutorials. Yet I, not even once got conda to work for me.
  - IanCal 4y ago
    The conda file is for conda to read, it's not a set of instructions for you. > How far down this rabbit hole is someone who just wants to run a software supposed to go? Depends how much you want to run it really. It's a free thing and making easily installable software across wildly different systems is hard. Providing conda setup stuff is actually pretty nice. These things are often hard to setup though, the alternative is waiting for someone else to do it for you.
  - mk_stjames 4y ago
    This isn't software. Software is something like MS Office. This is research code. It's raw. If you want to work with it, you need to know a little about how environments to build the code work. In the case of ML code using Python, nearly everyone is using Conda. Install Anaconda on your system. Create a base environment and make sure your terminal is now using Anaconda for python and not your system python. Then follow their one-liner to setup the environment with their yaml file. Conda will literally download and install all the python packages they need. It is really, really simple.
- klodolph 4y ago
  Having set up various diffusion models I'm gonna say that the "read tutorials and tinker with manually setting up an environment" is EXTREMELY fair criticism. Much of the ML ecosystem is kinda held together with duct tape and string. As I get older, I feel more and more that the initial reactions of people completely new to a software system are incredibly valuable, and the best response is to try and capture those reactions so you can find out what underlying problems (if any) that caused them. When you hire a junior engineer straight out of college, maybe they'll react with horror when they see what real production code looks like. It's EASY to just say that they'll get used to it. However, their reactions are sometimes correct and sometimes miss the mark. It takes work to sort it out, but the first step is capturing the initial reactions of people new to the system, and knee-jerk responses just get in the way. Most of the senior engineers can't see the problems. They've been surrounded by those problems for too long and the problems just turn into background noise.
pavon 4y ago
The intended audience of other ML developers and data scientists will likely be more familiar with Anaconda[1] than docker. I recommend starting with miniconda[2] which will install just the packages needed to run this project (as declared in that environment.yml file), whereas Anaconda is a giant batteries-included install. [1] https://www.anaconda.com/ https://www.anaconda.com/ [2] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html
- TekMol 4y ago
  Thanks I might try it later. But I will probably not have a big enough GPU? Can I rent a VM to run it, or do I need to rent a dedicated server?
  - klodolph 4y ago
    It's not just a question of whether the GPU is big enough. Sometimes you need to tweak this stuff so it runs on the type of GPU that you have. I would suggest trying it out first, and then looking at a VM later. Yes, you can rent a VM, you just need to get a VM with enough memory, and with a GPU attached. This is how I do it. Just remember to turn the VM off. A VM with a GPU may cost something like $0.45 an hour, which is over $300 per month.
    - TekMol 4y ago
      I'm totally fine with $0.45 per hour. Where do you rent your VM?
      - klodolph 4y ago
        I use GCS. There are some weird steps you have to go through--like your account might have a limit of 0 GPU instances, so you have to raise it. I also added storage read/write scope to the instance, and made a bucket that the VM's service account could write to, just to make it easier to get the output out. The Nvidia driver has randomly stopped working. I've often run out of system or GPU RAM. I created the VM through the cloud dashboard and then ssh'd to it to set it up. When I'm done, I stop the instance. While it's stopped, I only accrue charges for the disk volume.
deleted 4y ago
[deleted]
mk_stjames 4y ago
It's obvious you aren't the intended audience for the code; I mean this sincerely when I say their repo is absolutely standard and simple and easily implemented by any undergraduate student in a first year ML course. They literally give a conda environment yaml. Anaconda the most widely used distribution manager for Python and R that takes care of package installs and virtual environment management. If you don't know what that is, I would argue that isn't the fault of the researchers.

by TekMol 4y ago

teruakohatu 4y ago

> Why is running software still such a tricky thing in 2023, where you have to read tutorials and tinker with manually setting up an environment? I think this is unfair criticism. Conda is extremely common in data science, as common as Docker is in software development. While I agree docker is easier to use as a consumer, Conda automatically sets up the environment and a quick Google would show you how to install conda.

TekMol 4y ago
I just said Dockerfile because when I say "shellscript", people think that is oldschool. A dockerfile shows me every step of what has to be done in an unmistakable way. I can easily turn it into a shellscript. The conda file (environment.yml, right?) does not tell me how to set up the environment. It contains a list of .. I don't know what. It starts with: channels: - pytorch - nvidia - defaults I don't know what to do with that. It does not look like computer code. > a quick Google would show you how to install conda No. It leads me to yet another tutorial with a bunch of manual steps: https://docs.conda.io/projects/conda/en/latest/user-guide/install/linux.html https://docs.conda.io/projects/conda/en/latest/user-guide/in... It even starts with decisions I shall make of which I know nothing: Download the installer: - Miniconda installer for Linux. - Anaconda installer for Linux. How do I know which one the software in question needs? How far down this rabbit hole is someone who just wants to run a software supposed to go?
- extr 4y ago
  [flagged]
  - rowanG077 4y ago
    Conda is a special type of hell though. I have read the manual. Multiple tutorials. Yet I, not even once got conda to work for me.
- IanCal 4y ago
  The conda file is for conda to read, it's not a set of instructions for you. > How far down this rabbit hole is someone who just wants to run a software supposed to go? Depends how much you want to run it really. It's a free thing and making easily installable software across wildly different systems is hard. Providing conda setup stuff is actually pretty nice. These things are often hard to setup though, the alternative is waiting for someone else to do it for you.
- mk_stjames 4y ago
  This isn't software. Software is something like MS Office. This is research code. It's raw. If you want to work with it, you need to know a little about how environments to build the code work. In the case of ML code using Python, nearly everyone is using Conda. Install Anaconda on your system. Create a base environment and make sure your terminal is now using Anaconda for python and not your system python. Then follow their one-liner to setup the environment with their yaml file. Conda will literally download and install all the python packages they need. It is really, really simple.
klodolph 4y ago
Having set up various diffusion models I'm gonna say that the "read tutorials and tinker with manually setting up an environment" is EXTREMELY fair criticism. Much of the ML ecosystem is kinda held together with duct tape and string. As I get older, I feel more and more that the initial reactions of people completely new to a software system are incredibly valuable, and the best response is to try and capture those reactions so you can find out what underlying problems (if any) that caused them. When you hire a junior engineer straight out of college, maybe they'll react with horror when they see what real production code looks like. It's EASY to just say that they'll get used to it. However, their reactions are sometimes correct and sometimes miss the mark. It takes work to sort it out, but the first step is capturing the initial reactions of people new to the system, and knee-jerk responses just get in the way. Most of the senior engineers can't see the problems. They've been surrounded by those problems for too long and the problems just turn into background noise.

pavon 4y ago

The intended audience of other ML developers and data scientists will likely be more familiar with Anaconda[1] than docker. I recommend starting with miniconda[2] which will install just the packages needed to run this project (as declared in that environment.yml file), whereas Anaconda is a giant batteries-included install. [1] https://www.anaconda.com/ https://www.anaconda.com/ [2] https://docs.conda.io/en/latest/miniconda.html https://docs.conda.io/en/latest/miniconda.html

TekMol 4y ago
Thanks I might try it later. But I will probably not have a big enough GPU? Can I rent a VM to run it, or do I need to rent a dedicated server?
- klodolph 4y ago
  It's not just a question of whether the GPU is big enough. Sometimes you need to tweak this stuff so it runs on the type of GPU that you have. I would suggest trying it out first, and then looking at a VM later. Yes, you can rent a VM, you just need to get a VM with enough memory, and with a GPU attached. This is how I do it. Just remember to turn the VM off. A VM with a GPU may cost something like $0.45 an hour, which is over $300 per month.
  - TekMol 4y ago
    I'm totally fine with $0.45 per hour. Where do you rent your VM?
    - klodolph 4y ago
      I use GCS. There are some weird steps you have to go through--like your account might have a limit of 0 GPU instances, so you have to raise it. I also added storage read/write scope to the instance, and made a bucket that the VM's service account could write to, just to make it easier to get the output out. The Nvidia driver has randomly stopped working. I've often run out of system or GPU RAM. I created the VM through the cloud dashboard and then ssh'd to it to set it up. When I'm done, I stop the instance. While it's stopped, I only accrue charges for the disk volume.

deleted 4y ago

[deleted]

mk_stjames 4y ago

It's obvious you aren't the intended audience for the code; I mean this sincerely when I say their repo is absolutely standard and simple and easily implemented by any undergraduate student in a first year ML course. They literally give a conda environment yaml. Anaconda the most widely used distribution manager for Python and R that takes care of package installs and virtual environment management. If you don't know what that is, I would argue that isn't the fault of the researchers.