4 ms·

Yeah i make a living in the GPU space. I think my comment comes from colleagues having to hold my hand to set up their ML / Python environments with all of thei

Yeah i make a living in the GPU space. I think my comment comes from colleagues having to hold my hand to set up their ML / Python environments with all of their picadellos. In fact its bad enough that i have to use docker to create an insular environment tailored to their specific setup. And Python is like a 1000 times slower when its not using other libs like numpy.

pdntspa 3y ago
Are they not using venvs or something? It should be as simple as python -m venv venv; ./activate; pip install -r requirements.txt
- waynecochran 3y ago
  Everyone has their own way to do this. Every step is broken by some unfamiliar dependency that requires special arcane knowledge to fix. Part of me is a grumpy old man that doesn’t gravitate to the shiny new tools that come out every week that the younger devs keep up with :)
  - pdntspa 3y ago
    pip and venv are neither shiny nor new, it's the standard way of doing things for a while. I am an outsider to python and am incredibly thankful for this standardization, because i agree getting python env set up correctly before venv was a huge pain If your guys arent on this I'd suggest you get them on it, it dramatically simplifies setup
    - waynecochran 3y ago
      Here is a tiny excerpt try to get dvc to work just so I could get the training weights for deployment ... remember I don't develop much w Python... $ dvc pull Command 'dvc' not found, but can be installed with: sudo snap install dvc $ sudo snap install dvc error: This revision of snap "dvc" was published using classic confinement and thus may perform arbitrary system changes outside of the security sandbox that snaps are usually confined to, which may put your system at risk. If you understand and want to proceed repeat the command including --classic. ok I get dvc installed somehow -- don't remember. Time to get the weights... $ python3 -m dvc pull ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! Finally I just have my colleague manually copy the weights. This kind of thing went for hours.
      - pdntspa 3y ago
        Researchers are notorious for writing bad code What even is dvc edit: also- i'd avoid snap and just use your regular package manager.
        waynecochran 3y ago
        I think dvc is like git for large binary files. You need someway to manage your NN weights -- what are other methods?
        pdntspa 3y ago
        git lfs is what everyone is using, HF in particular
      - efiop 3y ago
        Hey, DVC maintainer here. Thanks for giving DVC a try! There are a few ways to install dvc, see https://dvc.org/doc/install/linux https://dvc.org/doc/install/linux With snap, you need to use `--classic` flag, as noted in https://dvc.org/doc/install/linux#install-with-snap https://dvc.org/doc/install/linux#install-with-snap Unfortunately that's just how snap works for us there :( Regarding the pull error, it simply looks like you don't have some credentials set up. See https://dvc.org/doc/user-guide/data-management/remote-storage/amazon-s3#amazon-s3-and-compatible-servers https://dvc.org/doc/user-guide/data-management/remote-storag... Still, the error could be better, so that's on us. Feel free to ping us in discord (see invite link in https://dvc.org/support https://dvc.org/support). I'm @ruslan there. We'll be happy to help.
        waynecochran 3y ago
        Thanks… i know my colleague uses it a lot. I generally use his models and don’t do much ML development yet. At some point I need to properly learn all of this. It seems ML tools are only for developers not for those who simply want to deploy and use the resulting NN.
- Narew 3y ago
  Unfortunatly its not that simple expecially for NVIDIA driver and cuda install. That's why we usually use conda that can handle cuda install but even with that some time it work flawlessly and some time not.
- dragonwriter 3y ago
  > Are they not using venvs or something? It should be as simple as python -m venv venv; ./activate; pip install -r requirements.txt In most cases, it would be possible to do close to that, but it is extremely common to run into things being distributed in the AI/ML space with install instructions that don’t include that, and instruct you to have a global install of a certain Python version, and then to pip install the dependencies (and globally install non-Python package dependencies, if there are any), so even if they’d work in a venv, you have to (1) indepently know you should be doing that, and (b) translate the instructions – which where (1) applies is usually trivial if all the dependencies are proper python packages, but can be more involved otherwise. So, yeah, I can see that a lot of the time the path of least resistance is just to create an isolated container environment for it.

pdntspa 3y ago

Are they not using venvs or something? It should be as simple as python -m venv venv; ./activate; pip install -r requirements.txt

waynecochran 3y ago
Everyone has their own way to do this. Every step is broken by some unfamiliar dependency that requires special arcane knowledge to fix. Part of me is a grumpy old man that doesn’t gravitate to the shiny new tools that come out every week that the younger devs keep up with :)
- pdntspa 3y ago
  pip and venv are neither shiny nor new, it's the standard way of doing things for a while. I am an outsider to python and am incredibly thankful for this standardization, because i agree getting python env set up correctly before venv was a huge pain If your guys arent on this I'd suggest you get them on it, it dramatically simplifies setup
  - waynecochran 3y ago
    Here is a tiny excerpt try to get dvc to work just so I could get the training weights for deployment ... remember I don't develop much w Python... $ dvc pull Command 'dvc' not found, but can be installed with: sudo snap install dvc $ sudo snap install dvc error: This revision of snap "dvc" was published using classic confinement and thus may perform arbitrary system changes outside of the security sandbox that snaps are usually confined to, which may put your system at risk. If you understand and want to proceed repeat the command including --classic. ok I get dvc installed somehow -- don't remember. Time to get the weights... $ python3 -m dvc pull ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help! Finally I just have my colleague manually copy the weights. This kind of thing went for hours.
    - pdntspa 3y ago
      Researchers are notorious for writing bad code What even is dvc edit: also- i'd avoid snap and just use your regular package manager.
      - waynecochran 3y ago
        I think dvc is like git for large binary files. You need someway to manage your NN weights -- what are other methods?
        pdntspa 3y ago
        git lfs is what everyone is using, HF in particular
    - efiop 3y ago
      Hey, DVC maintainer here. Thanks for giving DVC a try! There are a few ways to install dvc, see https://dvc.org/doc/install/linux https://dvc.org/doc/install/linux With snap, you need to use `--classic` flag, as noted in https://dvc.org/doc/install/linux#install-with-snap https://dvc.org/doc/install/linux#install-with-snap Unfortunately that's just how snap works for us there :( Regarding the pull error, it simply looks like you don't have some credentials set up. See https://dvc.org/doc/user-guide/data-management/remote-storage/amazon-s3#amazon-s3-and-compatible-servers https://dvc.org/doc/user-guide/data-management/remote-storag... Still, the error could be better, so that's on us. Feel free to ping us in discord (see invite link in https://dvc.org/support https://dvc.org/support). I'm @ruslan there. We'll be happy to help.
      - waynecochran 3y ago
        Thanks… i know my colleague uses it a lot. I generally use his models and don’t do much ML development yet. At some point I need to properly learn all of this. It seems ML tools are only for developers not for those who simply want to deploy and use the resulting NN.
Narew 3y ago
Unfortunatly its not that simple expecially for NVIDIA driver and cuda install. That's why we usually use conda that can handle cuda install but even with that some time it work flawlessly and some time not.
dragonwriter 3y ago
> Are they not using venvs or something? It should be as simple as python -m venv venv; ./activate; pip install -r requirements.txt In most cases, it would be possible to do close to that, but it is extremely common to run into things being distributed in the AI/ML space with install instructions that don’t include that, and instruct you to have a global install of a certain Python version, and then to pip install the dependencies (and globally install non-Python package dependencies, if there are any), so even if they’d work in a venv, you have to (1) indepently know you should be doing that, and (b) translate the instructions – which where (1) applies is usually trivial if all the dependencies are proper python packages, but can be more involved otherwise. So, yeah, I can see that a lot of the time the path of least resistance is just to create an isolated container environment for it.