3 ms·

We've been fine-ish with classic setup.py/setup.cfg + gha for publishing to pypi. But as we do OSS data science (gpu graph ai + viz), where conda is typical now

We've been fine-ish with classic setup.py/setup.cfg + gha for publishing to pypi. But as we do OSS data science (gpu graph ai + viz), where conda is typical nowadays... ... Have to admit: We recently ended up contracting conda packaging out because it was nowhere near clear enough to make sense for our core team to untangle. Would love to see a similar tutorial on a github flow packaging & publishing to conda. Still no convinced we're doing it right for subtleties like optional dependencies: equivalent of `pip install graphistry` vs `pip install graphistry[ai]` vs `graphistry[umap-learn]`, etc.

RojerGS 4y ago
Oh, I see... I personally barely use conga and I have no idea how that is done. I don't think I'll write any blog article like that any time soon :( Maybe you could do it!
plonk 4y ago
We do a lot of deep learning and image processing and pip works much better for us. PyTorch makes wheels that contain all reauired DLLs on all systems. Maybe conda isn’t needed anymore.
- notemaker 4y ago
  I think of conda as jupyter notebooks. It's ok to experiment with and get something running quickly, but it's a bad idea to use for production code. Personally I never use either of these tools.
- quantumduck 4y ago
  Conda let's you install a specific cuda version directly in a virtual environment with one click though. It's really useful when you have to switch between multiple PyTorch versions and convinient in general imo.
  - plonk 4y ago
    The PyTorch wheel comes with its own CUDA DLLs, they can’t be shared with other libraries but they won’t be your problem to install that way.
werewolf 4y ago
I found conda-forge community quite helpful here [1]. They make feedstock repositories based on templates that cover a lot of automation. Their bots pickup updated packages in pypi and automatically file merge requests, run tests and even merge updates if tests pass successfully. Basically you only need to maintain your recipe here and there when your dependencies change. 1. https://conda-forge.org/docs/user/introduction.html https://conda-forge.org/docs/user/introduction.html
- lmeyerov 4y ago
  Yep, that was one of the better docs! The contractor still took a couple weeks to figure out and get up. I assumed it'd be one evening for initial bulk as we already had setup.cfg etc, but after searching conda tutorials.. not surprised. Our ~final meta file is pretty simple, so no idea why the docs are so indirect.

RojerGS 4y ago

Oh, I see... I personally barely use conga and I have no idea how that is done. I don't think I'll write any blog article like that any time soon :( Maybe you could do it!

plonk 4y ago

We do a lot of deep learning and image processing and pip works much better for us. PyTorch makes wheels that contain all reauired DLLs on all systems. Maybe conda isn’t needed anymore.

notemaker 4y ago
I think of conda as jupyter notebooks. It's ok to experiment with and get something running quickly, but it's a bad idea to use for production code. Personally I never use either of these tools.
quantumduck 4y ago
Conda let's you install a specific cuda version directly in a virtual environment with one click though. It's really useful when you have to switch between multiple PyTorch versions and convinient in general imo.
- plonk 4y ago
  The PyTorch wheel comes with its own CUDA DLLs, they can’t be shared with other libraries but they won’t be your problem to install that way.

werewolf 4y ago

I found conda-forge community quite helpful here [1]. They make feedstock repositories based on templates that cover a lot of automation. Their bots pickup updated packages in pypi and automatically file merge requests, run tests and even merge updates if tests pass successfully. Basically you only need to maintain your recipe here and there when your dependencies change. 1. https://conda-forge.org/docs/user/introduction.html https://conda-forge.org/docs/user/introduction.html

lmeyerov 4y ago
Yep, that was one of the better docs! The contractor still took a couple weeks to figure out and get up. I assumed it'd be one evening for initial bulk as we already had setup.cfg etc, but after searching conda tutorials.. not surprised. Our ~final meta file is pretty simple, so no idea why the docs are so indirect.