4 ms·

I wonder how many days have been wasted on non-programmers trying to get their Conda environment up and running or similar. Half the data science stuff isn't re

I wonder how many days have been wasted on non-programmers trying to get their Conda environment up and running or similar. Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible.

lysecret 5y ago
have you heard of docker?
irrational 5y ago
Which begs the question, why has nobody built anything better?
- taeric 5y ago
  I'd argue that they probably have. It just isn't free.
- belval 5y ago
  Because it's a hard problem and people love hating on Python because it doesn't come with a way to handle all the compiled dependencies that work for every OS. Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
  - matsemann 5y ago
    But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?
    - analog31 5y ago
      There are 365k projects in the "official" package index. While not all of these are important, it's a tip-off to the magnitude of the problem. The habit of blowing past a problem by grabbing a random library and moving on to the next problem leaves us with a mess of dependencies. And many of those were either written by amateurs like us, not maintained, etc. Maybe other languages have fewer libraries, or maybe the habit of grabbing libraries at random evolved concurrently with the rise of Python. My team has a rule that we don't let a project get past a certain stage without proving that it can be installed and run on a clean machine and archiving all of the necessary repo's with the project. It's easily forgotten that testing your installer is part of testing your program.
    - NoThisIsMe 5y ago
      It's not a big problem in Python in general, only in scientific computing / number crunching projects, because of the dependencies on huge complex software, some of it ancient, written in C, Fortran, and C++. So why do we hear about this problem in Python a lot? Well, because it's what's used for the glue/frontend, which is what users work with directly. It's selection bias. Sure, another language might fare somewhat better or worse for this or that reason, but at the end of the day it's gonna be a pain in the ass (at least until next-generation, complete, deterministic, language-agnostic solutions like Nix/GUIX really gain traction).
    - belval 5y ago
      Python is for all intents and purposes a "glue" language. You don't do the heavy computing in Python, you just pull in a C++ library that has a Python interface. This adds a ton of friction because these dependencies will often not be precompiled so you need to have the right system libraries to build the module before using it. It's not much of a problem for other stacks because they either are fast enough that they have a library written in the same language for problem X (C#/Java/Rust) or they aren't targeting the same type of work (JS, Ruby, etc...). C++ has the exact same problem as Python and I'd argue that it's even worse.
    - m4x 5y ago
      I'm not sure that it is a problem in Python more than other languages. It might look worse because many Python projects use tools such as CUDA, which are notoriously dependent on the specific OS, architecture, method of installation etc. But that same issue will exist in most languages - if you're linking against CUDA, you will sometimes have problems with the package installation. Particularly if you try to run the code on a different OS, CPU architecture, using a different GPU, etc. I don't think it really has anything to do with Python. It just happens that most people doing work that depends on tricky packages such as CUDA also happen to be using Python.
- ryukafalz 5y ago
  They have: Nix and Guix for example can handle Python dependencies as well as native dependencies, and can build them reproducibly. They just haven’t caught on yet.
- nerdponx 5y ago
  There is the Mamba project, which is developing a drop-in Conda replacement, along with a bunch of other related tooling.
belval 5y ago
> non-programmers trying to get their Conda environment up and running I see this issue brought up a lot, but I have yet to see a language that addresses this reliably. By definition setting up an environment for non-programmer is a tall order, what language should they use?
- mnw21cam 5y ago
  The simple and easy Java way to do it is to just bundle everything into a Jar. Then it really is a single file "environment". Then you only have the problem of different Java versions rejecting the jar file because it is too new grumble.
  - _Algernon_ 5y ago
    Wouldn't this be the perfect use case for docker containers (or something equivalent). Once the initial Dockerfile is written its very low maintenance, though getting results out of it could probably be made easier for scientific use.
- matsemann 5y ago
  I'm just grumbling because even I as a professional dev can sometimes spend days getting some python project up and running correctly. Then I feel sorry for non-devs for which all this is only a tool.
- smitty1e 5y ago
  One hopes that Python's packaging story can be streamlined in the next few versions. Never needed Conda, but the packaging tool proliferation is an embarrassment, IMO.
nerdponx 5y ago
I think a lot of this has to do with just how bad/incomplete the docs are, how unnecessarily janky the shell integration is, and how the Anaconda launcher itself makes a huge mess and actively works against best practices. The docs for building your own packages are even worse, to the point where you basically are left copying snippets from Conda Forge to build anything nontrivial. Basically Conda is a tremendous engineering achievement, but it's very much still a "first draft" in a lot of ways, and Continuum/Anaconda made some weird decisions that work against its user-friendliness. Imagine for example if third-party repos on anaconda.org could have a description box, link to a homepage, etc...
BeFlatXIII 5y ago
I'm increasingly convinced that the majority of so-called "data science" is pure sciencism with little to no actual science for exactly this reason. It's reading correlations from digital tea leaves.
deleted 5y ago
[deleted]
tehnub 5y ago
My advice: Docker.
- m4x 5y ago
  Docker isn't a good fit in HPC environments, but you can achieve a similar thing using Singularity.
BeetleB 5y ago
> Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible. As someone who did scientific programming in other languages (Fortran/C++), I can assure you the nonreproducibility was there in those projects as well. Not because of the tech stack but because no one valued reproducibility. The current situation with notebooks isn't worse. It's more of the same. I think people criticize it more because notebooks are advertised as reproducible research.

lysecret 5y ago

have you heard of docker?

irrational 5y ago

Which begs the question, why has nobody built anything better?

taeric 5y ago
I'd argue that they probably have. It just isn't free.
belval 5y ago
Because it's a hard problem and people love hating on Python because it doesn't come with a way to handle all the compiled dependencies that work for every OS. Data science has a ton of moving library parts, it is genuinely difficult to distribute precompiled libraries for everyone when you have 2-3 actively maintained CUDA version with 2 cuDNN version for accelerators that change every 2 years. Most team fail to standardize on an environment (say Python 3.8, Ubuntu 20.04, CUDA 11.1, and cuDNN 8) and then get hung up on a dependency not building as if it's Python's fault that it does not have control of your entire OS.
- matsemann 5y ago
  But why is it such a big problem in Python compared to other stacks? Why does all python projects end up depending on you having those exact tools of things installed locally and the planets aligned a certain way, when other stacks do not?
  - analog31 5y ago
    There are 365k projects in the "official" package index. While not all of these are important, it's a tip-off to the magnitude of the problem. The habit of blowing past a problem by grabbing a random library and moving on to the next problem leaves us with a mess of dependencies. And many of those were either written by amateurs like us, not maintained, etc. Maybe other languages have fewer libraries, or maybe the habit of grabbing libraries at random evolved concurrently with the rise of Python. My team has a rule that we don't let a project get past a certain stage without proving that it can be installed and run on a clean machine and archiving all of the necessary repo's with the project. It's easily forgotten that testing your installer is part of testing your program.
  - NoThisIsMe 5y ago
    It's not a big problem in Python in general, only in scientific computing / number crunching projects, because of the dependencies on huge complex software, some of it ancient, written in C, Fortran, and C++. So why do we hear about this problem in Python a lot? Well, because it's what's used for the glue/frontend, which is what users work with directly. It's selection bias. Sure, another language might fare somewhat better or worse for this or that reason, but at the end of the day it's gonna be a pain in the ass (at least until next-generation, complete, deterministic, language-agnostic solutions like Nix/GUIX really gain traction).
  - belval 5y ago
    Python is for all intents and purposes a "glue" language. You don't do the heavy computing in Python, you just pull in a C++ library that has a Python interface. This adds a ton of friction because these dependencies will often not be precompiled so you need to have the right system libraries to build the module before using it. It's not much of a problem for other stacks because they either are fast enough that they have a library written in the same language for problem X (C#/Java/Rust) or they aren't targeting the same type of work (JS, Ruby, etc...). C++ has the exact same problem as Python and I'd argue that it's even worse.
  - m4x 5y ago
    I'm not sure that it is a problem in Python more than other languages. It might look worse because many Python projects use tools such as CUDA, which are notoriously dependent on the specific OS, architecture, method of installation etc. But that same issue will exist in most languages - if you're linking against CUDA, you will sometimes have problems with the package installation. Particularly if you try to run the code on a different OS, CPU architecture, using a different GPU, etc. I don't think it really has anything to do with Python. It just happens that most people doing work that depends on tricky packages such as CUDA also happen to be using Python.
ryukafalz 5y ago
They have: Nix and Guix for example can handle Python dependencies as well as native dependencies, and can build them reproducibly. They just haven’t caught on yet.
nerdponx 5y ago
There is the Mamba project, which is developing a drop-in Conda replacement, along with a bunch of other related tooling.

belval 5y ago

> non-programmers trying to get their Conda environment up and running I see this issue brought up a lot, but I have yet to see a language that addresses this reliably. By definition setting up an environment for non-programmer is a tall order, what language should they use?

mnw21cam 5y ago
The simple and easy Java way to do it is to just bundle everything into a Jar. Then it really is a single file "environment". Then you only have the problem of different Java versions rejecting the jar file because it is too new grumble.
- _Algernon_ 5y ago
  Wouldn't this be the perfect use case for docker containers (or something equivalent). Once the initial Dockerfile is written its very low maintenance, though getting results out of it could probably be made easier for scientific use.
matsemann 5y ago
I'm just grumbling because even I as a professional dev can sometimes spend days getting some python project up and running correctly. Then I feel sorry for non-devs for which all this is only a tool.
smitty1e 5y ago
One hopes that Python's packaging story can be streamlined in the next few versions. Never needed Conda, but the packaging tool proliferation is an embarrassment, IMO.

nerdponx 5y ago

I think a lot of this has to do with just how bad/incomplete the docs are, how unnecessarily janky the shell integration is, and how the Anaconda launcher itself makes a huge mess and actively works against best practices. The docs for building your own packages are even worse, to the point where you basically are left copying snippets from Conda Forge to build anything nontrivial. Basically Conda is a tremendous engineering achievement, but it's very much still a "first draft" in a lot of ways, and Continuum/Anaconda made some weird decisions that work against its user-friendliness. Imagine for example if third-party repos on anaconda.org could have a description box, link to a homepage, etc...

BeFlatXIII 5y ago

I'm increasingly convinced that the majority of so-called "data science" is pure sciencism with little to no actual science for exactly this reason. It's reading correlations from digital tea leaves.

deleted 5y ago

[deleted]

tehnub 5y ago

My advice: Docker.

m4x 5y ago
Docker isn't a good fit in HPC environments, but you can achieve a similar thing using Singularity.

BeetleB 5y ago

> Half the data science stuff isn't reproducible, not because of the science, but because getting the notebooks running with its dependencies is almost impossible. As someone who did scientific programming in other languages (Fortran/C++), I can assure you the nonreproducibility was there in those projects as well. Not because of the tech stack but because no one valued reproducibility. The current situation with notebooks isn't worse. It's more of the same. I think people criticize it more because notebooks are advertised as reproducible research.