3 ms·

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python cod

I loved Python. It was easy to learn, very powerful, has libraries for everything... Then I started supporting researchers and scientists who wrote "python code" to run simulations etc. Most of it's pretty basic, install some scientific code published by some research group. They chuck their data in and run it. But then they started abusing virtual environments, writing their own code, cutting and pasting, commenting out random lines because they saw someone else "fix" something that way... and they all want Jupyter notebooks. Now it's like an eternal September plus I get to deal with annoyingly slow package managers like Conda and rough academic projects with poor documentation and little testing.

Tcepsa 4y ago
That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.
- jmuhlich 4y ago
  To make this more clear as I have had the same experience and want more people to enjoy it: Mamba is more or less a drop-in replacement for the conda tool with the guts rewritten in C++ and as a result is much faster to resolve dependencies -- I want to say an order of magnitude faster or more. It's even noticeably faster at downloading and installing. You just conda install -c conda-forge mamba in your base environment and then you can start running mamba instead of conda.
- ajoseps 4y ago
  Yeah I was looking for this comment. +1 for mamba
zitterbewegung 4y ago
The situations such as improving package management in python is a problem but fundamentally in any software stack you will eventually find bad code. Especially as the code base gets larger and people move on. Instead you can try to fight it by teaching the rest of the group how to write more idiomatic code but that only goes so far. Also training and documentation can go out the window when the objective is to ship something that works once.
deleted 4y ago
[deleted]
mattpallissard 4y ago
Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that. Notebooks on the other hand are a god damned abomination. Obligatory past discussion; https://news.ycombinator.com/item?id=17856700 https://news.ycombinator.com/item?id=17856700
- bluedino 4y ago
  Couldn't you just do that with pip?
  - mattpallissard 4y ago
    No conda is for dependencies of your python environment. You need python itself, shell utilities, and some dependencies are dynamically linked. Think package manager for the `--prefix` compilation flag AND `LD_LIBRARY_PATH`
    - myhf 4y ago
      conda can also be used to manage python packages, but its dependency solver is very slow and sometimes completely fails. Once you have created a conda environment, it is usually much more practical to install packages with pip or mamba.
- abdullahkhalids 4y ago
  > Notebooks on the other hand are a god damned abomination. I completely disagree. Notebooks are an excellent way of doing literate programming, because they go beyond plain text, and integrate richly formatted text, tables, figures, videos with code. This is excellent where the human is following a line of reasoning, and offloading the computation to the computer. The reasoning is explained via the non-code, made concrete via the code cells, and its effects shown via the code cells. Of course, people can abuse the conveniences of the medium to write really bad code, and follow some really bad practices. But that doesn't mean the medium is bad. In fact, I think if you forced people to stop using notebooks, their productivity would drop by quite a bit. Notebooks are tool for thought [1]. The solution is to teach them better practices, and provide better tools within the notebook interface that will help people write better code. [1] https://numinous.productions/ttft/ https://numinous.productions/ttft/
isoprophlex 4y ago
Having been in the same situation as you, you have to bring some technical leadership if you want to change things. And some empathy, these people are stuck in a local maximum, they try their best probably with limited tools and knowledge. Conda is a shitshow for sure: teach them how things could be better. Notebooks are a reproducibility nightmare: show them how nice eg. Pycharm is People are unqualified to write O(n) code in less than O(n^3) time: identify skill gaps and get their management to sign off on some software development / python training. If you can make a convincing case why your approach is better, you can make everyone's life better.

Tcepsa 4y ago

That sounds similar to my experience with the Python ecosystem (of which I have a profound dislike at the moment). If you haven't tried Mamba yet I encourage you to give it a shot; for me it has had a pretty significant impact on the time I have to spend sitting around waiting for Conda packages to install.

jmuhlich 4y ago
To make this more clear as I have had the same experience and want more people to enjoy it: Mamba is more or less a drop-in replacement for the conda tool with the guts rewritten in C++ and as a result is much faster to resolve dependencies -- I want to say an order of magnitude faster or more. It's even noticeably faster at downloading and installing. You just conda install -c conda-forge mamba in your base environment and then you can start running mamba instead of conda.
ajoseps 4y ago
Yeah I was looking for this comment. +1 for mamba

zitterbewegung 4y ago

The situations such as improving package management in python is a problem but fundamentally in any software stack you will eventually find bad code. Especially as the code base gets larger and people move on. Instead you can try to fight it by teaching the rest of the group how to write more idiomatic code but that only goes so far. Also training and documentation can go out the window when the objective is to ship something that works once.

deleted 4y ago

[deleted]

mattpallissard 4y ago

Conda may suck in some ways but it solved a big problem; installing software without root accesss. It effectively removed os dependencies allowing for operating system changes. HPC operating systems were effectively ossified prior to that. Notebooks on the other hand are a god damned abomination. Obligatory past discussion; https://news.ycombinator.com/item?id=17856700 https://news.ycombinator.com/item?id=17856700

bluedino 4y ago
Couldn't you just do that with pip?
- mattpallissard 4y ago
  No conda is for dependencies of your python environment. You need python itself, shell utilities, and some dependencies are dynamically linked. Think package manager for the `--prefix` compilation flag AND `LD_LIBRARY_PATH`
  - myhf 4y ago
    conda can also be used to manage python packages, but its dependency solver is very slow and sometimes completely fails. Once you have created a conda environment, it is usually much more practical to install packages with pip or mamba.
abdullahkhalids 4y ago
> Notebooks on the other hand are a god damned abomination. I completely disagree. Notebooks are an excellent way of doing literate programming, because they go beyond plain text, and integrate richly formatted text, tables, figures, videos with code. This is excellent where the human is following a line of reasoning, and offloading the computation to the computer. The reasoning is explained via the non-code, made concrete via the code cells, and its effects shown via the code cells. Of course, people can abuse the conveniences of the medium to write really bad code, and follow some really bad practices. But that doesn't mean the medium is bad. In fact, I think if you forced people to stop using notebooks, their productivity would drop by quite a bit. Notebooks are tool for thought [1]. The solution is to teach them better practices, and provide better tools within the notebook interface that will help people write better code. [1] https://numinous.productions/ttft/ https://numinous.productions/ttft/

isoprophlex 4y ago

Having been in the same situation as you, you have to bring some technical leadership if you want to change things. And some empathy, these people are stuck in a local maximum, they try their best probably with limited tools and knowledge. Conda is a shitshow for sure: teach them how things could be better. Notebooks are a reproducibility nightmare: show them how nice eg. Pycharm is People are unqualified to write O(n) code in less than O(n^3) time: identify skill gaps and get their management to sign off on some software development / python training. If you can make a convincing case why your approach is better, you can make everyone's life better.