5 ms·

Setup Anaconda, Jupyter, and Rust

71 points by batterylow 6y ago

superbcarrot 6y ago
From the banner for "Data Analysis with Rust Notebooks" this looks someone realised that both data stuff and Rust are popular on their own and decided to put them together to sell an e-book. It's cool that this is at all possible but ultimately it forces Rust into a use case that it isn't well suited for. I can't see a good reason to pick it over Python/R/Julia.
- whimsicalism 6y ago
  Speed, static types. Definitely not a general alternative.
- batterylow 6y ago
  In the section before there is agreement with your comment: > Can we write and execute all our code in a Jupyter Notebook? Yes! Should we? Probably not. However, I enjoy the workflow, and making this an enjoyable process is important to me.
- kyllo 6y ago
  Yeah I think Rust's place in this ecosystem is a C/C++ replacement. All the Python and R data science packages call into native code to do the heavy lifting--the linear algebra, gradient descent etc. The next Tensorflow or PyTorch type framework could be implemented in Rust instead of C++. But it shouldn't matter to the end users--they would still use the Python or R bindings so they can have an interactive REPL environment to do their analysis in. Julia is interesting because it's performant enough to implement these things directly--Julia's machine learning frameworks are written in pure Julia (with the exception of the calls to CUDA libraries for GPU) and can achieve near-native performance, so solving the "two language problem" is part of its value proposition. I do wonder how valuable it is for the whole stack to be written in one language, and whether that will blur the lines between the software engineers who implement data science packages and the data scientists who use them.
  - amkkma 6y ago
    Julia can do GPU kernels (Intel, AMD, Nvidia) also, they just aren't all implemented in pure Julia (or tuned) yet, hence the CUDA lib calls. https://juliagpu.org/ https://juliagpu.org/ https://juliagpu.org/2020-09-28-gemmkernels/ https://juliagpu.org/2020-09-28-gemmkernels/
pratio 6y ago
Why Rust? Conda and Jupyter are python based. What does one hope to achieve with a setup like this?
- jamesmishra 6y ago
  Rust is a great language for writing Python extensions, using a library like PyO3. Some of the most CPU-intensive parts of my company's feature engineering pipeline are now handled in Rust. I would enjoy writing Rust for Python even more if I could compile and run Rust straight from the same Jupyter notebooks that I prototype my Python code with. https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3
  - pratio 6y ago
    I wasn't aware of pyo3 thanks for that.
- RayDonnelly 6y ago
  conda, although written in Python, is language (and somewhat OS) agnostic. We provide lots of C, C++, R and Rust packages as well as Python ones.
1MachineElf 6y ago
I was initially confused over the name conflict with the GUI Linux distro installer Anacoda: https://en.wikipedia.org/wiki/Anaconda_(installer) https://en.wikipedia.org/wiki/Anaconda_(installer) In the context of this post, Anaconda is a Python and R distribution using the conda package manager: https://en.wikipedia.org/wiki/Anaconda_(Python_distribution) https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)
- whimsicalism 6y ago
  The python and R distribution is much better known than the GUI Linux installer.
  - ibotty 6y ago
    Depends on the bubble you are in.
aardvarkr 6y ago
The decision to shoehorn rust into a jupyter notebook is baffling, especially for something like data science. I like rust and all but come on. Just because it’s possible doesn’t mean you should.
j-wags 6y ago
> Once Miniconda is installed, we need to create and configure our environment. If you added Miniconda to your PATH environment during the installation process, then you can run these commands directly from Terminal, Powershell, or CMD. AFAIK, the PATH modification route is no longer recommended. Instead one should use, eg `. miniconda3/etc/profile.d/conda.sh` to add a particular miniconda installaiton to the environment.
- kowlo 6y ago
  It does say "if", but with MiniConda's CLI installer it's currently > Do you wish the installer to initialize Miniconda3 by running conda init [yes|no]: with the default as [no] as well
auxym 6y ago
Worth mentioning that commercial use of anaconda repositories (including via miniconda, to my understanding) is no longer free and requires a commercial license. https://www.anaconda.com/blog/anaconda-commercial-edition-faq https://www.anaconda.com/blog/anaconda-commercial-edition-fa... Beware if you are following this guide for your work.
- StreakyCobra 6y ago
  > We clarified our definition of commercial usage in our Terms of Service in an update on Sept. 30, 2020. The new language states that use by individual hobbyists, students, universities, non-profit organizations, or businesses with less than 200 employees is allowed, and all other usage is considered commercial and thus requires a business relationship with Anaconda. Looks like small companies are exonerated from commercial license.
  - batterylow 6y ago
    > businesses with less than 200 employees is allowed This is great!
    - deleted 6y ago
      [deleted]
    - bawana 6y ago
      No businesses with >200 employees should be allowed to exist.
      - batterylow 6y ago
        I don't know about that... I just meant that it's exceptionally reasonable
      - anchpop 6y ago
        why? seems like that would make any large scale business venture completely impossible and would cut us off from important economies of scale

71 points by batterylow 6y ago

From the banner for "Data Analysis with Rust Notebooks" this looks someone realised that both data stuff and Rust are popular on their own and decided to put them together to sell an e-book. It's cool that this is at all possible but ultimately it forces Rust into a use case that it isn't well suited for. I can't see a good reason to pick it over Python/R/Julia.

whimsicalism 6y ago
Speed, static types. Definitely not a general alternative.
batterylow 6y ago
In the section before there is agreement with your comment: > Can we write and execute all our code in a Jupyter Notebook? Yes! Should we? Probably not. However, I enjoy the workflow, and making this an enjoyable process is important to me.
kyllo 6y ago
Yeah I think Rust's place in this ecosystem is a C/C++ replacement. All the Python and R data science packages call into native code to do the heavy lifting--the linear algebra, gradient descent etc. The next Tensorflow or PyTorch type framework could be implemented in Rust instead of C++. But it shouldn't matter to the end users--they would still use the Python or R bindings so they can have an interactive REPL environment to do their analysis in. Julia is interesting because it's performant enough to implement these things directly--Julia's machine learning frameworks are written in pure Julia (with the exception of the calls to CUDA libraries for GPU) and can achieve near-native performance, so solving the "two language problem" is part of its value proposition. I do wonder how valuable it is for the whole stack to be written in one language, and whether that will blur the lines between the software engineers who implement data science packages and the data scientists who use them.
- amkkma 6y ago
  Julia can do GPU kernels (Intel, AMD, Nvidia) also, they just aren't all implemented in pure Julia (or tuned) yet, hence the CUDA lib calls. https://juliagpu.org/ https://juliagpu.org/ https://juliagpu.org/2020-09-28-gemmkernels/ https://juliagpu.org/2020-09-28-gemmkernels/

pratio 6y ago

Why Rust? Conda and Jupyter are python based. What does one hope to achieve with a setup like this?

jamesmishra 6y ago
Rust is a great language for writing Python extensions, using a library like PyO3. Some of the most CPU-intensive parts of my company's feature engineering pipeline are now handled in Rust. I would enjoy writing Rust for Python even more if I could compile and run Rust straight from the same Jupyter notebooks that I prototype my Python code with. https://github.com/PyO3/pyo3 https://github.com/PyO3/pyo3
- pratio 6y ago
  I wasn't aware of pyo3 thanks for that.
RayDonnelly 6y ago
conda, although written in Python, is language (and somewhat OS) agnostic. We provide lots of C, C++, R and Rust packages as well as Python ones.

1MachineElf 6y ago

I was initially confused over the name conflict with the GUI Linux distro installer Anacoda: https://en.wikipedia.org/wiki/Anaconda_(installer) https://en.wikipedia.org/wiki/Anaconda_(installer) In the context of this post, Anaconda is a Python and R distribution using the conda package manager: https://en.wikipedia.org/wiki/Anaconda_(Python_distribution) https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)

whimsicalism 6y ago
The python and R distribution is much better known than the GUI Linux installer.
- ibotty 6y ago
  Depends on the bubble you are in.

aardvarkr 6y ago

The decision to shoehorn rust into a jupyter notebook is baffling, especially for something like data science. I like rust and all but come on. Just because it’s possible doesn’t mean you should.

j-wags 6y ago

> Once Miniconda is installed, we need to create and configure our environment. If you added Miniconda to your PATH environment during the installation process, then you can run these commands directly from Terminal, Powershell, or CMD. AFAIK, the PATH modification route is no longer recommended. Instead one should use, eg `. miniconda3/etc/profile.d/conda.sh` to add a particular miniconda installaiton to the environment.

kowlo 6y ago
It does say "if", but with MiniConda's CLI installer it's currently > Do you wish the installer to initialize Miniconda3 by running conda init [yes|no]: with the default as [no] as well

auxym 6y ago

Worth mentioning that commercial use of anaconda repositories (including via miniconda, to my understanding) is no longer free and requires a commercial license. https://www.anaconda.com/blog/anaconda-commercial-edition-faq https://www.anaconda.com/blog/anaconda-commercial-edition-fa... Beware if you are following this guide for your work.

StreakyCobra 6y ago
> We clarified our definition of commercial usage in our Terms of Service in an update on Sept. 30, 2020. The new language states that use by individual hobbyists, students, universities, non-profit organizations, or businesses with less than 200 employees is allowed, and all other usage is considered commercial and thus requires a business relationship with Anaconda. Looks like small companies are exonerated from commercial license.
- batterylow 6y ago
  > businesses with less than 200 employees is allowed This is great!
  - deleted 6y ago
    [deleted]
  - bawana 6y ago
    No businesses with >200 employees should be allowed to exist.
    - batterylow 6y ago
      I don't know about that... I just meant that it's exceptionally reasonable
    - anchpop 6y ago
      why? seems like that would make any large scale business venture completely impossible and would cut us off from important economies of scale