5 ms·

Rust is easier than C++ for this kind of work because of its packages. In some ways Rust is closer to Python for prototyping an idea to see if it works. Rust

Rust is easier than C++ for this kind of work because of its packages. In some ways Rust is closer to Python for prototyping an idea to see if it works. Rust gets challenging when you have to deal with large systems, which is not a problem for research. Researchers / scientists will sometimes not even write functions. It's that small of a scope.

st1x7 6y ago
> Researchers / scientists will rarely even write functions. It's that small of a scope Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread. The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need functions. Those are completely different worlds. Choosing Rust for something that small is a waste of time. Expecting the people who write so little code that their code doesn't require functions, to be comfortable with Rust is even a bigger stretch.
- proverbialbunny 6y ago
  The code I'm talking about needs to run fast (or be type safe) or you wouldn't consider using Rust. But internally in a library or a function that gets ran over and over again and needs to run super fast often isn't a lot of lines of code. So what I mean by a researcher may not even write a function is, they write the meat of the algorithm, it gets handed off to software engineers, and they turn it into a library or OOP it or whatever, and then run it on servers. My current project in Python is massive compared to a normal one atm and it's about 2-3 pages of code in Jupyter. In Rust, if the libraries are there, it would be about the same size. I have 8 functions atm. It's also the end of the project and I'm in talks with data engineers to wrap it in OOP. They'll handle the IO. Or at least, that's how I've always done it and I've done a project in Rust.
  - deleted 6y ago
    [deleted]
  - mattkrause 6y ago
    I'm a little jealous you work somewhere where there are data engineers to hand off to!
    - proverbialbunny 6y ago
      I have to push to get them hired sometimes. Technically they're infrastructure software engineers without the title, but most people don't know what that is, so it can be easier to call them data engineers. The company doesn't give titles like that, they're just "the cloud people" with a vanilla software engineer title. Unless you're a BI / business analyst, there are probably software engineers that work with AWS / gCloud at your company, who setup the database(s) and do the backend. What you want to do is called an IT request, where you request they help out when you need to productionize. It's bad form for a data scientist or any other kind of researcher to have the keys to the live production servers. While it can be fun to learn that stuff, there is a barrier of entry, specially monitoring software and unit tests. This is more on the PM (or DPM (data project manager)) side but: https://www.datascience-pm.com/domino-data-science-lifecycle/ https://www.datascience-pm.com/domino-data-science-lifecycle... can give a good map of the process. Many who do research are on their own, not apart of larger teams and are disconnected, so they don't know what options they have. Maybe that roadmap can help, depending on your situation. When it comes to productionization, I've had too many years of software engineers rewriting my models and adding bugs left and right. The second they want to rewrite it from a notebook into a py file (or equivalent language, usually Java), I get cautious. I find the best process skips these issues and the software engineers will thank you for it: Automate most of the productionization process. Using notebooks as an example, python has libraries that can import notebooks, so you don't have to copy paste code into a py file. Instead an OOP wrapper (which they love) can be written with the interface they like. They can do the IO like streaming data or batching or whatever. They can do the unit tests and monitoring part. What you can do is write the function calls inside of the interface. Usually you want a .predict() method, similar to how ML algos work, but .run() is popular too. Inside of that, put all of the functions from the notebook you want called and in what order. In metaphor it's like writing a header file in C or C++. This py file will then import the notebook and call those functions. The beauty of this is at least four fold: 1) no new bugs 2) no explaining how the model works, because you're doing that part and 3) If you want to update the model, you can update the notebook and then no work needs to be done in the py file. It's importing the notebook, so it will import the changes. It's fully automated. 4) You can have EDA, plots, documentation, all that stuff that usually takes up 40-60% of the notebook, not be required to remove it, because it will not get called. The only downside is that every cell in the notebook that the py file will call needs to be wrapped in a function. However, this is a good idea by default, because you'll get global variable type bugs when you run the cells out of order if your cells are not wrapped in a function, so to avoid that you're probably already writing functions in your notebooks. *Sometimes code has to go fast, and notebooks tend to use dataframes, which is fine for 99% of the work out there. In that case, instead of having the software engineers rewriting my model, because of the problems mentioned above, I will instead 1) Write libraries in C/C++ that accelerate parts of the notebook. And if that can't be done I will 2) write the model directly while prototyping in a language like Rust, C, C++, or Java. I'd rather write it than have someone port it. Most will not do this, because it's beyond the average researcher's skill, but it alleviates so many problems, and I'm not afraid of statically typed languages.
- 7thaccount 6y ago
  I think I'm in agreement with you. One usually needs Fortran or C++ or Matlab or Python with Numpy for numerical computing that is doing matrix or sparse matrix work (a lot of numerical computing is like this). I don't know of any uses that are so small as to not need functions outside of some extremely small/simplified models for explanatory purposes. There is also symbolic mathematics that Mathematica really excels at although it's pretty good at a lot of things these days. If someone just needs fast code to loop through some data, they should be able to use whatever compiled or JIT language they see fit. Heck, it sounds like bash/ask might even fit this use case if the code is so small as to not need functions.
  - vram22 6y ago
    >Heck, it sounds like bash/ask might even fit this use case if the code is so small as to not need functions. Assuming that "ask" was a typo and you meant "awk", awk does support user-defined functions, as does bash. https://www.gnu.org/software/gawk/manual/html_node/User_002ddefined.html#:~:text=9.2%20User-Defined%20Functions,awk%20what%20they%20should% https://www.gnu.org/software/gawk/manual/html_node/User_002d... https://tldp.org/LDP/abs/html/functions.html https://tldp.org/LDP/abs/html/functions.html The support for that in both is somewhat limited as compared to "full" programming languages, though.
    - 7thaccount 6y ago
      Yep, autocorrect I guess. My point being if someone is just doing a little string work than she'll scripting can be very performant and also dead simple. I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything.
      - proverbialbunny 6y ago
        >I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything. It typically is. If you have the right libraries, it's only a few lines of code to do a lot of math. C = A + B could be matrix math, not just single variables being added.
zozbot234 6y ago
C++ still has a few desirable features re: value-parameterized templates and compile-time metaprogramming that currently lack an idiomatic Rust equivalent. In principle, you can replicate anything C++ can do via procedural macros, but it's not necessarily easy. Also proc macros are a nightly-only feature, so there's that.
- steveklabnik 6y ago
  Proc macros are not a nightly only feature.

st1x7 6y ago

> Researchers / scientists will rarely even write functions. It's that small of a scope Scientific computing can mean so many different things that if feels like some of us are talking past each other in this thread. The person in the article talks about needing to get out every ounce of performance and writing an application that identifies genome variants at scale. You're talking code so small that it doesn't need functions. Those are completely different worlds. Choosing Rust for something that small is a waste of time. Expecting the people who write so little code that their code doesn't require functions, to be comfortable with Rust is even a bigger stretch.

proverbialbunny 6y ago
The code I'm talking about needs to run fast (or be type safe) or you wouldn't consider using Rust. But internally in a library or a function that gets ran over and over again and needs to run super fast often isn't a lot of lines of code. So what I mean by a researcher may not even write a function is, they write the meat of the algorithm, it gets handed off to software engineers, and they turn it into a library or OOP it or whatever, and then run it on servers. My current project in Python is massive compared to a normal one atm and it's about 2-3 pages of code in Jupyter. In Rust, if the libraries are there, it would be about the same size. I have 8 functions atm. It's also the end of the project and I'm in talks with data engineers to wrap it in OOP. They'll handle the IO. Or at least, that's how I've always done it and I've done a project in Rust.
- deleted 6y ago
  [deleted]
- mattkrause 6y ago
  I'm a little jealous you work somewhere where there are data engineers to hand off to!
  - proverbialbunny 6y ago
    I have to push to get them hired sometimes. Technically they're infrastructure software engineers without the title, but most people don't know what that is, so it can be easier to call them data engineers. The company doesn't give titles like that, they're just "the cloud people" with a vanilla software engineer title. Unless you're a BI / business analyst, there are probably software engineers that work with AWS / gCloud at your company, who setup the database(s) and do the backend. What you want to do is called an IT request, where you request they help out when you need to productionize. It's bad form for a data scientist or any other kind of researcher to have the keys to the live production servers. While it can be fun to learn that stuff, there is a barrier of entry, specially monitoring software and unit tests. This is more on the PM (or DPM (data project manager)) side but: https://www.datascience-pm.com/domino-data-science-lifecycle/ https://www.datascience-pm.com/domino-data-science-lifecycle... can give a good map of the process. Many who do research are on their own, not apart of larger teams and are disconnected, so they don't know what options they have. Maybe that roadmap can help, depending on your situation. When it comes to productionization, I've had too many years of software engineers rewriting my models and adding bugs left and right. The second they want to rewrite it from a notebook into a py file (or equivalent language, usually Java), I get cautious. I find the best process skips these issues and the software engineers will thank you for it: Automate most of the productionization process. Using notebooks as an example, python has libraries that can import notebooks, so you don't have to copy paste code into a py file. Instead an OOP wrapper (which they love) can be written with the interface they like. They can do the IO like streaming data or batching or whatever. They can do the unit tests and monitoring part. What you can do is write the function calls inside of the interface. Usually you want a .predict() method, similar to how ML algos work, but .run() is popular too. Inside of that, put all of the functions from the notebook you want called and in what order. In metaphor it's like writing a header file in C or C++. This py file will then import the notebook and call those functions. The beauty of this is at least four fold: 1) no new bugs 2) no explaining how the model works, because you're doing that part and 3) If you want to update the model, you can update the notebook and then no work needs to be done in the py file. It's importing the notebook, so it will import the changes. It's fully automated. 4) You can have EDA, plots, documentation, all that stuff that usually takes up 40-60% of the notebook, not be required to remove it, because it will not get called. The only downside is that every cell in the notebook that the py file will call needs to be wrapped in a function. However, this is a good idea by default, because you'll get global variable type bugs when you run the cells out of order if your cells are not wrapped in a function, so to avoid that you're probably already writing functions in your notebooks. *Sometimes code has to go fast, and notebooks tend to use dataframes, which is fine for 99% of the work out there. In that case, instead of having the software engineers rewriting my model, because of the problems mentioned above, I will instead 1) Write libraries in C/C++ that accelerate parts of the notebook. And if that can't be done I will 2) write the model directly while prototyping in a language like Rust, C, C++, or Java. I'd rather write it than have someone port it. Most will not do this, because it's beyond the average researcher's skill, but it alleviates so many problems, and I'm not afraid of statically typed languages.
7thaccount 6y ago
I think I'm in agreement with you. One usually needs Fortran or C++ or Matlab or Python with Numpy for numerical computing that is doing matrix or sparse matrix work (a lot of numerical computing is like this). I don't know of any uses that are so small as to not need functions outside of some extremely small/simplified models for explanatory purposes. There is also symbolic mathematics that Mathematica really excels at although it's pretty good at a lot of things these days. If someone just needs fast code to loop through some data, they should be able to use whatever compiled or JIT language they see fit. Heck, it sounds like bash/ask might even fit this use case if the code is so small as to not need functions.
- vram22 6y ago
  >Heck, it sounds like bash/ask might even fit this use case if the code is so small as to not need functions. Assuming that "ask" was a typo and you meant "awk", awk does support user-defined functions, as does bash. https://www.gnu.org/software/gawk/manual/html_node/User_002ddefined.html#:~:text=9.2%20User-Defined%20Functions,awk%20what%20they%20should% https://www.gnu.org/software/gawk/manual/html_node/User_002d... https://tldp.org/LDP/abs/html/functions.html https://tldp.org/LDP/abs/html/functions.html The support for that in both is somewhat limited as compared to "full" programming languages, though.
  - 7thaccount 6y ago
    Yep, autocorrect I guess. My point being if someone is just doing a little string work than she'll scripting can be very performant and also dead simple. I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything.
    - proverbialbunny 6y ago
      >I normally think of scientific programming as something involving lots of large matrices, but that isn't the case for everything. It typically is. If you have the right libraries, it's only a few lines of code to do a lot of math. C = A + B could be matrix math, not just single variables being added.

zozbot234 6y ago

C++ still has a few desirable features re: value-parameterized templates and compile-time metaprogramming that currently lack an idiomatic Rust equivalent. In principle, you can replicate anything C++ can do via procedural macros, but it's not necessarily easy. Also proc macros are a nightly-only feature, so there's that.

steveklabnik 6y ago
Proc macros are not a nightly only feature.