3 ms·

> I've been wondering what might move data scientists to Rust. I think dynamic typing is the missing piece. Rust's safety guarantees are great and I like the

> I've been wondering what might move data scientists to Rust. I think dynamic typing is the missing piece. Rust's safety guarantees are great and I like the language, but the type system is a barrier for learners, especially people who aren't full-time devs. Ease of experimentation and flexibility are key for this use case.

CharlesW 3y ago
> Rust's safety guarantees are great and I like the language, but the type system is a barrier for learners, especially people who aren't full-time devs. This surprises me. Hasn't everyone working with data (even at an "enthusiast" level) had to understand data types in the context of databases?
- rirze 3y ago
  You underestimate the gusto of data engineers/scientists to just assume types.
- spenczar5 3y ago
  Yes, but Rust’s type system is waaaaay more sophisticated than your typical SQL database. Talk about “generic associated types” and you can see people start to shrink back in fear. Plus, a surprisingly enormous fraction of the NoSQL hysteria was that “you don’t even need a schema!”
  - josephg 3y ago
    > Plus, a surprisingly enormous fraction of the NoSQL hysteria was that “you don’t even need a schema!” The problem with this is that you have a schema regardless. In "schemaless" systems, you still have a schema. You just don't have it written down anywhere. And because its not written down, the database doesn't enforce your schema. So its very common for nosql databases to have records with missing fields or subtly corrupted data. That said, there's a reason why nosql became popular. I think a large part of it is that SQL databases are too inflexible. You have to define the entire schema ahead of time, and schema evolution is way too difficult. When I'm writing software, I don't define all of my columns ahead of time. I want to add and change stuff as I go. I wish databases were more flexible. Like, let me just add arbitrary extra fields to my records. Let the database recommend promoting a dynamic field to a static field. ("Hey, all your records have a name: string field. Wanna make that required?"). I love programming languages with similar features. I love typescript for prototyping because I can start with the type any (or wildcard fields in my interfaces - like [k: string]: any). And then I can formalize my types later when I understand my software better. I don't think schemas are a mistake for beginners. I think the opposite - they're amazing guard rails. But I agree with other commenters here. Rust's types include information about ownership, lifetime, containment (Box / Rc), traits and the stored data type all in one go. Its terrifying to beginners and exhausting for experts to use. I've been writing rust for years, and the Future trait still scares me.
- jstx1 3y ago
  There's a bit of general dev snobbery to assume that data scientists can't code, or don't want to learn to code, or are able to understand so many other things but not a compiled language or a static type system.
  - chongli 3y ago
    It's not about ability to code, willingness to learn, or ability to understand. It's about friction. When you're iterating quickly and you're thinking about complicated mathematics and ideas, a complicated type system just adds a ton of friction. This is especially annoying when you know what you're doing is correct but the type system is unsatisfied and demands proof.
    - throwawaymaths 3y ago
      the place in DL/ML where typing could be the most useful is to keep you from trying to multiply differently sized tensors together. So the irony is that the one place where typing WOULD help DL/ML, is the place where rust is weak in its typing system (dependent types and const generics)
      - dgacmu 3y ago
        I keep checking the progress on dfdx for this reason. It does what I (and, I assume from context, you) want: Provides static checking of tensor shapes. Which is fantastic. Not quite as much inference as I'd like but I love getting compile-time errors that I forgot to transpose before a matmul. It depends on the generic_const_exprs feature which is still, to quote, "highly experimental": https://github.com/rust-lang/rust/issues/76560 https://github.com/rust-lang/rust/issues/76560 Definitely not for production use, but it gives a flavor for where things can head in the medium term, and it's .. it's nice. You could imagine future type support allowing even more inference for some intermediate shapes, of course, but even what it has now is really nice. Like this cute little convnet example: https://github.com/coreylowman/dfdx/blob/main/examples/nightly-conv-net.rs https://github.com/coreylowman/dfdx/blob/main/examples/night...
- 0cf8612b2e1e 3y ago
  A lot of science/engineering/ML/data analysis happens from hacking around with only a vague idea of what to expect. Appeasing a type checker simultaneously is an enormous blow to productivity. Obviously possible, but challenging to get into a flow state when you cannot quickly iterate on ideas.
  - lnxg33k1 3y ago
    I mean you can still hold data in hashmaps no? Or even serde::Value(s)? Its not that everything has to be a type
    - ArrowH3ad 3y ago
      If we're going to bypass the type system entirely then why are we using Rust at all? Might as well just use Python at this point
      - lnxg33k1 3y ago
        Who said entirely? I’ve used hashmaps prototyping my code, still used types in other areas, then converting dynamic maps to types too, Was just suggesting ways to handle types during prototyping
- otikik 3y ago
  Not with an ownership model and lifetimes. Most of the scientists I know have not even dealt with parametric types.
DarkNova6 3y ago
It is not a barrier for learners. It is a barrier for people that just want to hack something together that works for one particular use-case. Python is not popular because it is "easy" (far from it if you have to maintain it). But because it is easy to create executable code that does what you want, without having to have a background in coding.
PartiallyTyped 3y ago
The type system is certainly not a barrier, it’s actually rather straightforward unless I am missing something. If you refer to lifetimes and borrow checker, those are different.

CharlesW 3y ago

> Rust's safety guarantees are great and I like the language, but the type system is a barrier for learners, especially people who aren't full-time devs. This surprises me. Hasn't everyone working with data (even at an "enthusiast" level) had to understand data types in the context of databases?

rirze 3y ago
You underestimate the gusto of data engineers/scientists to just assume types.
spenczar5 3y ago
Yes, but Rust’s type system is waaaaay more sophisticated than your typical SQL database. Talk about “generic associated types” and you can see people start to shrink back in fear. Plus, a surprisingly enormous fraction of the NoSQL hysteria was that “you don’t even need a schema!”
- josephg 3y ago
  > Plus, a surprisingly enormous fraction of the NoSQL hysteria was that “you don’t even need a schema!” The problem with this is that you have a schema regardless. In "schemaless" systems, you still have a schema. You just don't have it written down anywhere. And because its not written down, the database doesn't enforce your schema. So its very common for nosql databases to have records with missing fields or subtly corrupted data. That said, there's a reason why nosql became popular. I think a large part of it is that SQL databases are too inflexible. You have to define the entire schema ahead of time, and schema evolution is way too difficult. When I'm writing software, I don't define all of my columns ahead of time. I want to add and change stuff as I go. I wish databases were more flexible. Like, let me just add arbitrary extra fields to my records. Let the database recommend promoting a dynamic field to a static field. ("Hey, all your records have a name: string field. Wanna make that required?"). I love programming languages with similar features. I love typescript for prototyping because I can start with the type any (or wildcard fields in my interfaces - like [k: string]: any). And then I can formalize my types later when I understand my software better. I don't think schemas are a mistake for beginners. I think the opposite - they're amazing guard rails. But I agree with other commenters here. Rust's types include information about ownership, lifetime, containment (Box / Rc), traits and the stored data type all in one go. Its terrifying to beginners and exhausting for experts to use. I've been writing rust for years, and the Future trait still scares me.
jstx1 3y ago
There's a bit of general dev snobbery to assume that data scientists can't code, or don't want to learn to code, or are able to understand so many other things but not a compiled language or a static type system.
- chongli 3y ago
  It's not about ability to code, willingness to learn, or ability to understand. It's about friction. When you're iterating quickly and you're thinking about complicated mathematics and ideas, a complicated type system just adds a ton of friction. This is especially annoying when you know what you're doing is correct but the type system is unsatisfied and demands proof.
  - throwawaymaths 3y ago
    the place in DL/ML where typing could be the most useful is to keep you from trying to multiply differently sized tensors together. So the irony is that the one place where typing WOULD help DL/ML, is the place where rust is weak in its typing system (dependent types and const generics)
    - dgacmu 3y ago
      I keep checking the progress on dfdx for this reason. It does what I (and, I assume from context, you) want: Provides static checking of tensor shapes. Which is fantastic. Not quite as much inference as I'd like but I love getting compile-time errors that I forgot to transpose before a matmul. It depends on the generic_const_exprs feature which is still, to quote, "highly experimental": https://github.com/rust-lang/rust/issues/76560 https://github.com/rust-lang/rust/issues/76560 Definitely not for production use, but it gives a flavor for where things can head in the medium term, and it's .. it's nice. You could imagine future type support allowing even more inference for some intermediate shapes, of course, but even what it has now is really nice. Like this cute little convnet example: https://github.com/coreylowman/dfdx/blob/main/examples/nightly-conv-net.rs https://github.com/coreylowman/dfdx/blob/main/examples/night...
0cf8612b2e1e 3y ago
A lot of science/engineering/ML/data analysis happens from hacking around with only a vague idea of what to expect. Appeasing a type checker simultaneously is an enormous blow to productivity. Obviously possible, but challenging to get into a flow state when you cannot quickly iterate on ideas.
- lnxg33k1 3y ago
  I mean you can still hold data in hashmaps no? Or even serde::Value(s)? Its not that everything has to be a type
  - ArrowH3ad 3y ago
    If we're going to bypass the type system entirely then why are we using Rust at all? Might as well just use Python at this point
    - lnxg33k1 3y ago
      Who said entirely? I’ve used hashmaps prototyping my code, still used types in other areas, then converting dynamic maps to types too, Was just suggesting ways to handle types during prototyping
otikik 3y ago
Not with an ownership model and lifetimes. Most of the scientists I know have not even dealt with parametric types.

DarkNova6 3y ago

It is not a barrier for learners. It is a barrier for people that just want to hack something together that works for one particular use-case. Python is not popular because it is "easy" (far from it if you have to maintain it). But because it is easy to create executable code that does what you want, without having to have a background in coding.

PartiallyTyped 3y ago

The type system is certainly not a barrier, it’s actually rather straightforward unless I am missing something. If you refer to lifetimes and borrow checker, those are different.