3 ms·

This was the subject of the talk that I gave with a cospeaker at RustConf. Effectively, the shortcomings of dynamic type systems when it comes to building data

This was the subject of the talk that I gave with a cospeaker at RustConf. Effectively, the shortcomings of dynamic type systems when it comes to building data science oriented services. You mention that it's much omre laborious to write a struct that matches your data exactly - however, I've found that, at this stage in the process you already have some sort of schema (mental or formal) of what your data is, and it's fairly trivial to write out a structured representation of it. What this then buys you is a semantic understanding, in your code, of what 'bad data' is and how it's handled. You know that if you get a string where you expected an int, that's a problem. You also know that if data is missing, eg: null, that you'll handle it. In Python, a function can erroneously return None. To your data, this is just a missing value, not a bug. But there is a semantic difference, and a cost associated with that bug. All of these problems, limited strictly to working with data, are far easier to react to and deal with in a static language. We absolutely conceded the exploratory phase to Python. It is, and will likely remain, an excellent and near-ideal tool for that phase.

andybak 10y ago
> In Python, a function can erroneously return None. If it hurts stop doing it. I can think of several ways to solve this ambiguity. Off the top of my head: 1. Create a custom class to represent your data's 'None' and treat the real 'None' as an error. 2. Stick to a fairly reasonable convention that all functions and methods are terminated with a single return statement. This would make it pretty hard to get bitten by the implicit None (which has never actually bitten me in the real world)
- staticassertion 10y ago
  Absolutely, and we did this in fact in some of our code. But at that point you're solving the problem with types, which is my point. And it involves a pattern, not something that's enforced across a codebase.
  - johnobrien1010 10y ago
    It seems like the argument is because the language doesn't force you to use static types, in the long run it could be bad. But since it provides you the option to use static types, I think the limitation is more in whether you do so or not, not in the language. If anything, by not forcing you to use static types, the language provides more options, which seems like a good thing...
    - staticassertion 10y ago
      > It seems like the argument is because the language doesn't force you to use static types, in the long run it could be bad. Not so much 'in the long run' but moreso 'as soon as you end the exploratory phase'. > I think the limitation is more in whether you do so or not, not in the language Perhaps if the type systems were equal in power, but they are not. > If anything, by not forcing you to use static types, the language provides more options, which seems like a good thing... The problem here is that touching an untyped interface can break inference, and break assumptions about types, meaning that by having the type system be optional you potentially poke holes in it. But to be clear, I think that the typed python approach is valid, and can be a great way to help with these problems. I am not saying to ditch python and go with another typed language - just saying that the untyped approach has caveats. Maybe you're right and optional systems are better, but I think the point to get across is that a typed understanding of your data helps a lot.
disgruntledphd2 10y ago
Was your talk recorded? I'd really like a link if so.
- staticassertion 10y ago
  https://www.youtube.com/watch?v=lY10kTcM8ek https://www.youtube.com/watch?v=lY10kTcM8ek We talk about technical debt in data science and how rust (and the type system) can help.

andybak 10y ago

> In Python, a function can erroneously return None. If it hurts stop doing it. I can think of several ways to solve this ambiguity. Off the top of my head: 1. Create a custom class to represent your data's 'None' and treat the real 'None' as an error. 2. Stick to a fairly reasonable convention that all functions and methods are terminated with a single return statement. This would make it pretty hard to get bitten by the implicit None (which has never actually bitten me in the real world)

staticassertion 10y ago
Absolutely, and we did this in fact in some of our code. But at that point you're solving the problem with types, which is my point. And it involves a pattern, not something that's enforced across a codebase.
- johnobrien1010 10y ago
  It seems like the argument is because the language doesn't force you to use static types, in the long run it could be bad. But since it provides you the option to use static types, I think the limitation is more in whether you do so or not, not in the language. If anything, by not forcing you to use static types, the language provides more options, which seems like a good thing...
  - staticassertion 10y ago
    > It seems like the argument is because the language doesn't force you to use static types, in the long run it could be bad. Not so much 'in the long run' but moreso 'as soon as you end the exploratory phase'. > I think the limitation is more in whether you do so or not, not in the language Perhaps if the type systems were equal in power, but they are not. > If anything, by not forcing you to use static types, the language provides more options, which seems like a good thing... The problem here is that touching an untyped interface can break inference, and break assumptions about types, meaning that by having the type system be optional you potentially poke holes in it. But to be clear, I think that the typed python approach is valid, and can be a great way to help with these problems. I am not saying to ditch python and go with another typed language - just saying that the untyped approach has caveats. Maybe you're right and optional systems are better, but I think the point to get across is that a typed understanding of your data helps a lot.

disgruntledphd2 10y ago

Was your talk recorded? I'd really like a link if so.

staticassertion 10y ago
https://www.youtube.com/watch?v=lY10kTcM8ek https://www.youtube.com/watch?v=lY10kTcM8ek We talk about technical debt in data science and how rust (and the type system) can help.