3 ms·

When I'm wrangling data, it's not necessarily that I don't want to have to deal with the type checker. It's that I have yet to see a statically typed language w

When I'm wrangling data, it's not necessarily that I don't want to have to deal with the type checker. It's that I have yet to see a statically typed language whose type system is designed to help solve the kinds of problems I have when I'm wrangling data. The distressing example that nags at me here is how the headline feature of Apache Spark 2.0 is that they Greenspunned a dynamic typing mechanism into it. And, in doing so, realized improvements in ergonomics, performance, and memory consumption. Understanding why and how is an instructive lesson. The most promising typing system for arbitrary data wrangling that I've seen so far is the one that's built into a Python package called Dagster. It seems reminiscent, as best I can tell, of dependent typing. (Although I haven't actually spent any time with a dependently typed language, so that could be a misunderstanding on my part.) The actual checks are done at run time, of course, but I don't see any particular reason, aside perhaps from the complexity involved, why a similar mechanism couldn't be implemented as a static type check.

wk_end 6y ago
What are the sorts of problems you have when wrangling data that type systems don't help with? I'm not a data scientist, but I'm a type system guy, and I'd like to learn about ways that type systems are failing users. I took a very quick peak at Dagster and the Google results for "apache spark 2.0 dynamic typing" (are you talking about "Datasets"?) and nothing immediately jumped out as a clear sort of elucidation of what you're saying.
- snicker7 6y ago
  In data science, types change all the times. E.g. when dealing with tabular data, the columns (which might be used as type parameters) change frequently. In Julia, for example, see DataFrames.jl vs. TypedTables.jl for some trade-offs. In general, type annotations are useful if you know the business requirements beforehand. Data science is exploratory, where type annotations can be a form of premature abstraction. Languages with traits (e.g. Scala) might be a decent compromise.
  - rowanG077 6y ago
    But now you are talking about type annotations and not static typing. You can have static typing without any type annotations
  - wk_end 6y ago
    It sounds like the structural typing in Typescript might help a lot here - you don’t need to formally define any types for a record, the language just infers and tracks fields/types for you. I don’t know what the current state of data science is for TS though.
- deleted 6y ago
  [deleted]
canadianfella 6y ago
Greenspunned?

wk_end 6y ago

What are the sorts of problems you have when wrangling data that type systems don't help with? I'm not a data scientist, but I'm a type system guy, and I'd like to learn about ways that type systems are failing users. I took a very quick peak at Dagster and the Google results for "apache spark 2.0 dynamic typing" (are you talking about "Datasets"?) and nothing immediately jumped out as a clear sort of elucidation of what you're saying.

snicker7 6y ago
In data science, types change all the times. E.g. when dealing with tabular data, the columns (which might be used as type parameters) change frequently. In Julia, for example, see DataFrames.jl vs. TypedTables.jl for some trade-offs. In general, type annotations are useful if you know the business requirements beforehand. Data science is exploratory, where type annotations can be a form of premature abstraction. Languages with traits (e.g. Scala) might be a decent compromise.
- rowanG077 6y ago
  But now you are talking about type annotations and not static typing. You can have static typing without any type annotations
- wk_end 6y ago
  It sounds like the structural typing in Typescript might help a lot here - you don’t need to formally define any types for a record, the language just infers and tracks fields/types for you. I don’t know what the current state of data science is for TS though.
deleted 6y ago
[deleted]

canadianfella 6y ago

Greenspunned?