4 ms·

Statically typed dataframes are exactly why I created the Empirical programming language: https://www.empirical-soft.com https://www.empirical-soft.com It can

Statically typed dataframes are exactly why I created the Empirical programming language: https://www.empirical-soft.com https://www.empirical-soft.com It can infer the column names and types from a CSV file at compile time. Here's an example that misspells the "ask" column as if it were plural: let quotes = load("quotes.csv") sort quotes by (asks - bid) / bid The error is caught before the script is run: Error: symbol asks was not found I had to use a lot of computer-science techniques to get this working, like type providers and compile-time function evaluation. I'm really proud of the novelty of it and even won Y Combinator's Startup School grant for it. Unfortunately, it didn't go anywhere as a project. Turns out that static typing isn't enough of a selling point for people to drop Python. I haven't touched Empirical in four years, but my code and my notes are still publicly available on the website.

theLiminator 2y ago
Does this require that the file is available locally or does it do network io at compile time?
- chrisaycock 2y ago
  The inferencing logic needs to sample the file, so (1) the file path must be determined at compile time and (2) the file must be available to be read at compile time. If neither condition is true---like the filename is a runtime parameter, for example---then the user must supply the type in advance. There is no magic here. No language can guess the type of anything without seeing what the thing is.
  - theLiminator 2y ago
    Yeah, i think that's what limits the utility of such systems. Polars does typechecking at query planning time. So before you really do computation. I don't expect that much can improve over this model due to the aforementioned limitations. I think needing network access or file access at compile time is a semi-hard blocker for statically typed dataframes.
noworriesnate 2y ago
Wow this is amazing!! Thanks for sharing! I love how you really expanded on the idea of executing code at compile time. You should be proud. You probably already know this but for people like me to switch "all" it would take would be: 1. A plotting library like ggplot2 or plotnine 2. A machine learning library, like scikit 3. A dashboard framework like streamlit or shiny 4. Support for Empirical in my cloud workspace environment, which is Jupyter based, and where I have to execute all the code, because that's where the data is and has to stay due to security Just like how Polars is written in Rust and has Python bindings, I wonder if there's a market for 1 and 2 written in Rust and then having bindings to Python, Empirical, R, Julia etc. I feel like 4 is just a matter of time if Empirical becomes popular, but I think 3 would have to be implemented specifically for Empirical. I think the idea of statically typed dataframes is really useful and you were ahead of your time. Maybe one day the time will be right.

theLiminator 2y ago

Does this require that the file is available locally or does it do network io at compile time?

chrisaycock 2y ago
The inferencing logic needs to sample the file, so (1) the file path must be determined at compile time and (2) the file must be available to be read at compile time. If neither condition is true---like the filename is a runtime parameter, for example---then the user must supply the type in advance. There is no magic here. No language can guess the type of anything without seeing what the thing is.
- theLiminator 2y ago
  Yeah, i think that's what limits the utility of such systems. Polars does typechecking at query planning time. So before you really do computation. I don't expect that much can improve over this model due to the aforementioned limitations. I think needing network access or file access at compile time is a semi-hard blocker for statically typed dataframes.

noworriesnate 2y ago

Wow this is amazing!! Thanks for sharing! I love how you really expanded on the idea of executing code at compile time. You should be proud. You probably already know this but for people like me to switch "all" it would take would be: 1. A plotting library like ggplot2 or plotnine 2. A machine learning library, like scikit 3. A dashboard framework like streamlit or shiny 4. Support for Empirical in my cloud workspace environment, which is Jupyter based, and where I have to execute all the code, because that's where the data is and has to stay due to security Just like how Polars is written in Rust and has Python bindings, I wonder if there's a market for 1 and 2 written in Rust and then having bindings to Python, Empirical, R, Julia etc. I feel like 4 is just a matter of time if Empirical becomes popular, but I think 3 would have to be implemented specifically for Empirical. I think the idea of statically typed dataframes is really useful and you were ahead of your time. Maybe one day the time will be right.