3 ms·

I created Empirical specifically to have statically typed Dataframes (tables). let trades = load("trades.csv") from trades select where symbol == "AAPL" a

I created Empirical specifically to have statically typed Dataframes (tables). let trades = load("trades.csv") from trades select where symbol == "AAPL" and size > 1000 Empirical uses type providers and automatic compile-time function evaluation to sample the CSV file and infer a type ahead of time. The above can be run in a REPL or in a stand-alone script and the result is statically typed! (There is no "gradual" or "optional" types; it is fully static.) Empirical has generic types, which is a syntactic sugar for templates. Here is an example of a five-minute volume-weighted average price (VWAP): func wavg(ws, vs) = sum(ws * vs) / sum(ws) from trades select vwap = wavg(size, price) by symbol, bar(timestamp, 5m) I created Empirical for time-series analysis (I work in finance) and began with the question, "What if q/kdb+ were more like Haskell?" The result is very different from either language, but I wanted to make table actions feel dynamic even though they are fully static. To run a script that takes a command-line argument for the location of the CSV file, just supply the type ahead of time since there isn't a compile-time path to the file. data Trade: symbol: String, timestamp: Timestamp, price: Float64, size: Int64 end let trades = csv_load{Trade}(argv[1]) Feel free to try it out. https://www.empirical-soft.com https://www.empirical-soft.com

gavinray 5y ago
This is pretty interesting. Are you familiar with the "Morel" dialect of SML? It's written by Julian Hyde, who has authored several RDBMS and is the lead maintainer of the Apache Calcite relational algebra framework. Your tool reminds me a bit of the work in Morel, where LINQ-like operators have been integrated into the core of the language: https://github.com/julianhyde/morel https://github.com/julianhyde/morel
- chrisaycock 5y ago
  I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's key innovation is that it can infer type from an external source so as long as the source can be derived at compile time. let path = "/path/to/files" let fname = "quotes.csv" let quotes = load(path + "/" + fname) Since load() requires a compile-time parameter, Empirical automatically resolves the file's location ahead of time. Specifically, Empirical determines that the variables are constants and that the plus operator is pure; computing the resulting value has no side effects. Thus, the Empirical compiler accommodates load() by computing the parameter during semantic analysis. This is automatic compile-time function evaluation. As for load(), it is a macro whose expansion includes inferring the schema of the table by sampling the first few lines of the file. This is a type provider. Now contrast that logic against something like Apache Spark. It's written in a statically typed language (Scala), but its table semantics are dynamically typed! That's because Spark holds the schema as a runtime value and addresses column names as strings. Spark's mechanism for inferring type means it can't maintain static typing. Empirical is the only system I know of that can infer types while staying static.
  - sideeffffect 5y ago
    > Empirical is the only system I know of that can infer types while staying static. F#'s Type Providers can do that too. https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/type-providers/ https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...
    - chrisaycock 5y ago
      F# is how I learned about type providers: http://fsprojects.github.io/FSharp.Data/library/CsvProvider.html http://fsprojects.github.io/FSharp.Data/library/CsvProvider.... But look at that syntax: CsvProvider<"file.csv">.Load("file.csv") The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string. F# has Literals that mitigate some of this, but my understanding is that they are pretty limited: https://github.com/fsharp/fslang-suggestions/issues/539 https://github.com/fsharp/fslang-suggestions/issues/539 On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc. func filename(n): let value = String(n + 1) return "/path/to/" + value + ".csv" end load(filename(7)) I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.
      - chusk3 5y ago
        Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.

gavinray 5y ago

This is pretty interesting. Are you familiar with the "Morel" dialect of SML? It's written by Julian Hyde, who has authored several RDBMS and is the lead maintainer of the Apache Calcite relational algebra framework. Your tool reminds me a bit of the work in Morel, where LINQ-like operators have been integrated into the core of the language: https://github.com/julianhyde/morel https://github.com/julianhyde/morel

chrisaycock 5y ago
I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's key innovation is that it can infer type from an external source so as long as the source can be derived at compile time. let path = "/path/to/files" let fname = "quotes.csv" let quotes = load(path + "/" + fname) Since load() requires a compile-time parameter, Empirical automatically resolves the file's location ahead of time. Specifically, Empirical determines that the variables are constants and that the plus operator is pure; computing the resulting value has no side effects. Thus, the Empirical compiler accommodates load() by computing the parameter during semantic analysis. This is automatic compile-time function evaluation. As for load(), it is a macro whose expansion includes inferring the schema of the table by sampling the first few lines of the file. This is a type provider. Now contrast that logic against something like Apache Spark. It's written in a statically typed language (Scala), but its table semantics are dynamically typed! That's because Spark holds the schema as a runtime value and addresses column names as strings. Spark's mechanism for inferring type means it can't maintain static typing. Empirical is the only system I know of that can infer types while staying static.
- sideeffffect 5y ago
  > Empirical is the only system I know of that can infer types while staying static. F#'s Type Providers can do that too. https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/type-providers/ https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...
  - chrisaycock 5y ago
    F# is how I learned about type providers: http://fsprojects.github.io/FSharp.Data/library/CsvProvider.html http://fsprojects.github.io/FSharp.Data/library/CsvProvider.... But look at that syntax: CsvProvider<"file.csv">.Load("file.csv") The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string. F# has Literals that mitigate some of this, but my understanding is that they are pretty limited: https://github.com/fsharp/fslang-suggestions/issues/539 https://github.com/fsharp/fslang-suggestions/issues/539 On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc. func filename(n): let value = String(n + 1) return "/path/to/" + value + ".csv" end load(filename(7)) I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.
    - chusk3 5y ago
      Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.