4 ms·

I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's

I took a quick look through the example for relational extensions. I don't see anything about reading a file, much less inferring type from a file. Empirical's key innovation is that it can infer type from an external source so as long as the source can be derived at compile time. let path = "/path/to/files" let fname = "quotes.csv" let quotes = load(path + "/" + fname) Since load() requires a compile-time parameter, Empirical automatically resolves the file's location ahead of time. Specifically, Empirical determines that the variables are constants and that the plus operator is pure; computing the resulting value has no side effects. Thus, the Empirical compiler accommodates load() by computing the parameter during semantic analysis. This is automatic compile-time function evaluation. As for load(), it is a macro whose expansion includes inferring the schema of the table by sampling the first few lines of the file. This is a type provider. Now contrast that logic against something like Apache Spark. It's written in a statically typed language (Scala), but its table semantics are dynamically typed! That's because Spark holds the schema as a runtime value and addresses column names as strings. Spark's mechanism for inferring type means it can't maintain static typing. Empirical is the only system I know of that can infer types while staying static.

sideeffffect 5y ago
> Empirical is the only system I know of that can infer types while staying static. F#'s Type Providers can do that too. https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/type-providers/ https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...
- chrisaycock 5y ago
  F# is how I learned about type providers: http://fsprojects.github.io/FSharp.Data/library/CsvProvider.html http://fsprojects.github.io/FSharp.Data/library/CsvProvider.... But look at that syntax: CsvProvider<"file.csv">.Load("file.csv") The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string. F# has Literals that mitigate some of this, but my understanding is that they are pretty limited: https://github.com/fsharp/fslang-suggestions/issues/539 https://github.com/fsharp/fslang-suggestions/issues/539 On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc. func filename(n): let value = String(n + 1) return "/path/to/" + value + ".csv" end load(filename(7)) I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.
  - chusk3 5y ago
    Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.

sideeffffect 5y ago

> Empirical is the only system I know of that can infer types while staying static. F#'s Type Providers can do that too. https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/type-providers/ https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/typ...

chrisaycock 5y ago
F# is how I learned about type providers: http://fsprojects.github.io/FSharp.Data/library/CsvProvider.html http://fsprojects.github.io/FSharp.Data/library/CsvProvider.... But look at that syntax: CsvProvider<"file.csv">.Load("file.csv") The file name is listed twice! Conceivably a macro could hide that, but we'd still have the issue that the string has to be written fully. I.e., I can't just call a function to generate the string. F# has Literals that mitigate some of this, but my understanding is that they are pretty limited: https://github.com/fsharp/fslang-suggestions/issues/539 https://github.com/fsharp/fslang-suggestions/issues/539 On the other hand, Empirical can perform automatic compile-time function evaluation on any legal expression, including across variables, user-defined functions, user-defined datatypes, etc. func filename(n): let value = String(n + 1) return "/path/to/" + value + ".csv" end load(filename(7)) I could put the filename in a Dataframe embedded in my source, then reference the row/column I want. Empirical's automatic CTFE handles it.
- chusk3 5y ago
  Nitpick here = the `Load` member takes a path to _any_ file of the same schema as the sample, but you can also just use the sample at runtime with the use of `Load()`. So it doesn't have to be as redundant as you've shown here, the flexibility is all about loading other instances of the same data shape.