3 ms·

For anyone interested in array programming, the person who made k eventually created a query language called q: https://code.kx.com/q/ref/ https://code.kx.com/

For anyone interested in array programming, the person who made k eventually created a query language called q: https://code.kx.com/q/ref/ https://code.kx.com/q/ref/ Now days we refer to tables-within-a-language as "dataframes". These are widely available in R, pandas, Spark, etc. The q language inspired me to write my own language a few years back that added static typing to dataframes: https://www.empirical-soft.com https://www.empirical-soft.com

titanomachy 4y ago
Your work might predate this, but spark/scala now has good support for typed dataframes (they’re called DataSets).
- chrisaycock 4y ago
  Does Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Datasets require an explicit type provided by the user. Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.
kthielen 4y ago
Very cool, this approach will inevitably catch on, I think. I built a similar typed PL and compiler for Morgan Stanley back in 2013: https://github.com/morganstanley/hobbes https://github.com/morganstanley/hobbes
- chrisaycock 4y ago
  I read about Hobbes years ago. The embeddability certainly caught my attention. I wrote Empirical specifically for the use case of wanting to read a CSV file while inferring the type at compile time. Basically, a ton of compile-time function evaluation mixed with type providers.
  - kthielen 4y ago
    Yeah, I had this type constraint system for doing that kind of thing, so I can load a file at compile time to get the type structure and then compile/evaluate to process data in the file or files. Same thing for network connections between communicating processes (type constraints at compile-time decide a protocol, machine code at run-time assumes that protocol).

titanomachy 4y ago

Your work might predate this, but spark/scala now has good support for typed dataframes (they’re called DataSets).

chrisaycock 4y ago
Does Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Datasets require an explicit type provided by the user. Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.

kthielen 4y ago

Very cool, this approach will inevitably catch on, I think. I built a similar typed PL and compiler for Morgan Stanley back in 2013: https://github.com/morganstanley/hobbes https://github.com/morganstanley/hobbes

chrisaycock 4y ago
I read about Hobbes years ago. The embeddability certainly caught my attention. I wrote Empirical specifically for the use case of wanting to read a CSV file while inferring the type at compile time. Basically, a ton of compile-time function evaluation mixed with type providers.
- kthielen 4y ago
  Yeah, I had this type constraint system for doing that kind of thing, so I can load a file at compile time to get the type structure and then compile/evaluate to process data in the file or files. Same thing for network connections between communicating processes (type constraints at compile-time decide a protocol, machine code at run-time assumes that protocol).