3 ms·
Your work might predate this, but spark/scala now has good support for typed dataframes (they’re called DataSets).
by titanomachy 4y ago
Your work might predate this, but spark/scala now has good support for typed dataframes (they’re called DataSets).
- chrisaycock 4y agoDoes Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Datasets require an explicit type provided by the user. Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.