3 ms·
Does Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Dat
by chrisaycock 4y ago
Does Spark's Dataset have something like the "inferSchema" option when reading a CSV file? Everything I've read in Spark's documentation makes me think that Datasets require an explicit type provided by the user.
Empirical can infer a CSV's schema at compile time. If the file path can be determined at compile time, then the Empirical compiler will sample the CSV file and determine an appropriate type before the user's code is ever run. No need for an explicit type from the user, and yet we still have static typing.