5 ms·
> “ Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely sep
by mlthoughts2018 6y ago
> “ Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely separate, unrelated things?“
Yes, absolutely. Very high level design goals are completely different between those use cases and you would be forced to make mutually exclusive trade offs to solve any one of those problems - which will directly lead to concessions in terms of the other use cases. No one generic csv parser can be optimal to all three.
- StefanKarpinski 6y agoCSV.jl seems to be proof that you're wrong about that. More concretely, the only difference between parsing to a data frame, parsing to a parquet file, and executing a query over data is a) where you put the data in memory and b) optionally calling some code to operate on the data as you go. It's hard to see how those applications require radically different designs.
- mlthoughts2018 6y agoThis is not accurate. CSV.jl optimizes for speed but does not optimize for data transfer to other systems like Spark (which, by necessity, causes worse performance in order to add features around type marshaling). Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark. For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.