4 ms·

That you believe this violates the principle of "do one thing and do it well" indicates a certain "abstraction blindness". Do you really believe that "parse CSV

That you believe this violates the principle of "do one thing and do it well" indicates a certain "abstraction blindness". Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely separate, unrelated things? It seems clear to me in any case that with the right abstraction, those are the same thing composed with different choices of what to do with the parsed data. At a higher level, this seems like a classic Julia versus Python dynamic. In the Python ecosystem there are a dozen different versions of each thing that make different trade-offs and specialize in different ways. When you're picking one, you have to spend a week evaluating all the possible CSV parsers (for example). In the Julia ecosystem, the ideal it to have one really high-quality, fast, flexible implementation that everyone collaborates on to make it great for all kinds of use cases. And the language design and the compiler make this possible, since you can write pretty abstract code and compose it with various types and get a really fast implementation. The annoying and tricky logic of parsing the CSV format only needs to be done once, and reused for all different situations. This isn't done for "macho bragging rights" but because it is a waste of time and effort to implement a dozen half-baked versions of the same thing. When there are a dozen separate implementations of something like this, they also end up disagreeing on their behavior, which causes no end of headaches.

mlthoughts2018 6y ago
> “ Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely separate, unrelated things?“ Yes, absolutely. Very high level design goals are completely different between those use cases and you would be forced to make mutually exclusive trade offs to solve any one of those problems - which will directly lead to concessions in terms of the other use cases. No one generic csv parser can be optimal to all three.
- StefanKarpinski 6y ago
  CSV.jl seems to be proof that you're wrong about that. More concretely, the only difference between parsing to a data frame, parsing to a parquet file, and executing a query over data is a) where you put the data in memory and b) optionally calling some code to operate on the data as you go. It's hard to see how those applications require radically different designs.
  - mlthoughts2018 6y ago
    This is not accurate. CSV.jl optimizes for speed but does not optimize for data transfer to other systems like Spark (which, by necessity, causes worse performance in order to add features around type marshaling). Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark. For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.
drej 6y ago
> In the Python ecosystem there are a dozen different versions of each thing that make different trade-offs and specialize in different ways. When you're picking one, you have to spend a week evaluating all the possible CSV parsers (for example). It saddens me to read this weird statement. It saddens me as I was a rather early adopter in Julia (v0.3.2, if I remember correctly) and really believed its mission. I don't quite understand why various ecosystems cannot peacefully coexist without these untrue and completely avoidable statements.

mlthoughts2018 6y ago

> “ Do you really believe that "parse CSV to a data frame" and "parse CSV to a parquet file" and "parse CSV and apply a query to it in-place" are completely separate, unrelated things?“ Yes, absolutely. Very high level design goals are completely different between those use cases and you would be forced to make mutually exclusive trade offs to solve any one of those problems - which will directly lead to concessions in terms of the other use cases. No one generic csv parser can be optimal to all three.

StefanKarpinski 6y ago
CSV.jl seems to be proof that you're wrong about that. More concretely, the only difference between parsing to a data frame, parsing to a parquet file, and executing a query over data is a) where you put the data in memory and b) optionally calling some code to operate on the data as you go. It's hard to see how those applications require radically different designs.
- mlthoughts2018 6y ago
  This is not accurate. CSV.jl optimizes for speed but does not optimize for data transfer to other systems like Spark (which, by necessity, causes worse performance in order to add features around type marshaling). Design for general case speed is directly at odds with designing either for executing SQL over top of the file (in which case the trade offs are about indexing the data for record-wise or column-wise operations) or type marshaling for an extremely narrow use case like reading into Spark. For example, parsing for speed vs parsing for Spark interop are very likely to handle datetime parsing or null parsing significantly differently.

drej 6y ago

> In the Python ecosystem there are a dozen different versions of each thing that make different trade-offs and specialize in different ways. When you're picking one, you have to spend a week evaluating all the possible CSV parsers (for example). It saddens me to read this weird statement. It saddens me as I was a rather early adopter in Julia (v0.3.2, if I remember correctly) and really believed its mission. I don't quite understand why various ecosystems cannot peacefully coexist without these untrue and completely avoidable statements.