4 ms·

A typical user will rarely ever use the built in CSV library in Python, and would choose among the many third party options for much greater performance, ease o

by mlthoughts2018 6y ago

A typical user will rarely ever use the built in CSV library in Python, and would choose among the many third party options for much greater performance, ease of use, parity with rectangular data structure tools like pandas, and so forth. Comparing anything against the built-in CSV reader in Python is pretty much an exercise in nothing. In 20 years of programming Python professionally, I have used the built in CSV parser about 5 total times, and all were use cases where 10-20x improvement of CSV parser performance did not matter whatsoever to the application.

StefanKarpinski 6y ago
The fact that the standard CSV reader is not good enough to actually use except in toy applications is itself an issue, frankly.
- mlthoughts2018 6y ago
  I disagree completely. It’s perfectly good enough for non-specialized use cases, which is exactly right for a stdlib offering. For anything else, factor it out as separate third party options (even if maintained by PSF itself) so users only install what they need for their special application. One person may install something that’s blazingly fast. Someone else may install something because it can convert the data to an in memory parquet format, someone else may use something that can virtualize SQL queries over top of disk-backed csvs. They should all be different for the nuance of each application.
  - StefanKarpinski 6y ago
    Interestingly, CSV.jl can do all of those. This seems like a classic case of lack of composability in Python: you need a different CSV parser for each and every application even though the core logic of parsing the CSV format is the same. This is because anything fast has to be implemented in very concrete, specialized C code. In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.
    - drej 6y ago
      Lack of composability? What? Of course you can use one parser for everything, what’s stopping you from doing that. Separating IO from business logic and composing the two is a pretty standard practice. Not sure I follow the argument here.
      - coliveira 6y ago
        The argument is simple: Python is by design made so that if you want things to go fast then you need to implement it in C. This creates a constant need to use specialized libraries (such as pandas), where in other languages they can simply do a better job in optimizing the standard library.
        mlthoughts2018 6y ago
        This is untrue, you can compartmentalize your C implementations using tools like Cython. The fact that you write these as isolated special implementations is a good thing and not a deficiency of generics. This way it can be gradual. You only target certain functions or modules for optimized C implementation, you don’t waste static typing overhead or compliance to a generic interface on the 99% of the code that will have no material gain from any optimization. It’s similar with gradual typing in Python as well. Annotate what you need, omit annotations for what you don’t. Implement an extension module or multiple dispatch for what you need, use plain Python for everything else.
    - mlthoughts2018 6y ago
      > “ Interestingly, CSV.jl can do all of those.“ That sounds like a poorly designed library to me. Instead of doing one thing well, it forces users to consume something meant as a jack of all trades, which may not meet their use case. I wish such a thing wasn’t forced in a standard package - rather split into different ones. Then I can easily compose them according to my needs, instead of having the stdlib force feed me some set of tradeoffs I might not want. I’m sure these extra third party libraries exist or will exist in Julia just as they do in Python and users will have lots of good options - but what a shame for people to take a needlessly myopic view that core libraries need to be obsessed either with performance or with breadth of use case coverage coupled into one library, which seems to purely be for macho bragging rights or something?
- drej 6y ago
  Except that it can. I hardly ever use something other than stdlib’s csv. It’s a perfectly good csv parser for production use.
  - cbkeller 6y ago
    Doesn't that validate the applicability of the benchmark then? GP said: > A typical user will rarely ever use the built in CSV library in Python as a criticism of the benchmark, but if a typical user does use stdlib’s csv after all, then it seems like your disagreement is perhaps with GP and not with Stefan.
    - mlthoughts2018 6y ago
      No you’re mixing up two things. 1. Python’s std csv parser is very fast & versatile, perfectly good for a range of use cases that don’t have unusual needs. 2. For any production use cases whose needs aren’t met by the std csv parser, nobody would use it and nobody has to - plenty of other options exist (rightfully) as separate packages that allow the user to choose a different point of trade-off properties if they don’t like the std module’s defaults. Nothing says the std isn’t good enough for production - it is. Just most people will use something application-specific and they have tons of options.
    - giantrobot 6y ago
      It does validate the benchmark but that doesn't make the benchmark particularly useful. In both Python and Julia if loading your data from CSV is a bottleneck you'll do something else. If it's not a bottleneck it doesn't matter. If you've got a Python code base it would cost far more in terms of effort and dollars to convert it all to Julia than to just find a better way to load data. If you've got a green field project and CSV reading is a big portion of the task load then it makes sense to pick Julia I guess.

StefanKarpinski 6y ago

The fact that the standard CSV reader is not good enough to actually use except in toy applications is itself an issue, frankly.

mlthoughts2018 6y ago
I disagree completely. It’s perfectly good enough for non-specialized use cases, which is exactly right for a stdlib offering. For anything else, factor it out as separate third party options (even if maintained by PSF itself) so users only install what they need for their special application. One person may install something that’s blazingly fast. Someone else may install something because it can convert the data to an in memory parquet format, someone else may use something that can virtualize SQL queries over top of disk-backed csvs. They should all be different for the nuance of each application.
- StefanKarpinski 6y ago
  Interestingly, CSV.jl can do all of those. This seems like a classic case of lack of composability in Python: you need a different CSV parser for each and every application even though the core logic of parsing the CSV format is the same. This is because anything fast has to be implemented in very concrete, specialized C code. In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.
  - drej 6y ago
    Lack of composability? What? Of course you can use one parser for everything, what’s stopping you from doing that. Separating IO from business logic and composing the two is a pretty standard practice. Not sure I follow the argument here.
    - coliveira 6y ago
      The argument is simple: Python is by design made so that if you want things to go fast then you need to implement it in C. This creates a constant need to use specialized libraries (such as pandas), where in other languages they can simply do a better job in optimizing the standard library.
      - mlthoughts2018 6y ago
        This is untrue, you can compartmentalize your C implementations using tools like Cython. The fact that you write these as isolated special implementations is a good thing and not a deficiency of generics. This way it can be gradual. You only target certain functions or modules for optimized C implementation, you don’t waste static typing overhead or compliance to a generic interface on the 99% of the code that will have no material gain from any optimization. It’s similar with gradual typing in Python as well. Annotate what you need, omit annotations for what you don’t. Implement an extension module or multiple dispatch for what you need, use plain Python for everything else.
  - mlthoughts2018 6y ago
    > “ Interestingly, CSV.jl can do all of those.“ That sounds like a poorly designed library to me. Instead of doing one thing well, it forces users to consume something meant as a jack of all trades, which may not meet their use case. I wish such a thing wasn’t forced in a standard package - rather split into different ones. Then I can easily compose them according to my needs, instead of having the stdlib force feed me some set of tradeoffs I might not want. I’m sure these extra third party libraries exist or will exist in Julia just as they do in Python and users will have lots of good options - but what a shame for people to take a needlessly myopic view that core libraries need to be obsessed either with performance or with breadth of use case coverage coupled into one library, which seems to purely be for macho bragging rights or something?
drej 6y ago
Except that it can. I hardly ever use something other than stdlib’s csv. It’s a perfectly good csv parser for production use.
- cbkeller 6y ago
  Doesn't that validate the applicability of the benchmark then? GP said: > A typical user will rarely ever use the built in CSV library in Python as a criticism of the benchmark, but if a typical user does use stdlib’s csv after all, then it seems like your disagreement is perhaps with GP and not with Stefan.
  - mlthoughts2018 6y ago
    No you’re mixing up two things. 1. Python’s std csv parser is very fast & versatile, perfectly good for a range of use cases that don’t have unusual needs. 2. For any production use cases whose needs aren’t met by the std csv parser, nobody would use it and nobody has to - plenty of other options exist (rightfully) as separate packages that allow the user to choose a different point of trade-off properties if they don’t like the std module’s defaults. Nothing says the std isn’t good enough for production - it is. Just most people will use something application-specific and they have tons of options.
  - giantrobot 6y ago
    It does validate the benchmark but that doesn't make the benchmark particularly useful. In both Python and Julia if loading your data from CSV is a bottleneck you'll do something else. If it's not a bottleneck it doesn't matter. If you've got a Python code base it would cost far more in terms of effort and dollars to convert it all to Julia than to just find a better way to load data. If you've got a green field project and CSV reading is a big portion of the task load then it makes sense to pick Julia I guess.