6 ms·

You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way

You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Regarding your specific points: 1. Yes, it's possible to rig up some way of parsing in parallel in Python, like splitting the CSV file into multiple files and then parsing them in separate processes and then joining that. But no matter how you do it, you have to jump through extra hoops to do it. And that is not something most users do. 2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? Most Python users use Pandas and the CSV parsing that it provides. If that changes, then it would be good to do a new benchmark. 3. Yes, floating-point decoding is hard, but Julia's is fully accurate, so if it's also faster than Python's how is it an unfair comparison? If Julia was cheating by doing imprecise float parsing, sure, that would be a valid complaint, but it's not. These "conflations" sound suspiciously like excuses. If float parsing is slow, make it faster. If concrete object decoding is slow, stop doing that. If threading would be faster, bite the bullet and use threading. If the language doesn't provide good support for these things, then maybe the language is at fault.

mlthoughts2018 6y ago
A typical user will rarely ever use the built in CSV library in Python, and would choose among the many third party options for much greater performance, ease of use, parity with rectangular data structure tools like pandas, and so forth. Comparing anything against the built-in CSV reader in Python is pretty much an exercise in nothing. In 20 years of programming Python professionally, I have used the built in CSV parser about 5 total times, and all were use cases where 10-20x improvement of CSV parser performance did not matter whatsoever to the application.
- StefanKarpinski 6y ago
  The fact that the standard CSV reader is not good enough to actually use except in toy applications is itself an issue, frankly.
  - mlthoughts2018 6y ago
    I disagree completely. It’s perfectly good enough for non-specialized use cases, which is exactly right for a stdlib offering. For anything else, factor it out as separate third party options (even if maintained by PSF itself) so users only install what they need for their special application. One person may install something that’s blazingly fast. Someone else may install something because it can convert the data to an in memory parquet format, someone else may use something that can virtualize SQL queries over top of disk-backed csvs. They should all be different for the nuance of each application.
    - StefanKarpinski 6y ago
      Interestingly, CSV.jl can do all of those. This seems like a classic case of lack of composability in Python: you need a different CSV parser for each and every application even though the core logic of parsing the CSV format is the same. This is because anything fast has to be implemented in very concrete, specialized C code. In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.
      - drej 6y ago
        Lack of composability? What? Of course you can use one parser for everything, what’s stopping you from doing that. Separating IO from business logic and composing the two is a pretty standard practice. Not sure I follow the argument here.
        coliveira 6y ago
        The argument is simple: Python is by design made so that if you want things to go fast then you need to implement it in C. This creates a constant need to use specialized libraries (such as pandas), where in other languages they can simply do a better job in optimizing the standard library.
        6y ago
dmw_ng 6y ago
Regarding 2), code like "for row in pandas.DataFrame(...).iterrows():" where the DataFrame was populated by some array-based reader is creating an indirection. Most good pandas code of course doesn't look like this, but it's regularly unavoidable, hence "There are use cases where either representation is preferred". Without specifying our use case, it is meaningless to talk about "faster" or "slower". For 3) there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs", additionally users do not care only about speed, for a huge number of applications, absolute reproduction of precision may be the difference between a good simulation and data science quackery. We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. Ignoring performance, this is criteria for deciding whether a solution is fit for purpose at all. The overall point otherwise makes sense: most DS folk don't and maybe shouldn't care about this stuff, however this is HN, and elsewhere in the comments there are plenty of examples of people who do.
- StefanKarpinski 6y ago
  > there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs" You seem to be under the misapprehension that mainstream CSV libraries parse float values with reduced precision in order to go faster. They do not. A decimal floating-point string represents a precise mathematical value and the only acceptable value for a CSV parser to produce is that value correctly rounded to a Float64 (or some other type if requested). Any CSV parser that does anything else is simply incorrect. > We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. You may be referring to a bug mentioned elsewhere that fields like `-123.` with a leading minus sign and a trailing decimal point are misinterpreted as strings. That is a bug, plain and simple. It has nothing to do with float parsing speed/accuracy tradeoffs.
  - deleted 6y ago
    [deleted]
datanecdote 6y ago
> 2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? I am a Python data science user. If data gets big enough such that loading time is a bottleneck, I use parquet files instead of CSV, and PyArrow to load them into pandas. It’s a one line change. The creator of Pandas is now leading the Arrow project. It’s very seamless. Don’t know if I’m typical but that’s me.
- ViralBShah 6y ago
  Perhaps not directly relevant to your point here, but thought it would be interesting to anyone following along. Jacob Quinn (karbacca) also has a Julia package for integrating Julia into the Arrow ecosystem: https://github.com/JuliaData/Arrow.jl https://github.com/JuliaData/Arrow.jl
  - datanecdote 6y ago
    Thanks Viral. To be clear, I’m a python user who’s cheering for Julia, because I live the problems of python and do see the potential of Julia as a better path. But unfortunately I’m not prepared to be the early adopter (at least in my day job), and will wait until other, braver users have sanded off the rough edges. God speed and good luck.
    - ViralBShah 6y ago
      That's a completely reasonable viewpoint. Many users of Julia and contributors start out experimenting with it and then end up bringing it into their work when they feel comfortable with it. I hope you will have the same experience one day.
formerly_proven 6y ago
> If float parsing is slow, make it faster. Use a binary data format or hexfloats instead. Decimal floats are a pain and especially pointless if they are being generated from and converted back to IEEE 754.
orbifold 6y ago
I installed CSV.jl and tried it out on the insurance dataset from here "https://support.spatialkey.com/spatialkey-sample-csv-data/ https://support.spatialkey.com/spatialkey-sample-csv-data/". First import took >10s just to precompile and then CSV.File took 5.21s to read the file. A fresh pandas install takes ~10s to import and then reads this file in 81.2 ms ± 1.25 ms. CSV.jl takes 38 ms on a second run so it is twice as fast, however especially to someone that just starts using Julia, these >100x slowdowns constantly happen. Especially when using libraries that heavily rely on multiple dispatch specialisation for performance.
- ViralBShah 6y ago
  I tried out the dataset you mention. I use the latest Julia 1.5.2 and CSV.jl 0.7.7 and load the CSV package in 1s, read the file first time in 5s and then in 0.03s the second time. The precompile time you mention is only for the very first time you download a new version of a package, which is then cached for subsequent sessions. Your generalization from a one time precompilation of the CSV.jl package to >100x slowdowns happening constantly is simply not true. julia> @time using CSV 1.161851 seconds (2.05 M allocations: 131.757 MiB, 0.46% gc time) julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 4.884074 seconds (6.09 M allocations: 311.878 MiB, 1.90% gc time) julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 0.030916 seconds (147.27 k allocations: 11.361 MiB)
  - orbifold 6y ago
    So I want to plot the data I just loaded. Since I wrote the comment I had closed the Julia window (not to make a point, btw.) julia> import CSV (takes ~1-2 seconds) julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...) I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the dependency story in Python is equally insane, so whatever. As I'm typing this, I'm waiting for Gadfly to precompile. julia> @time import Gadfly (104.460307 seconds) python> import matplotlib.pyplot as plt (~10s if you are unlucky and it has to generate the font cache) Maybe it's a good idea to do a scatter plot of two of the fields? Let's find out julia> @time Gadfly.plot(x=data.eq_site_limit, y=data.hu_site_limit) 0.320751 seconds but that number is not accurate, in fact it took so long for the browser to open the result, that I had time to check the documentation on backends and see whether I'm missing something. python> %time plt.scatter(data['eq_site_limit'], data['hu_site_limit']) 41.9 ms, a bit more for the window to open Turns out that is not a useful plot... maybe a histogram? julia> @time plot(x=data.eq_site_limit, Geom.histogram(bincount=10)) 0.042802 seconds It in fact again takes several seconds until the browser shows a plot. python> %time plt.hist(data["eq_site_limit"]) ~100ms and the window with the plot opens instantly.
coldtea 6y ago
>You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Well, in Python typically one would use pandas "to parse a CSV file into a data frame", not the standard csv parser (which doesn't deal with data frames anyway), no?
- cbkeller 6y ago
  That’s why Pandas read_csv is what they benchmarked
  - coldtea 6y ago
    Ah, alrighty then.

mlthoughts2018 6y ago

A typical user will rarely ever use the built in CSV library in Python, and would choose among the many third party options for much greater performance, ease of use, parity with rectangular data structure tools like pandas, and so forth. Comparing anything against the built-in CSV reader in Python is pretty much an exercise in nothing. In 20 years of programming Python professionally, I have used the built in CSV parser about 5 total times, and all were use cases where 10-20x improvement of CSV parser performance did not matter whatsoever to the application.

StefanKarpinski 6y ago
The fact that the standard CSV reader is not good enough to actually use except in toy applications is itself an issue, frankly.
- mlthoughts2018 6y ago
  I disagree completely. It’s perfectly good enough for non-specialized use cases, which is exactly right for a stdlib offering. For anything else, factor it out as separate third party options (even if maintained by PSF itself) so users only install what they need for their special application. One person may install something that’s blazingly fast. Someone else may install something because it can convert the data to an in memory parquet format, someone else may use something that can virtualize SQL queries over top of disk-backed csvs. They should all be different for the nuance of each application.
  - StefanKarpinski 6y ago
    Interestingly, CSV.jl can do all of those. This seems like a classic case of lack of composability in Python: you need a different CSV parser for each and every application even though the core logic of parsing the CSV format is the same. This is because anything fast has to be implemented in very concrete, specialized C code. In Julia, you can write something generic and let the compiler specialize it for you, so you get many different ways of applying the same thing for free.
    - drej 6y ago
      Lack of composability? What? Of course you can use one parser for everything, what’s stopping you from doing that. Separating IO from business logic and composing the two is a pretty standard practice. Not sure I follow the argument here.
      - coliveira 6y ago
        The argument is simple: Python is by design made so that if you want things to go fast then you need to implement it in C. This creates a constant need to use specialized libraries (such as pandas), where in other languages they can simply do a better job in optimizing the standard library.
        6y ago

dmw_ng 6y ago

Regarding 2), code like "for row in pandas.DataFrame(...).iterrows():" where the DataFrame was populated by some array-based reader is creating an indirection. Most good pandas code of course doesn't look like this, but it's regularly unavoidable, hence "There are use cases where either representation is preferred". Without specifying our use case, it is meaningless to talk about "faster" or "slower". For 3) there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs", additionally users do not care only about speed, for a huge number of applications, absolute reproduction of precision may be the difference between a good simulation and data science quackery. We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. Ignoring performance, this is criteria for deciding whether a solution is fit for purpose at all. The overall point otherwise makes sense: most DS folk don't and maybe shouldn't care about this stuff, however this is HN, and elsewhere in the comments there are plenty of examples of people who do.

StefanKarpinski 6y ago
> there inherently is no optimal solution to "a problem with large performance-correctness tradeoffs" You seem to be under the misapprehension that mainstream CSV libraries parse float values with reduced precision in order to go faster. They do not. A decimal floating-point string represents a precise mathematical value and the only acceptable value for a CSV parser to produce is that value correctly rounded to a Float64 (or some other type if requested). Any CSV parser that does anything else is simply incorrect. > We saw elsewhere in the thread that Julia's float parsing code at least presently handles certain syntaxes incorrectly. You may be referring to a bug mentioned elsewhere that fields like `-123.` with a leading minus sign and a trailing decimal point are misinterpreted as strings. That is a bug, plain and simple. It has nothing to do with float parsing speed/accuracy tradeoffs.
- deleted 6y ago
  [deleted]

datanecdote 6y ago

> 2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? I am a Python data science user. If data gets big enough such that loading time is a bottleneck, I use parquet files instead of CSV, and PyArrow to load them into pandas. It’s a one line change. The creator of Pandas is now leading the Arrow project. It’s very seamless. Don’t know if I’m typical but that’s me.

ViralBShah 6y ago
Perhaps not directly relevant to your point here, but thought it would be interesting to anyone following along. Jacob Quinn (karbacca) also has a Julia package for integrating Julia into the Arrow ecosystem: https://github.com/JuliaData/Arrow.jl https://github.com/JuliaData/Arrow.jl
- datanecdote 6y ago
  Thanks Viral. To be clear, I’m a python user who’s cheering for Julia, because I live the problems of python and do see the potential of Julia as a better path. But unfortunately I’m not prepared to be the early adopter (at least in my day job), and will wait until other, braver users have sanded off the rough edges. God speed and good luck.
  - ViralBShah 6y ago
    That's a completely reasonable viewpoint. Many users of Julia and contributors start out experimenting with it and then end up bringing it into their work when they feel comfortable with it. I hope you will have the same experience one day.

formerly_proven 6y ago

> If float parsing is slow, make it faster. Use a binary data format or hexfloats instead. Decimal floats are a pain and especially pointless if they are being generated from and converted back to IEEE 754.

orbifold 6y ago

I installed CSV.jl and tried it out on the insurance dataset from here "https://support.spatialkey.com/spatialkey-sample-csv-data/ https://support.spatialkey.com/spatialkey-sample-csv-data/". First import took >10s just to precompile and then CSV.File took 5.21s to read the file. A fresh pandas install takes ~10s to import and then reads this file in 81.2 ms ± 1.25 ms. CSV.jl takes 38 ms on a second run so it is twice as fast, however especially to someone that just starts using Julia, these >100x slowdowns constantly happen. Especially when using libraries that heavily rely on multiple dispatch specialisation for performance.

ViralBShah 6y ago
I tried out the dataset you mention. I use the latest Julia 1.5.2 and CSV.jl 0.7.7 and load the CSV package in 1s, read the file first time in 5s and then in 0.03s the second time. The precompile time you mention is only for the very first time you download a new version of a package, which is then cached for subsequent sessions. Your generalization from a one time precompilation of the CSV.jl package to >100x slowdowns happening constantly is simply not true. julia> @time using CSV 1.161851 seconds (2.05 M allocations: 131.757 MiB, 0.46% gc time) julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 4.884074 seconds (6.09 M allocations: 311.878 MiB, 1.90% gc time) julia> @time CSV.read("/Users/viral/Downloads/FL_insurance_sample.csv"); 0.030916 seconds (147.27 k allocations: 11.361 MiB)
- orbifold 6y ago
  So I want to plot the data I just loaded. Since I wrote the comment I had closed the Julia window (not to make a point, btw.) julia> import CSV (takes ~1-2 seconds) julia> data = CSV.File("/User/orbifold/Downloads/FL_insurance_sample.csv) (takes seconds again...) I want to explore the data. I vaguely remember there is Gadfly to do that. For some reason it depends on FFTW and a whole bunch of other packages, but the dependency story in Python is equally insane, so whatever. As I'm typing this, I'm waiting for Gadfly to precompile. julia> @time import Gadfly (104.460307 seconds) python> import matplotlib.pyplot as plt (~10s if you are unlucky and it has to generate the font cache) Maybe it's a good idea to do a scatter plot of two of the fields? Let's find out julia> @time Gadfly.plot(x=data.eq_site_limit, y=data.hu_site_limit) 0.320751 seconds but that number is not accurate, in fact it took so long for the browser to open the result, that I had time to check the documentation on backends and see whether I'm missing something. python> %time plt.scatter(data['eq_site_limit'], data['hu_site_limit']) 41.9 ms, a bit more for the window to open Turns out that is not a useful plot... maybe a histogram? julia> @time plot(x=data.eq_site_limit, Geom.histogram(bincount=10)) 0.042802 seconds It in fact again takes several seconds until the browser shows a plot. python> %time plt.hist(data["eq_site_limit"]) ~100ms and the window with the plot opens instantly.

coldtea 6y ago

>You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Well, in Python typically one would use pandas "to parse a CSV file into a data frame", not the standard csv parser (which doesn't deal with data frames anyway), no?

cbkeller 6y ago
That’s why Pandas read_csv is what they benchmarked
- coldtea 6y ago
  Ah, alrighty then.