8 ms·

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module)

by dmw_ng 6y ago

There is a ton of conflation in this post, yesterdays post, and the comments. * Comparing parallel implementations (Julia) to single thread (Python csv module). This is still quite relevant because in places like Python, it is impossible to parallelize the csv module without essentially updating its native implementation. That's not true in other places, where some simple wrapper code may be all required to get that "10x faster" number. * Comparing concrete object decoding (Python csv module) to array decoding (PyArrow). Producing concrete objects representing every row (e.g. tuples of strings) will always be slower because of the pressure it puts on the heap and memory. Storing 10 million rows with 10 cols of 8-byte floats in an array might only generate 800 MB of memory bandwidth, the equivalent concrete list-of-floats PyObjects would come out at 6160 MB bandwidth and a ton of CPU burned in the allocator. There are use cases where either representation is preferred, and using e.g. PyArrow's parser then simply iterating the result as concrete objects, worst case the result will be slower than directly decoding to PyObject to begin with. * Lumping floating point decoding (a problem with large performance-correctness tradeoffs) in with CSV parsing. It's hard to decode floats both quickly and precisely, it's also impractical to describe in the context of a comparison of CSV parsers which language/implementation might decode floats better and why that is better.

spockz 6y ago
So what about using Flatbuffers[1] to avoid memory allocation of the data and only have the structure allocated additionally? [1]: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/
- secondcoming 6y ago
  Binary will always beat the pants off human-readable formats. I doubt Excel can export to Flatbuffer, Protobuf, MessagePack, Parquet, etc. So CSV is pretty much here to stay, even though correctly parsing it is more difficult that it seems. To gain even from parallelism, make all your rows a fixed size. Arbitrary length fields aren't your friend if you want max speed.
liuliu 6y ago
For 1), it is simple, but not trivial to parallelize csv parsing correctly, hence the yesterday’s post. Otherwise, agree on 2 and 3. Object wrapping, number decoding can be slow on the whole “file to dataframe object” path, but trivially parallelizable.
nextos 6y ago
I would also say, for those not familiar with Julia, that the main advantage of Julia vs Python or R is that you can write performant code in Julia that will be fast enough for most scenarios. For example, there are efforts to write a pure BLAS in Julia that is still performant [1]. If you are into numerical computing, you will quickly understand this is crazy cool. A consequence of that is composability. Most libraries are pure Julia, small, and play nicely with each other. [1] https://discourse.julialang.org/t/11634 https://discourse.julialang.org/t/11634
- mlthoughts2018 6y ago
  This is also very easy to achieve in Python by using Cython to selectively optimize code. That way for the 99% of code where such optimizations are meaningless and their requirement for static typing is a liability, you can ignore them, and only focus on the 1% of use cases where it matters.
  - Rotareti 6y ago
    > This is also very easy to achieve in Python by using Cython to selectively optimize code There is also mypyc [0] coming up on the horizon. https://github.com/python/mypy/tree/master/mypyc https://github.com/python/mypy/tree/master/mypyc
  - jakobnissen 6y ago
    ... but Cython is not Python, is it? The argument that "Python can do it, you just have to use something other than Python) is very weird to me. I began looking into Julia because I was forced into Cython during a project and I loathed having to add a separate, statically compiled section of my code that did not integrate well with the rest of Python (e.g. no stack traces through Cython), and kept breaking (because of some linker issues in Conda)
    - mlthoughts2018 6y ago
      Cython is a superset of Python. If it’s valid Python, it’s valid Cython. Cython supports the full range of Exceptions and Python stack traces, and you can annotate on a per-function basis whether you want this in Cython. Managing Cython implementations as separate extension modules is a feature, not a bug - it separates concerns and allows only spending any effort on type system constructs in the small amount of cases where those optimizations will help you. Cython is extremely easy to compile and standard Python packaging tools work easily with it out of the box, conda included.
StefanKarpinski 6y ago
You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Regarding your specific points: 1. Yes, it's possible to rig up some way of parsing in parallel in Python, like splitting the CSV file into multiple files and then parsing them in separate processes and then joining that. But no matter how you do it, you have to jump through extra hoops to do it. And that is not something most users do. 2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? Most Python users use Pandas and the CSV parsing that it provides. If that changes, then it would be good to do a new benchmark. 3. Yes, floating-point decoding is hard, but Julia's is fully accurate, so if it's also faster than Python's how is it an unfair comparison? If Julia was cheating by doing imprecise float parsing, sure, that would be a valid complaint, but it's not. These "conflations" sound suspiciously like excuses. If float parsing is slow, make it faster. If concrete object decoding is slow, stop doing that. If threading would be faster, bite the bullet and use threading. If the language doesn't provide good support for these things, then maybe the language is at fault.
- mlthoughts2018 6y ago
  A typical user will rarely ever use the built in CSV library in Python, and would choose among the many third party options for much greater performance, ease of use, parity with rectangular data structure tools like pandas, and so forth. Comparing anything against the built-in CSV reader in Python is pretty much an exercise in nothing. In 20 years of programming Python professionally, I have used the built in CSV parser about 5 total times, and all were use cases where 10-20x improvement of CSV parser performance did not matter whatsoever to the application.
  - StefanKarpinski 6y ago
    The fact that the standard CSV reader is not good enough to actually use except in toy applications is itself an issue, frankly.
anakaine 6y ago
I'm surprised to have not seen a mention of the python library Dask in this thread. Whilst I agree that reading CSVs in Python could be a lot quicker, most people do tend to use third party libraries to do so. Dask is able to multi-core multi-thread almost all operations, including reading in a CSV. You can go from a Dask data frame to a pandas dataframe.

by dmw_ng 6y ago

spockz 6y ago

So what about using Flatbuffers[1] to avoid memory allocation of the data and only have the structure allocated additionally? [1]: https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/

secondcoming 6y ago
Binary will always beat the pants off human-readable formats. I doubt Excel can export to Flatbuffer, Protobuf, MessagePack, Parquet, etc. So CSV is pretty much here to stay, even though correctly parsing it is more difficult that it seems. To gain even from parallelism, make all your rows a fixed size. Arbitrary length fields aren't your friend if you want max speed.

liuliu 6y ago

For 1), it is simple, but not trivial to parallelize csv parsing correctly, hence the yesterday’s post. Otherwise, agree on 2 and 3. Object wrapping, number decoding can be slow on the whole “file to dataframe object” path, but trivially parallelizable.

nextos 6y ago

I would also say, for those not familiar with Julia, that the main advantage of Julia vs Python or R is that you can write performant code in Julia that will be fast enough for most scenarios. For example, there are efforts to write a pure BLAS in Julia that is still performant [1]. If you are into numerical computing, you will quickly understand this is crazy cool. A consequence of that is composability. Most libraries are pure Julia, small, and play nicely with each other. [1] https://discourse.julialang.org/t/11634 https://discourse.julialang.org/t/11634

mlthoughts2018 6y ago
This is also very easy to achieve in Python by using Cython to selectively optimize code. That way for the 99% of code where such optimizations are meaningless and their requirement for static typing is a liability, you can ignore them, and only focus on the 1% of use cases where it matters.
- Rotareti 6y ago
  > This is also very easy to achieve in Python by using Cython to selectively optimize code There is also mypyc [0] coming up on the horizon. https://github.com/python/mypy/tree/master/mypyc https://github.com/python/mypy/tree/master/mypyc
- jakobnissen 6y ago
  ... but Cython is not Python, is it? The argument that "Python can do it, you just have to use something other than Python) is very weird to me. I began looking into Julia because I was forced into Cython during a project and I loathed having to add a separate, statically compiled section of my code that did not integrate well with the rest of Python (e.g. no stack traces through Cython), and kept breaking (because of some linker issues in Conda)
  - mlthoughts2018 6y ago
    Cython is a superset of Python. If it’s valid Python, it’s valid Cython. Cython supports the full range of Exceptions and Python stack traces, and you can annotate on a per-function basis whether you want this in Cython. Managing Cython implementations as separate extension modules is a feature, not a bug - it separates concerns and allows only spending any effort on type system constructs in the small amount of cases where those optimizations will help you. Cython is extremely easy to compile and standard Python packaging tools work easily with it out of the box, conda included.

StefanKarpinski 6y ago

You may be missing the point of the comparison, which is this: to compare the performance of the de facto standard CSV parsers of each system when used the way that a typical user would use them, i.e. to parse a CSV file into a data frame. Regarding your specific points: 1. Yes, it's possible to rig up some way of parsing in parallel in Python, like splitting the CSV file into multiple files and then parsing them in separate processes and then joining that. But no matter how you do it, you have to jump through extra hoops to do it. And that is not something most users do. 2. I don't entirely follow this point. Perhaps using PyArrow's parser would be faster than what is timed here, but is that what the typical Python data science user would do? Most Python users use Pandas and the CSV parsing that it provides. If that changes, then it would be good to do a new benchmark. 3. Yes, floating-point decoding is hard, but Julia's is fully accurate, so if it's also faster than Python's how is it an unfair comparison? If Julia was cheating by doing imprecise float parsing, sure, that would be a valid complaint, but it's not. These "conflations" sound suspiciously like excuses. If float parsing is slow, make it faster. If concrete object decoding is slow, stop doing that. If threading would be faster, bite the bullet and use threading. If the language doesn't provide good support for these things, then maybe the language is at fault.

mlthoughts2018 6y ago
A typical user will rarely ever use the built in CSV library in Python, and would choose among the many third party options for much greater performance, ease of use, parity with rectangular data structure tools like pandas, and so forth. Comparing anything against the built-in CSV reader in Python is pretty much an exercise in nothing. In 20 years of programming Python professionally, I have used the built in CSV parser about 5 total times, and all were use cases where 10-20x improvement of CSV parser performance did not matter whatsoever to the application.
- StefanKarpinski 6y ago
  The fact that the standard CSV reader is not good enough to actually use except in toy applications is itself an issue, frankly.

anakaine 6y ago

I'm surprised to have not seen a mention of the python library Dask in this thread. Whilst I agree that reading CSVs in Python could be a lot quicker, most people do tend to use third party libraries to do so. Dask is able to multi-core multi-thread almost all operations, including reading in a CSV. You can go from a Dask data frame to a pandas dataframe.