7 ms·
Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.
by zephyrfalcon 10y ago
Python's csv module uses an internal module _csv which is written in C. So I'm not sure it's all that surprising that a Go implementation is a bit slower.
- chrisper 10y agoYou should contribute this knowledge to the GitHub issue.
- jlarocco 10y agoA good idea, but I think it's common knowledge. Lots of modules in the stdlib are written in C.
- chrisper 10y agoWell, it was news to me. Maybe because I only use Python casually.
- baq 10y agoultimately it's of academic interest - the end user doesn't care what precisely is going on under the hood.
- chrisper 10y agoI thought it is rather part of the solution/answer to the issue. It is expected to be slower, so it's not necessarily broken. Kind of it's a feature, not a bug.
- 0xFFC 10y agoHow about Java? It is quite funny Java version is much faster than Python version even when Python version does use C ? Something fishy going on.
- Anderkent 10y agoThe java one doesn't do as much as the python one: `line.split(',')` doesn't handle quoted commas, escape characters, different csv separators, etc.
- andrepd 10y agoYeah, calling from python introduces a layer of inefficiency.
- justin66 10y agoIt's as if there's more to software quality than the choice of tools.
- jerf 10y agoThere's nothing mysterious about the Java version... that's not a CSV parser. The other two things are truly CSV parsers. (Inasmuch as there is such a thing for such an ill-defined format. (No, the RFC is not determinative.)) It's easy to be faster if you do fundamentally less. Not necessarily wrong, depending on your task, but it's not comparable.
- __s 10y agoThe java source is using str.split(',') which is completely different than Python's offering of dialect/delimiter/etc https://github.com/python/cpython/blob/master/Lib/csv.py#L24 https://github.com/python/cpython/blob/master/Lib/csv.py#L24
- nradov 10y agoThe Java code is defective. It's not checking for double quotes. The CSV format allows for commas inside column values by surrounding with double quotes, and then you can also put double quotes within such values by escaping them as double double quotes. Fix those defects and the Java code will be a little slower. With modern JVMs, Java can occasionally actually be faster than native compiled languages due to dynamic optimization at runtime.
- Cyph0n 10y agoI don't think the answer is that simple. Perhaps there's a key optimization that the Go team didn't consider.
- aartur 10y agoI run the benchmark using PyPy (which doesn't have this C extension) and got a result about 20% slower compared to CPython (ie. still faster than Go). EDIT. I also did a funny thing and replaced the CPython C _csv.so extensions with pure Python version _csv.py, from PyPy. It run about 80 (eighty) times slower. It shows what wonders does JIT do (at least to some code).
- jjawssd 10y agoWould be a great experiment to Cythonize PyPy's _csv.so
- masklinn 10y agoThat sounds completely worthless, PyPy doesn't need a Cython version, and its library is a Python version of CPython's native csv module.
- chrisseaton 10y agoPyPy doesn't have a _csv.so - that was the main point of the comment you replied to.
- rdtsc 10y agoBut not sure if it matters. Go is free to use a C module to load csv files as well. It can use assembly or other tricks as well perhaps.
- weberc2 10y agoThere's a performance cost to calling between c and Go, and sharing memory between the two makes for hard to predict GC behavior. I doubt it would be faster than a pure Go implementation.