4 ms·

The Pandas CSV reader (Python) is one of the best and most versatile implementations out there. Also very fast. https://pandas.pydata.org/pandas-docs/stable/re

by joshlk 6y ago

The Pandas CSV reader (Python) is one of the best and most versatile implementations out there. Also very fast. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

andreareina 6y ago
N.B. Floats won't rountdrip by default (you need to call `read_csv(..., float_format="round_trip")`)
nojito 6y ago
Disagree. Data.table’s fread is leagues ahead of pandas. https://www.rdocumentation.org/packages/data.table/versions/1.12.8/topics/fread https://www.rdocumentation.org/packages/data.table/versions/... Fread has automatic footer detection and has automatic skip logic to help parse out mangled headers in some csvs.
- vharuck 6y ago
  fread can also accept a shell command which it will run and use the result as input. I use that to grep the lines I want from files too big for RAM.
  - nojito 6y ago
    Same here! I also use fwrite to send it back to the shell to continue my processing pipeline. https://www.rdocumentation.org/packages/data.table/versions/1.12.8/topics/fwrite https://www.rdocumentation.org/packages/data.table/versions/... It's mind boggling how little fanfare data.table has despite it being the best way to handle tabular data in 2020.
- jsmith99 6y ago
  fread is amazing. Whenever I look at the docs and see all the options for dealing with nested lists (like the within-column-seperator) I always pray never to have need of such a thing.
- jointpdf 6y ago
  Additionally, RStudio has a built in GUI widget for importing .csv (and other) files: https://support.rstudio.com/hc/en-us/articles/218611977-Importing-Data-with-RStudio https://support.rstudio.com/hc/en-us/articles/218611977-Impo.... I find it quite useful for breezing through the common gotchas with importing data, and it’s probably the easiest way for inexperienced programmers (e.g. business analyst types used to working in Excel) to work with .csv.
valera_rozuvan 6y ago
Wow! That's a nifty little library. Just looking at the amount of options going into the main parser routine is mind blowing [1]. And the figuring out of encoding is also complex [2]. Once again, this is just to demonstrate that writing your own CSV parser from scratch is a total waste of time. Just use tools provided by your language. Many languages provide native support for CSV parsing. For example see docs for Microsoft C# [3]. ---------- [1] https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/parsers.py#L530-L587 https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/p... [2] https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/parsers.py#L954-L991 https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/p... [3] https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/how-to-generate-xml-from-csv-files https://docs.microsoft.com/en-us/dotnet/csharp/programming-g...
- tjalfi 6y ago
  string.Split is far from native support. If you don’t want to use third-party libraries then TextFieldParser[0] is part of the .NET framework. I use CsvHelper[1] in my projects. It can do pretty much anything although the newer versions have some dependencies. [0] https://docs.microsoft.com/en-us/dotnet/api/microsoft.visualbasic.fileio.textfieldparser?view=netcore-3.1 https://docs.microsoft.com/en-us/dotnet/api/microsoft.visual... [1] https://joshclose.github.io/CsvHelper/ https://joshclose.github.io/CsvHelper/
VWWHFSfQ 6y ago
Also xsv [0] and the related Rust csv crate [1]. xsv in particular is the best CLI csv tool I've ever used. And I've used them all. [0] https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv [1] https://github.com/BurntSushi/rust-csv https://github.com/BurntSushi/rust-csv
geraldbauer 6y ago
In Rubyland there's the versatile CsvReader library / gem [1] that supports HXL, ARFF, CVY and more "exotic" or modern flavors / dialects among many others. [1]: https://github.com/csvreader/csvreader https://github.com/csvreader/csvreader
apiology 6y ago
I wrote a Python utility to help our folks at our analytics company load/unload/move data among CSV files and SQL databases. Getting the parts that work with CSVs to "just work" when presented with an arbitrary CSV file has been super interesting - even with Pandas at our disposal. https://github.com/bluelabsio/records-mover/blob/d18ec02fdf504efc1512272fffd8bac7b8b2f734/records_mover/records/delimited/sniff.py https://github.com/bluelabsio/records-mover/blob/d18ec02fdf5... I wrote up some of the experience here: https://github.com/bluelabsio/records-mover/blob/master/docs/RECORDS_SPEC.md#hints https://github.com/bluelabsio/records-mover/blob/master/docs...
ken 6y ago
Are we talking "fast", or merely "fast for Python"?
- ganstyles 6y ago
  It sounds like you may not consider anything python can do to be fast, so I'm thinking the latter.
roelschroeven 6y ago
In what ways is it better than the csv module in the Python Standard Library? Or in other words, in which cases is it worth the effort to install pandas just for reading CSV files if your project otherwise doesn't use pandas?
lucb1e 6y ago
The built-in import csv never failed to do everything I needed, what does this Panda thing add?

by joshlk 6y ago

andreareina 6y ago

N.B. Floats won't rountdrip by default (you need to call `read_csv(..., float_format="round_trip")`)

nojito 6y ago

Disagree. Data.table’s fread is leagues ahead of pandas. https://www.rdocumentation.org/packages/data.table/versions/1.12.8/topics/fread https://www.rdocumentation.org/packages/data.table/versions/... Fread has automatic footer detection and has automatic skip logic to help parse out mangled headers in some csvs.

vharuck 6y ago
fread can also accept a shell command which it will run and use the result as input. I use that to grep the lines I want from files too big for RAM.
- nojito 6y ago
  Same here! I also use fwrite to send it back to the shell to continue my processing pipeline. https://www.rdocumentation.org/packages/data.table/versions/1.12.8/topics/fwrite https://www.rdocumentation.org/packages/data.table/versions/... It's mind boggling how little fanfare data.table has despite it being the best way to handle tabular data in 2020.
jsmith99 6y ago
fread is amazing. Whenever I look at the docs and see all the options for dealing with nested lists (like the within-column-seperator) I always pray never to have need of such a thing.
jointpdf 6y ago
Additionally, RStudio has a built in GUI widget for importing .csv (and other) files: https://support.rstudio.com/hc/en-us/articles/218611977-Importing-Data-with-RStudio https://support.rstudio.com/hc/en-us/articles/218611977-Impo.... I find it quite useful for breezing through the common gotchas with importing data, and it’s probably the easiest way for inexperienced programmers (e.g. business analyst types used to working in Excel) to work with .csv.

valera_rozuvan 6y ago

Wow! That's a nifty little library. Just looking at the amount of options going into the main parser routine is mind blowing [1]. And the figuring out of encoding is also complex [2]. Once again, this is just to demonstrate that writing your own CSV parser from scratch is a total waste of time. Just use tools provided by your language. Many languages provide native support for CSV parsing. For example see docs for Microsoft C# [3]. ---------- [1] https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/parsers.py#L530-L587 https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/p... [2] https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/parsers.py#L954-L991 https://github.com/pandas-dev/pandas/blob/v1.0.3/pandas/io/p... [3] https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/linq/how-to-generate-xml-from-csv-files https://docs.microsoft.com/en-us/dotnet/csharp/programming-g...

tjalfi 6y ago
string.Split is far from native support. If you don’t want to use third-party libraries then TextFieldParser[0] is part of the .NET framework. I use CsvHelper[1] in my projects. It can do pretty much anything although the newer versions have some dependencies. [0] https://docs.microsoft.com/en-us/dotnet/api/microsoft.visualbasic.fileio.textfieldparser?view=netcore-3.1 https://docs.microsoft.com/en-us/dotnet/api/microsoft.visual... [1] https://joshclose.github.io/CsvHelper/ https://joshclose.github.io/CsvHelper/

VWWHFSfQ 6y ago

Also xsv [0] and the related Rust csv crate [1]. xsv in particular is the best CLI csv tool I've ever used. And I've used them all. [0] https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv [1] https://github.com/BurntSushi/rust-csv https://github.com/BurntSushi/rust-csv

geraldbauer 6y ago

In Rubyland there's the versatile CsvReader library / gem [1] that supports HXL, ARFF, CVY and more "exotic" or modern flavors / dialects among many others. [1]: https://github.com/csvreader/csvreader https://github.com/csvreader/csvreader

apiology 6y ago

I wrote a Python utility to help our folks at our analytics company load/unload/move data among CSV files and SQL databases. Getting the parts that work with CSVs to "just work" when presented with an arbitrary CSV file has been super interesting - even with Pandas at our disposal. https://github.com/bluelabsio/records-mover/blob/d18ec02fdf504efc1512272fffd8bac7b8b2f734/records_mover/records/delimited/sniff.py https://github.com/bluelabsio/records-mover/blob/d18ec02fdf5... I wrote up some of the experience here: https://github.com/bluelabsio/records-mover/blob/master/docs/RECORDS_SPEC.md#hints https://github.com/bluelabsio/records-mover/blob/master/docs...

ken 6y ago

Are we talking "fast", or merely "fast for Python"?

ganstyles 6y ago
It sounds like you may not consider anything python can do to be fast, so I'm thinking the latter.

roelschroeven 6y ago

In what ways is it better than the csv module in the Python Standard Library? Or in other words, in which cases is it worth the effort to install pandas just for reading CSV files if your project otherwise doesn't use pandas?

lucb1e 6y ago

The built-in import csv never failed to do everything I needed, what does this Panda thing add?