3 ms·
As of Pandas 1.4, you can use the pyarrow engine for reading a csv df = pd.read_csv("large.csv", engine="pyarrow") https://pythonspeed.com/articles/pandas
by philshem 4y ago
As of Pandas 1.4, you can use the pyarrow engine for reading a csv
df = pd.read_csv("large.csv", engine="pyarrow")
https://pythonspeed.com/articles/pandas-read-csv-fast/ https://pythonspeed.com/articles/pandas-read-csv-fast/
- deleted 4y ago[deleted]
- mkl 4y agoThey do that in the article too.
- FridgeSeal 4y agoYes but then I have to, 1. use python, 2. deal with the nightmare that is python packaging 3. hope I don't run out of memory while Python does its thing. Getting non-technical people to do all that (a project manager wanting to see some simple stats for example) becomes difficult. Having a single binary that's easier to distribute, faster to execute and can be plugged into an existing query tool (like DataGrip) is a huge win for actual usability.