4 ms·
cut solved your problem. Let's talk about why. cut is line oriented, like most Unix style filters. It needs to keep only one line in memory at most. If you sa
by jzwinck 8y ago
cut solved your problem. Let's talk about why.
cut is line oriented, like most Unix style filters. It needs to keep only one line in memory at most.
If you say:
pd.read_csv(f)[[x,y,z]]
It has to read and parse the entire 28GB into memory (because it is not lazily evaluated; cf Julia).
If you actually need to operate on three columns in memory and discard the rest, you should:
pd.read_csv(f, usecols=[x,y,z])
Then you get exactly what you need, and avoid swapping.
The lack of lazy evaluation does inhibit composition--just look at the myriad options in read_csv(), some of which are only there to enable eager evaluation to remain efficient.
- makapuf 8y agoOr just use python and iterate your file using csv lib to cater for strange cases while doing your computations along.
- _wmd 8y agoWhile Pandas csv parser is quite slow anyway, the reason Pandas is particularly slow in this case is because it insists on applying type detection to every field of every row read. I have no clue how to disable it, but it's default behaviour. Parsing isn't actually a tough problem – https://github.com/dw/csvmonkey https://github.com/dw/csvmonkey is a project of mine, it manages almost 2GB/sec throughput _per thread_ on a decade old Xeon