4 ms·
stdlib only excludes a bunch of the common answers (i.e. this sounds like a case where pandas etc would be useful, shifting work to native code). That said, 10
by detaro 8y ago
stdlib only excludes a bunch of the common answers (i.e. this sounds like a case where pandas etc would be useful, shifting work to native code). That said, 10 min for 5 GB sounds bad - that's only 8 MB/s?
Generally, it's hard to answer in abstract what the problem might be. Some starting points: You might be unnecessarily creating temporary objects - e.g. large lists. If you don't actually need csv parsing, the csv module might be slower than something simpler. A profiler might help you there. You can modify your program for multiple cores, but that's IMHO a later step once you got the baseline fast.