7 ms·
Just a heads up, pandas to Csv is laughably slow. You're literally better off converting the pandas df to arrow and then serialise to Csv with arrow, though the
by awild 5y ago
Just a heads up, pandas to Csv is laughably slow. You're literally better off converting the pandas df to arrow and then serialise to Csv with arrow, though the format is quite limited with arrow.
- NavinF 5y agoWell at least it’s better than numpy’s csv functions. For my use case (~4 million images, one row per image, 2048 fp32 columns for features extracted with resnet), parsing/encoding csv with pandas was 10x faster than with numpy. Of course serializing the arrays to/from npy was 10x faster than any csv library.