4 ms·
We use pandas for opening big data files, manipulating the data and then save it to a pickle file or a sql database for use consumption.
by chudi 9y ago
We use pandas for opening big data files, manipulating the data and then save it to a pickle file or a sql database for use consumption.
- gaius 9y agoDataFrame.to_sql() seems painfully slow, do you have any tips for speeding it up? What I do is dump it as CSV again after manipulation and then bulk load it into the DB using the DB's native tool. It's messy but 10x faster.