4 ms·
The pandas memory consumption is hilarious. The last time I was trying to use pandas, it was the hackernews data dump. It wasn't big. However when pandas start
by pleasecalllater 9y ago
The pandas memory consumption is hilarious.
The last time I was trying to use pandas, it was the hackernews data dump. It wasn't big. However when pandas started using the memory, my 32GB was just too little.
I just ended to convert the data within postgres, much faster, with sensible memory usage.
- jjawssd 9y agoAt what point did your memory consumption go too high? https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas https://stackoverflow.com/questions/25962114/how-to-read-a-6...
- pleasecalllater 9y agoAt the point when it was loading csv file with data. I think there was 5-6GB. My machine has 32GB of ram, and when usage of the python process alone increased to 35GB, I just stopped that. I've tried a couple of times, with similar results. I have loaded that csv file to postgres, the database had similar size. With indexing it was 15GB on disk. All queries were quite fast. So instead of loading the data to pandas, and making searches, I just wrote some SQL, and got the same results. However much faster, and with much smaller memory usage.
- F_J_H 9y agoI'm always a little baffled as to why people just don't dump the data into a database, and then use SQL for further data manipulation and analysis. I recently moved some data processing from Python/pandas into a database, and with SQL, the processing time went from several minutes to a couple seconds, (and that on a tiny VM). I understand that not everyone is familiar with databases and SQL, and so default to the toolset they know. But, the performance gains can make learning databases and SQL highly worthwhile. (And, much can be learned in a just few days, especially for those already familiar with working with data.)
- jampekka 9y agoSQL is a total pain for many types of data. Eg time-series analysis is horrible and large datasets are almost impossible to fetch to the application due to the hilariously inefficient serialization formats.
- F_J_H 9y agohmmm...that's not been my experience, but YMMV.
- pleasecalllater 9y agoIf large datasets are impossible to fetch from a SQL database, even more impossible is to load them to memory. From my point of view, the SQL database can store the huge dataset, with its changes, and I can iterate through the results, and make lots of nice queries getting only the data I need.
- jampekka 9y agoThis depends largely on the use case. For example PostgreSQL insists of transferring all data using ASCII encoding, so eg. high sampling rate floating point sensor readings are extremely slow to fetch from the database. And not all operations can be done incrementally by iterating through the results.
- anarazel 9y ago> This depends largely on the use case. For example PostgreSQL insists of transferring all data using ASCII encoding, so eg. high sampling rate floating point sensor readings are extremely slow to fetch from the database. It doesn't. There's a binary version of the protocol. The output conversion for that is near trivial (transformation to big endian).
- jampekka 9y agoI stand corrected. The problem was due to Python drivers at the time not supporting it. Now there appears to be asyncpg.