3 ms·
This is pretty cool! What functionality does this provide that Pandas doesn't? Can it handle tables too big for memory? For example, could I use it to read 10
by jastr 10y ago
This is pretty cool!
What functionality does this provide that Pandas doesn't?
Can it handle tables too big for memory? For example, could I use it to read 10 gigs from PostgreSQL to a CSV.
- thombashi 10y agoI would recommend that divide one SELECT query for 10GB data into for-loop that smaller SELECT query (like 1MB-data/query) execution and writing table by using LIMIT and OFFSET clauses. In that way, you don't need a large amount of memory.