3 ms·
Tried that in the past, but it's really slow. Pandas is effectively removed from my workflows because of issues like this. But, I have workarounds for these is
by chaps 4y ago
Tried that in the past, but it's really slow. Pandas is effectively removed from my workflows because of issues like this.
But, I have workarounds for these issues by loading everything into postgres under TEXT columns in a "raw" schema, then do some typecast tests in a descending list of types to get the smallest possible type to transfer to a new table in a "prod" schema. It's read-only data, so it's not a big deal to run it once, and builds out a chain of changes from csv -> sql.
Something like this could be done with pickling to avoid having to re-type every time I run the code (and I've done that for some past projects, but it's... ehhh).