2 ms·

I was trying https://github.com/sfu-db/connector-x https://github.com/sfu-db/connector-x and hacking around with this https://github.com/spitz-dan-l/postgres-bi

by m_ke 4y ago

I was trying https://github.com/sfu-db/connector-x https://github.com/sfu-db/connector-x and hacking around with this https://github.com/spitz-dan-l/postgres-binary-parser https://github.com/spitz-dan-l/postgres-binary-parser but it turned out that a COPY to csv using asyncpg and then converting to parquet was the fastest.

xwowsersx 4y ago
Ah. connector-x claims to be able to do a lot. Seems useful if it can actually deliver on that. Good to know that COPY with asyncpg is a quick approach. In my case, I had parquet to begin with because I accidentally deleted some production data (oopsies) and when you export a snapshot from RDS to S3, it is in Parquet. Thankfully, I now have a few tricks up my sleeve to quickly restore data, but that was stressful for a bit haha