3 ms·
Ive had issues in the other direction. Importing data using the copy from s3 command has a nasty failure state if the client loses connection to the db AND the
by Marazan 4y ago
Ive had issues in the other direction.
Importing data using the copy from s3 command has a nasty failure state if the client loses connection to the db AND the import command has a problem.
The query hangs with a row exclusive lock and the query _cannot_ be killed. Only option was to reboot the DB.
- higeorge13 4y agoYou mean this function: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.S3Import.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...? I don't get that Postgres RDS only allows you to import from csv, while the exports are strictly parquet. I know that Postgres implements the COPY command using csv or binary, but i would expect some end-to-end lifecycle implementation by AWS and parquet would at least enforce some schema and reduce some errors including those which caused yours.
- iknownothow 4y agoAs far I can tell, exporting query results from Postgres RDS to S3 uses Postgres's Copy and only supports CSV and Text [1]. Can you provide a link for parquet exports? [1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postgresql-s3-export.html#postgresql-s3-export-examples https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/postg...
- vanpythonista 4y agoYou can use a RDS Snapshot to export to S3 in Parquet format: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...