I guess the dependency order is nice, but I’ve been frustrated that Postgres doesn’t provide an efficient binary data format that user code can read or write ea

I guess the dependency order is nice, but I’ve been frustrated that Postgres doesn’t provide an efficient binary data format that user code can read or write easily. The SQL, text, and CSV formats are all reasonable and nice for small databases, but if you need to move and especially transform many gigabytes of Postgres data, you are out of luck. The disk binary format of pg_dump and COPY are opaque different formats, the COPY docs have this to say about the binary format: > To determine the appropriate binary format for the actual tuple data you should consult the PostgreSQL source, in particular the send and recv functions for each column’s data type (typically these functions are found in the src/backend/utils/adt/ directory of the source distribution) I didn’t find an easy to use 3rd party library for decoding or reading these native data types, although maybe one exists (then I would chalk my pain up to Google getting worse…) It feels like such a loss to spend CPU cores parsing ASCII CSV, or even worse interpreting & executing millions of SQL insert statements. To approach any kind of reasonable throughput near the maximum of disk or network IO the only answer is to have a big batch job export the whole DB into a reasonable binary format like parquet/arrow/whatever, and then use a replication job to keep it up to date using deltas. You know, if you have the time and money to set up a whole data warehouse thing. Then your transform job can read the 3rd party binary data format relatively quickly and do its transform, but when it comes to moving the data back into Postgres, the fastest I’ve found is to turn it back into CSV, stream the CSV into Postgres using COPY, and spend valuable Postgres CPU cycles having it re-parse the CSV again. Has anyone found a faster pathway to write bulk data into Postgres, perhaps that doesn’t involve at least a one trip through ASCII?