3 ms·
thx! Indeed, upon reading the source and sleeping on it, I agree, and in fact it looks like a single-user postgres backend with COPY FROM <filename> BINARY is a
by asah 3y ago
thx! Indeed, upon reading the source and sleeping on it, I agree, and in fact it looks like a single-user postgres backend with COPY FROM <filename> BINARY is approximately the same architecture as writing the database files in-place, and of course includes support for default values, triggers, constraints, TOAST and more.
I've reproduced the speed difference between pg COPY vs various cases.
Some results (middle result of 3 stable runs) from 1.4GB BINARY dump:
echo "drop table if exists tbl; create table tbl(city int2, temp int2);copy tbl FROM '/citydata.bin' binary;" | ./pg/bin/postgres --single -D tmp -p 9999 postgres;
real 0m34.508s
# switching to unlogged table
real 0m30.620s
# hardcoding heap_multi_insert() to be a NOOP (return early if ntuples>100)
# fyi, heap_multi_insert() gets called with n=1000 tuples per call
real 0m11.276s
# hardcoding skip_tuple = true in src/backend/commands/copyfrom.c:1142
real 0m6.894s
# after testing various things
time sh -c "tar cf - citydata.bin | (cd /tmp; tar xf -)"
real 0m2.811s
Note: I tried increasing the blocksize (--with-blocksize) and also MAX_BUFFERED_TUPLES (copyfrom.c:65) but as expected they didn't help, I guess n=1000 tuples amortizes the overhead.