4 ms·
A long time ago, I was working on a team that sorta misused Postgres as a data processing pipeline, so a lot of temp tables were involved. As the article said,
by frollogaston 1y ago
A long time ago, I was working on a team that sorta misused Postgres as a data processing pipeline, so a lot of temp tables were involved. As the article said, COPY was by far the fastest way to get data in from outside. But the next problem after that was INSERT (SELECT ...) statements going from one stage to another.
These queries had many joins and sometimes aggregates on huge tables, which took hours. Surprisingly, for most queries, I got nearly a linear speedup wrt core count (on a 32-core server) by running a separate query in parallel for separate ranges of pkeys. It was best to pick a table to shard that the query planner picked as the inner or outermost when you EXPLAIN the original query, cause that wouldn't change the query plan.
Postgres at the time had no kind of parallelization for a single query. It later got it, but limited to seq scans and other things later. So this method might still be advantageous today. It's not too useful for typical app backends where you want to query your data at one consistent timestamp, but I wonder if Postgres could build some feature that uses parallel queries at the same MVCC timestamp.