2 ms·
Great idea! The data model and code describe in "4.2 Scheduling Studies" is very similar to a PostgreSQL stored procedure scheduler system we've written to par
by JoelJacobson 5y ago
Great idea!
The data model and code describe in "4.2 Scheduling Studies" is very similar to a PostgreSQL stored procedure scheduler system we've written to parallelize large UPDATE/INSERT ... SELECT jobs, parallelizable thanks to having a "shard_id" column similar to the "p_key" column mentioned in the paper.
Currently, each INSERT reading directly from a SELECT query, must run as separate queries, by each worker, for each CPU, to parallelize the workload, to fully utilize the server hardware.
We already have parallelizable SELECT queries in PostgreSQL. If the future would bring us parallelizable UPDATE/INSERT queries, we would finally have all the ingredients to run a single SQL query that would not only perform the computations in parallel to produce the aggregated result, but also write it in parallel to the database.
Btw, I think "p_key" in the paper is confusing, as I first though it meant "primary key" but then understood it means "partition key".